Operative Experience vs. Competence: A Curriculum Concordance and Learning Curve Analysis

doi:10.1016/j.jsurg.2016.01.011

Journal of Surgical Education

Volume 73, Issue 4, July–August 2016, Pages 694-698

https://doi.org/10.1016/j.jsurg.2016.01.011 Get rights and content

Objective

Certification of completion of training in general surgery requires proof of competence of index operations by means of 3, level-4 consultant-validated procedural-based assessments. The aim of this study was to examine the relationship between index operative experience and competence.

Design

Higher surgical trainee procedural-based assessments were compared with e-logbooks to determine the relationship between index operative experience and achievement of a third level 4 competence (L4C) related to the indicative procedures of emergency laparotomy (EL, target 100), Hartmann procedure (5), appendicectomy (80), segmental colectomy (20), laparoscopic cholecystectomy (50), and inguinal hernia (80).

Setting

All trainees are from a single UK Deanery.

Participants

Consecutive 69 national training number higher surgical trainees were appointed to a single UK Deanery between 2007 and 2014.

Results

EL L4C was achieved at a median of 76 (15-136) cases, Hartmann procedure L4C at 17 (7-27) cases (p = 0.009 vs. EL), appendicectomy L4C at 107 (20-206) cases, segmental colectomy L4C at 52 (15-131) cases, laparoscopic cholecystectomy L4C at 72 (40-197) cases, and inguinal hernia L4C at 64 (17-132) cases.

Conclusions

The learning curve and caseload required to demonstrate L4C related to specific procedure varied over 4-fold, from 0.76 to 3.4 times the national indicative target number guidance. Certification of completion of training operative logbook number targets should be reconsidered to better reflect the competencies demanded by the curriculum.

Introduction

Good medical practice and proof of surgical competence have long been considered important. Indeed, the Royal College of Surgeons of Edinburgh original charter of 1505 stated,

“No manner of person [shall] employ our said crafts of surgery unless he be worthy and expert in all the subjects belonging to the said crafts, [and be] diligently and advisedly examined.”¹

Surgical education and postgraduate training is under scrutiny, not least because of its craft specialty status, but because of iterative and progressive initiatives aimed at improving yet shortening training time.2, 3 The traditional British surgical apprenticeship has undergone profound reconfiguration, culminating in a reformed postgraduate medical training program (modernizing medical careers) which, allied to a competency-based approach, demands quantifiable educational outcomes and unequivocal proof of ability.4, 5, 6 The Intercollegiate Surgical Curriculum Programme (ISCP)⁷ and the associated General Medical Council curricula have reshaped surgical training and have contained required levels of competence since their inception, but until 2013 no specific quantitative guidelines existed for the award of a certificate of completion of training (CCT) in general surgery. At this juncture, the UK Joint Committee on Surgical Training published specific CCT guidance, citing among other professional credentials, clinical competence, operative experience, and operative competence.⁸ The caseload components of the above include key procedures, chosen because of their clinical importance and because they can be used as a marker of experience across the breadth of the curriculum. Experience in each key procedure is compared with an indicative target number set at a level that corresponds to the first quartile from a modest cohort of successful national CCT applicants in previous years.⁹ These curricular numbers have caused a degree of angst because of fears that the levels may be unrealistic and unachievable in certain specialties and training posts.¹⁰

Validated assessment tools such as the procedural-based assessment (PBA) allow objective assessment of technical skills, with each procedure associated with a generic descriptor-based grading (Fig. 1).¹¹ Evidence of competence for each procedure is required by completion of 3 consultant-validated PBAs at level 4 (competent to perform independently and deal with complications, Fig. 1). The aim of this study was to examine the relationship between the volume of operative experience in the key indicative procedures required of all general surgery trainees and the level of competence achieved within a higher surgical training program in a single UK Deanery.

Section snippets

Materials and Methods

All national training number higher surgical trainees (HSTs) within a single UK deanery that had commenced higher surgical training between August 2007 and August 2014 were identified, and the relevant ISCP online portfolios were interrogated through the Head of School report function. Individual HST reports were created with particular reference to PBA dates and competence levels achieved for the 6 indicative procedures required of all general surgery HSTs (Table 1). Access to trainee

Results

A total of 69 consecutive HSTs were included in the analysis, and details regarding the grade of the HSTs and the median number of key indicative procedures performed are shown in Table 1.

A total of 8 HSTs had achieved all the PBA operative competences required for CCT (7 specialty trainee year 8 [ST8], 1 ST7 grade), and 7 HSTs had achieved CCT PBA competences in 5 of the 6 procedures (1 ST8, 2 ST7, 3 ST6, and 1 ST5). A total of 18 HSTs met the indicative operative target numbers for all 6

Discussion

This is the first study to examine the relationship between operative experience and competence within a contemporary UK surgical training program, and the principal finding was that the CCT-defined competence was achieved within the set operative target window in only 1 of 6 key indicative procedures, with competence ratios ranging over 4-fold, from a minimum of 0.76 (EL) up to a maximum of 3.4 (HMN). Arguably therefore, current CCT guidance targets of 60 inguinal hernia and 100 EL operations

Conclusion

This study has raised questions regarding curriculum-related specialty operative training standards that aid in the acquisition of the knowledge and skill set required for independent consultant practice. European Working Time Regulations have reduced surgical training hours, with the inherent risk of providing lower levels of operative experience than hitherto, and it is important that the standards currently defined are monitored to quality assure the future consultant workforce. The

References (23)

R.K. Reznick
Teaching and testing technical skills
Am J Surg
(1993)
A.B. Lowenfels
Highlights of the Royal College of Surgeons of Edinburgh Quincentenary Congress
Medscape
(2005)
Directive 2003/88/EC of the European Parliament and of the Council of 4 November 2003 concerning certain aspects of the...
Greenaway D. Shape of training final report. Shape of training expert advisory group. 2015. Available at:...
Temple J. On behalf of medical education england. Time for training. A review of the impact of the European working...
S.R. Moonesinghe et al.
Impact of reduction in working hours for doctors in training on postgraduate medical education and patients’ outcomes: systematic review
Br Med J
(2011)
Modernizing Medical Careers. Available at:...
The Intercollegiate Surgical Curriculum Program Website. Available at:...
Joint Committee on Surgical Training. Guidelines for the Award of a CCT in General Surgery 2013. Available at:...
W. Allum et al.
General surgery logbook survey
Bull R Coll Surg Engl
(2013)

C. Thomas et al.

Data from two deaneries regarding the ability of higher surgical training programmes to deliver curriculum targets

Bull R Coll Surg Engl

(2015)

Cited by (36)

Is the robotic revolution stunting surgical skills?
2024, Surgery Open Science
This perspective piece aims to examine the impact of the growing utilization of robotic platforms in general and minimally invasive surgery on surgical trainee experience, skill level, and comfort in performing general surgical and minimally invasive procedures following completion of training. We review current literature and explore the application of robotic surgery to surgical training, where minimum case thresholds and breadth distribution are well defined, and where development of surgical technique is historically gained through delicate tissue handling with haptic feedback rather than relying on visual feedback alone. We call for careful consideration as to how best to incorporate robotics in surgical training in order to embrace technological advances without endangering the surgical proficiency of the surgeons of tomorrow.
The large-scale incorporation of robotics into general and minimally invasive surgical training is something that most, if not all, trainees must grapple with in today's world, and the proportion of robotics is increasing. This shift may significantly negatively affect trainees in terms of surgical skill upon completion of training and must be approached with an appropriate degree of concern and thoughtfulness so as to protect the surgeons of tomorrow.
Defining the Disparity: A Multi-Institutional Analysis of Factors Associated With Decreased Resident Operative Experience
2024, Journal of Surgical Research
Technical learning in surgical training is multifaceted and existing literature suggests a positive relationship between case volume and proficiency. Little is known about factors associated with a decreased volume of operative experience. This study aimed to identify resident and program factors associated with general surgery residents (GSR) in the bottom quartile of logged case volume upon program completion.
A post hoc analysis of a multicenter study was used to examine case logs for categorical GSR. Participants included graduates between 2010 and 2020 from 20 programs. Residents below and above the 25th percentile for total operative volume were compared.
The present study includes 1343 GSR who graduated over the 11-y period. In total, 336 residents were below the 25th percentile and 1007 residents were above the 25th percentile. Those below the 25th percentile were more likely to be female (41% versus 34%, P = 0.02), identify as underrepresented in medicine (22% versus 14%, P < 0.01), and pursue fellowship (86% versus 80%, P = 0.01) compared to those above the 25th percentile. Residents below the 25th percentile were more likely to have graduated from a low volume program (55% versus 25%, P < 0.01) and from top National Institutes of Health funded institutions (57% versus 52%, P = 0.01).
This study identified individual and program characteristics associated with lower operative volume of GSR. Understanding such characteristics will aid surgical educators to achieve better equity in training.
Characterizing individual trainee learning curves in surgical training: Challenges and opportunities
2023, Surgeon
The surgical learning curve is an observable and measurable phenomenon. In the era of competency-based approaches to surgical training, monitoring the trajectory of individual trainee competence attainment could represent a meaningful method of formative and summative assessment. While technology can assist this approach, a number of significant barriers to the implementation of such assessment methods remain, including: accurate data collection, standard setting, and reliable assessment. Translating individual learning curve data into quantifiable case minimum targets in training poses further difficulties, and may not be possible for all procedures, particularly those that are less frequently performed and assessed. In spite of these challenges, significant benefits could be realized through an individualized approach to competency assessment using trainee learning curve data. Tracking competence acquisition against criterion-referenced standards could allow for targeted training and remediation, conforming with modern theories of adult education and empowering trainees to take control of their own learning. Learning curve data could also be used to assess the effects of educational interventions such as simulation-based training on subsequent competence acquisition rates. Ultimately, the individual learning curves of trainees could be used to inform personalised decisions regarding entrustment, credentialing, and certification, allowing training programmes to move beyond minimum operative experience targets as a crude proxy measure of competence.
Early Evaluation of a New French Surgery Course in the Best Practice of Dealing With Major Incidents and Mass Casualty Events
2023, Journal of Surgical Education
The main objective of this study is to evaluate the impact of a nationwide 5-month course aimed to prepare surgeons for Major Incidents through the acquisition of key knowledge and competencies. Learners’ satisfaction was also measured as a secondary objective.
This course was evaluated thanks to various teaching efficacy metrics, mainly based on Kirkpatrick's hierarchy in medical education. Gain in knowledge of participants was evaluated by multiple-choice tests. Self-reported confidence was measured with 2 detailed pre and post training questionnaires.
Creation in 2020 of a nationwide, optional and comprehensive Surgical Training in War and Disaster Situation as part of the French surgery residency program. In 2021, data was gathered regarding the impact of the course on participants’ knowledge and competencies.
The study included 26 students in the 2021 cohort (13 residents and 13 practitioners).
Mean scores were significantly higher in the post-test compared to the pre-test, showing significant increase in participants’ knowledge during the course: 73,3% vs. 47,3% respectively (p ≤ 0.001). Average learners’ confidence scores to perform technical procedures showed at least a +1-point increase on the Likert scale for 65% of items tested (p ≤ 0.001). 89% of items showed at least a +1-point increase on the Likert scale when it came to average learners’ confidence score on dealing with complicated situations (p ≤ 0.001). Our post-training satisfaction survey showed that 92% of all participants have noticed the impact of the course on their daily practice.
Our study shows that the third level of Kirkpatrick's hierarchy in medical education was reached. This course therefore appears to be meeting the objectives set by the Ministry of Health. Being only 2 years old, it is on the road to gathering momentum and further development.
Defining operative experience targets in surgical training: A systematic review
2022, Surgery (United States)
Citation Excerpt :
Evidence to date has suggested that targets in the United Kingdom and the United States may be set too low to ensure competency in most graduating trainees. For example, findings from large studies by De Siqueira, Abdelrahman, and Browne demonstrated that the mean number of procedures required for trainees to meet predefined levels of operative competence is greatly in excess of numbers mandated by the United Kingdom training curriculum for cholecystectomy, segmental colectomy, inguinal hernia repair, and appendicectomy procedures.6,20–22 Similarly, in 2017, Stride et al’s study identified the transition to meaningful autonomy (defined as requiring ‘passive help’ or ‘supervision only’) for the majority of residents in partial colectomy occurring after 60 cases, in excess of the 40 ‘large intestine’ procedures expected by the ACGME.9,31
The surgical learning curve is an observable and measurable phenomenon. Operative experience targets are well established as a proxy measure for operative competence in surgical training across jurisdictions. The aim of this study was to critique the available evidence regarding the relationship between operative experience in surgical training and trainee competence.
A systematic review of the PubMed, Embase, Web of Science, and Cochrane library databases was conducted in accordance with the Preferred Items for Systematic Reviews and Meta-Analyses guidelines. Articles were sought that defined the relationship between procedural volume in surgical training and trainee competence, proficiency, or mastery. The educational impact of included studies was evaluated using a modified Kirkpatrick model.
Of 3,672 records identified on database searching, 30 papers were ultimately included. Fourteen studies defined operative experience thresholds using operative time as a surrogate measure of competence, whereas another 8 used trainer assessments of operative performance (Kirkpatrick level 3). A further 5 studies were able to determine the relationship between trainee case volumes and subsequent patient outcomes (Kirkpatrick level 4b).
Many studies have recorded competent trainee performance in key index procedures after reaching experience threshold numbers in excess of currently mandated targets across jurisdictions. The evidence relating current operative experience targets to patient outcomes across a range of surgical subspecialties of surgical subspecialties is lacking. This review supports a move toward criterion-based referencing of operative performance targets in surgical training.
Reduction in ionizing radiation exposure during minimally invasive anterior plate osteosynthesis of distal radius fracture: Naive versus deliberate practice
2022, Hand Surgery and Rehabilitation
This study aimed to decrease surgeon exposure to ionizing radiation through a new learning technique, “deliberate practice”, which consists in improving performance by setting goals with feedback. The hypothesis was that exposure to ionizing radiation during distal radius fracture surgery using the minimally invasive plate osteosynthesis (MIPO) technique decreased faster with “deliberate” practice than with “naïve” practice. Radiographic dosimetry was measured in the first 30 fractures operated on by MIPO by 6 surgeons. The first 3 surgeons operated “naively” (Group 1) and the next 3 according to the “deliberate” procedure (Group 2). Group 2 received weekly feedback (number of exposed hands, number of fluoroscopic views, exposure duration, and X-ray dose). An expert, using fluoroscopic images and surgical videos, provided suggestions for improvement. Mean number of exposed hands was 23.66 in Group 1 and 1.9 in Group 2. Mean number of fluoroscopic views was 78.31 and 35.0, respectively. Mean X-ray exposure time was 74.34 and 32.89 s, respectively. Mean dosimetry was 1.40 mGy (and 0.59 mGy, respectively. The hypothesis was thus confirmed: dosimetry decreased faster in Group 2 than in Group 1. Teaching this deliberate practice should be generalized, to decrease the growth phase and increase the plateau phase of the learning curve.
Le but de ce travail était de diminuer l’exposition aux radiations ionisantes des chirurgiens grâce à une nouvelle technique d’apprentissage, la “pratique délibérée”. Elle consiste à améliorer la performance en se fixant des objectifs avec leur retour d’expérience. L’hypothèse était que l’exposition aux radiations ionisantes des chirurgiens ostéosynthésant des fractures de l’extrémité distale du radius selon une technique mini-invasive (MIPO) diminuait plus vite avec “pratique délibérée” qu’avec “pratique naïve”. La dosimétrie X des 30 premières fractures opérées par MIPO par six chirurgiens a été mesurée. Les trois premiers chirurgiens opéraient selon une pratique naïve (groupe 1) et les trois suivants selon une pratique délibérée (groupe 2). Ces derniers recevaient un retour d’expérience hebdomadaire (nombre de mains sous les rayons X, nombre d’incidences fluoroscopiques, durée et dose d’exposition aux rayons X). Des pistes d’amélioration étaient données par un expert grâce aux clichés fluoroscopiques et aux vidéos chirurgicales. Le nombre moyen de mains sous les rayons X était de 23,66 (groupe 1) et 1,9 (groupe 2). Le nombre moyen d’incidences fluoroscopiques était de 78,31 (groupe 1) et 35,0 (groupe 2). La durée d’exposition moyenne aux rayons X était de 74,34 s (groupe 1) et 32,89 s (groupe 2). La dosimétrie moyenne était de 1,40 mGy (groupe 1) et 0,59 mGy (groupe 2). L’hypothèse était vérifiée puisque la dosimétrie diminuait plus vite dans le groupe 2 que dans le groupe 1. L’utilisation de la pratique délibérée dans l’enseignement de la chirurgie devrait être généralisée pour diminuer la phase de croissance et augmenter le niveau de la phase en plateau de la courbe d’apprentissage.

View all citing articles on Scopus

View full text

Original ReportsOperative Experience vs. Competence: A Curriculum Concordance and Learning Curve Analysis

Objective

Design

Setting

Participants

Results

Conclusions

Introduction

Section snippets

Materials and Methods

Results

Discussion

Conclusion

Am J Surg

Highlights of the Royal College of Surgeons of Edinburgh Quincentenary Congress

Medscape

Impact of reduction in working hours for doctors in training on postgraduate medical education and patients’ outcomes: systematic review

Br Med J

General surgery logbook survey

Bull R Coll Surg Engl

Data from two deaneries regarding the ability of higher surgical training programmes to deliver curriculum targets

Bull R Coll Surg Engl

Original Reports
Operative Experience vs. Competence: A Curriculum Concordance and Learning Curve Analysis