Simulation for Medical Specialty Initial Certification and Maintenance of Certification in the United States of America

Surgical skills simulation (SSS) tests the application of factual knowledge and shows how knowledge is applied, representing the second and third levels of Miller’s Pyramid of Learning. SSS permits high-stakes scenario testing in safe environments. Therefore, SSS incorporation into initial specialty certification began in 2002 in Australia and New Zealand. The United States began SSS incorporation into specialty certification in 2008. This paper will determine where the United States stands in the process of SSS incorporation into specialty certification. Google scholar Internet and PubMed searches phrased “medical board certification surgical skills simulation”, performed on September 1, 2016 yielded 16 relevant articles. Hand search on September 1, 2016 yielded 7 additional articles. In 2008, cardiac catheterization simulation was required for interventional cardiology maintenance of certification (MOC). In 2010 the American Board of Anesthesiology (ABA) required SSS as part of the MOC program. In 2014, the summative assessment, Colorectal Objective Assessment of Technical Skills became part of the American Board of Colon and Rectal Surgery certification. In 2017, SSS will be added to the ABA initial certification examination. The United States has been slow to incorporate SSS into initial certification and MOC. Assessment validation, capital and recurring costs, personnel, physical facility and time requirements are barriers limiting SSS expansion into specialty certification processes. As SSS allows rapid technical skill assessment, without posing a threat to patients, expansion of SSS into initial certification and MOC programs represents non-maleficence and beneficence, and should be encouraged. Simulation for Medical Specialty Initial Certification and Maintenance of Certification in the United States of America


Introduction
Surgical skills simulation (SSS) allows surgical specialties to perform measurable and significant portions of initial and recurrent training with box trainers, partial-task trainers, full-body mannequins, and in virtual reality and haptic environments, instead of on living patients [1]. Construct, external, face, and predictive validity, reliability, and fidelity, have permitted comparative effectiveness evaluation of different surgical skills simulators (see Table 1). In 2002, the Australian and New Zealand College of Anesthetists (ANZCA) initiated SSS use for initial medical board certification, when participation in a 2.5-day five module SSS program became requisite for ANZCA fellowship (see Table 2) [2].
In 2003, the Israeli Board of Anesthesiology added an objective structured clinical evaluation (OSCE) of five 15-minute simulation stations to the Israeli Board of Anesthesiology examination (see Table  3) [1,3]. The British Royal College of Anaesthetists (RCOA) followed suit with a 17 station OSCE as part of initial certification [1]. This OSCE has since been increased to 18 stations, of which 2 are new task test questions. The College of Anaesthetists of Ireland (CAI) has used the RCOA OSCE as a basis for the CAI OSCE. In 2004, the European Board of Vascular Surgery (EBVS) added a validated, three-module SSS examination to the EBVS certification process [4,5]. An endovascular component was added in 2007 (see Table 4) [6,7]. The EBVS SSS examination uses Imperial College Evaluation of ProcEducre-specific Skill (ICEPS) and objective structure assessment of technical skills (OSATS) rating scales [7]. Since 2004, simulation-based medical device training has been a United States Food and Drug Administration approval requirement [8]. However, American specialty boards have been slower to require SSS as part of initial certification. Skills simulation is rated as a more effective learning technique than conventional didactic sessions. Conventional didactic assessment via multiple-choice questions, essay-type questions, and oral examinations demonstrate factual knowledge, representing levels 1 and 2 of Miller's Pyramid of Learning [1]. Simple SSS tests the application of factual knowledge. SSS scenarios, OSCEs, and skills demonstration task trainers show how knowledge is applied. Thus, SSS directly tests levels 1,2 and 3 of clinical competence in Miller's Pyramid of Learning [1]. Therefore, SSS can satisfy continuing medical Educcation requirements for national or state medical board licensure renewal and specialty board maintenance of certification (MOC), at a higher rate per hour spent than conventional didactic sessions [9].
Medical malpractice insurers can increase the stakes for SSS participation by adding the financial incentive of rEducced premiums, as team-based skills simulation can rEducce malpractice claims [9]. Medical malpractice insurers can also choose to provide SSS training at no cost to covered surgeons [10]. For example, in 2005 the Controlled Risk Insurance Company of Harvard's Risk Management Foundation provided FLS training to 112 laparoscopists as part of a patient safety initiative [10]. Institutions may receive grant funding for SSS, as has been done for FLS training [10]. By facilitating safer initial training and promoting skill maintenance and teamwork, surgical skills simulation may contribute to increased health care quality and safety [8].

Methods
A Google scholar Internet search phrased "medical board certification surgical skills simulation", performed on September 1, 2016 yielded 15 relevant references published from 2012 to 2016. A PubMed search on September 1, 2016, phrased "medical board certification surgical skills simulation" without language or time restrictions yielded 1 relevant article. Hand search on September 1, 2016 yielded 7 articles as shown in the literature flowchart Figure 1, resulting in 23 included articles. The derived status of simulation for medical specialty initial certification and maintenance of certification as described below is summarized in Table 5.

Anesthesiology
Maintenance of certification in anesthesiology simulation course: The formative Maintenance of Certification in Anesthesiology MOCA® Simulation Courses began in 2010 as a required part of the Practice Performance Assessment and Improvement element [11]. The MOCA® Simulation Courses are overseen by the American Society of Anesthesiologists' (ASA) Simulation Editorial Board (SEB). A MOCA® Simulation Course is required once per 10-year A MOCA® Simulation Course is required once per 10-year MOC cycle. There are 39 ASAendorsed 6-to 8-hour long MOCA® Simulation Courses, spread across 21 states, which are home to 75% of course registrants. MOCA® Simulation Courses include a minimum of one scenario per primary anesthesiologist, with role-playing and debriefing. Thus, MOCA® Simulation Courses are consistent with the adult learning cycle [1,12]. Anesthesiology skills simulation allows for experience in coping with the 30% of anesthesiology that has to manage non-routine events. Nonroutine anesthesiology events range from hyperkalemia to malignant hyperthermia and airway fires [11,13].
By 2014, 94% of the 2,700 MOCA® Simulation Course participants found the courses realistic, and had reportedly changed an element of their practice due to the evaluation, self-reflection, and personal development plan that result from the skills simulation [1,11]. Thus, MOCA® Simulation Courses should achieve Kirkpatrick level 4a and 4b outcome evaluation by practice changes that directly benefit patients, whereas written and oral board examinations at most achieve Kirkpatrick level 3 behavioral change through knowledge and skills application. However, randomized controlled trials evaluating the MOCA® Simulation Courses' effectiveness in achieving Kirkpatrick level 4a and 4b outcomes are lacking [1].

Initial certification:
In 2017 SSS OSCEs without high-fidelity mannequins in full-scale crisis management scenarios, will be added to the American Board of Anesthesiology initial certification examination [2].

Colon and rectal surgery
The specialty certification, summative, eight-module Colorectal Objective Assessment of Technical Skills (COSATS) examination is the product of the Operative Competency Committee established in 2009 by the American Society of Colon and Rectal Surgeons [14]. In 2014, COSATS became a required part of the American Board of Colon and Rectal Surgery (ABCRS) certification [4]. COSATS evaluates elective, emergency, endoscopic, laparoscopic, and open procEducres. The COSATS modules are: coloanal anastomosis, colonoscopy, ileal pouch anal anastomosis, hand-sewn anastomosis, laparoscopic ileorectal anastomosis, laparoscopic sigmoidectomy, pelvic bleed, and rectal prolapse. Assessment is based on OSATS [14]. COSATS has shown construct validity, inter-rater and inter-station reliability [14]. COSATS is complementary to the written and oral board examinations as different constructs are tested [14]. However, as candidates who fail COSATS passed the ABCRS oral examination, it has been suggested that the existing written and oral examinations do not provide appropriate technical assessment [4]. The foregoing exemplifies the need for SSS incorporation into specialty certification: Assurance of adequate technical competency assessment [4].

General surgery
Initial board certification: In 2008, completion of Advanced Cardiac Life Support (ACLS), Advanced Trauma Life Support (ATLS), and the Fundamentals of Laparoscopic Surgery (FLS) web-and laparoscopic box-trainer-based training and summative assessment became prerequisites for American Board of Surgery certification eligibility [1]. FLS is derived from the McGill University Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) physical laparoscopic simulator. MISTELS has proven construct, external, and predictive validity [1].
FLS uses five tasks, with pass/fail grading based on contrasting groups frequency distributions of competent and non-competent surgeons and is optimized based on sensitivity and specificity [14]. FLS satisfies Messick's validity conceptual framework: construct, content, and criterion (concurrent and predictive) validity, effective response process with minimal variance, and appropriateness of the assessment to the purpose of testing [14]. There are at least 80 FLS test centers located in 6 countries, with assessment by 184 volunteer proctors, who have proctor MOC requirements to assure FLS validity [15]. However, the FLS has been questioned as being too rudimentary, lacking advanced technical skills testing [4].

Interventional cardiology
Maintenance of certification: Cardiac catheterization simulation with five scenarios was begun as means of Medical Knowledge Self-Evaluation for MOC in 2008 [8,16].

Obstetrics and gynecology
Initial certification: Simulation has been pilot tested for the 2017 maternal-fetal-medicine examinations, which may include sonographic simulation [17]. Candidates will be allowed to use mannequins to illustrate oral examination responses.
Physician reentry into active practice: recertification or recredentialing: Most American health care facilities require a time-

Validity
The extent to which a measurement instrument truly measures what it is supposed to measure, in the right way, and positively influencing learning.

Concurrent validity
The extent to which the results of a test are consistent with the results of another test that is intended to assess the same thing.

Construct validity
The extent to which the assessment, and the individual components of the assessment, tests relevant professional constructs. Differentiation of experts and novices indicates construct validity.
Content validity or conceptual fidelity The extent to which measurement components reflect the abilities (knowledge, skills or behaviors) it is designed to measure.
Criterion validity A measure comprised of concurrent and predictive validity.
Ecological validity Absence of simulation environment effects on participants' performance. A form of external validity.
Face validity or physical fidelity The measurement of how much users (test subjects) feel that the right things are being assessed in the right way.

Population validity
The generalizability of simulation outcomes to other populations. A form of external validity.
Reliability or predictive validity The degree to which an assessment predicts expected outcomes. For example, a measure of attitudes (behavior) toward preventive care should correlate significantly with preventive care behaviors.

Station Content
Airway management Noninvasive surgical airway management in an obstructed airway or inadequate oxygenation from compromised ventilation or difficult intubation.

Anesthetic emergency
Interface between equipment and drugs with hypoxia or increased airway pressure.
Human performance Crisis resource management based teamwork.

Resuscitation
Advanced Cardiac Life Support (ACLS) skills including vascular access.
Trauma management Cervical spine, chest, and head injury management.

Mechanical ventilation
Artificial lung ventilator adjustment based on blood gas results.
Operating room emergency Intra-operative event management as personnel called-in to assist. Simulated operating room with high-fidelity mannequin.
Regional anesthesia Show surface anatomy knowledge, place regional anesthesia with appropriate local anesthetic volume, manage nerve block complications. Standardized patient.
Trauma management Advanced Trauma Life Support (ATLS) skills with an Emergency Department standardized patient / mannequin.
Simulator for Testing Radiological and Endovascular Skills (STRESS) Glass model with catheter and guidewires. bound volume of active cases for initial credentialing and subsequent re-credentialing. There are numerous reasons including accidental injury, care giving, and relocation that can lead to a two year or longer leave from active medical practice. Physicians currently in active practice may want to change their scope of practice to include aspects unpracticed for two years or longer [18]. On a case-by-case basis, when attempting to return to active practice, SSS may be used in lieu of, or to complement an active patient caseload [8]. Simulation can be used to assess residual skills and determine the curriculum to be completed to attain current performance competency [18]. Mount Sinai Medical Center, New York, created a 1-to 6-week simulation-based retraining program with up to 60 hours of 1-on-1 high-fidelity simulation [8,18]. At times return to clinical residency is more appropriate than simulation-based retraining [18].

Strengths of simulation-associated board certification:
By providing technical skill assessment, SSS allows surgeons to undergo more expansive testing than written or oral examinations in a safe environment, without posing a threat to patients [4,8]. Psychomotor, mega-cognitive, leadership, and communication skills are assessed via presentation of multiple problems in a complex environment [8]. High-stakes situations: Rare, challenging, and critical situations can be recreated as simulation exercises [19]. Multiple scenarios are covered quickly [2]. Recorded simulation allows for review, reflection, correction, and directed improvement [8].

Weaknesses of simulation-associated board certification:
Surgical skills simulation's cost-effectiveness is as yet unproven [1,5]. Correlation between surgical skills simulation performance and clinical performance needs further substantiation [1,5]. Surgical skills simulation should be a component, but not form the entirety of what should be multimodal surgeon competency evaluation [1]. If lacking ergonomic validity, simulation-based testing may not be representative of surgeons' real life clinical performance. Test anxiety may worsen examinees' performance, while other examinees may display a hypervigilant Hawthorne effect improved performance [8,19]. certification examination may need to be increased to 70 questions, while removing the easiest and hardest questions [15]. Assuring proctor qualifications adds to cost and logistic requirements of SBA. Continuous quality improvement (CQI) grading rubric and checklist creation is challenging and recurrently time consuming [19]. Given a five-year period from conception to implementation, CQI for SSS can be cumbersome.

The future of simulation in medical board certification
Initial certification: The six-module Fundamentals of Arthroscopic Surgery Training (FAST) program, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), or the Copernicus Initiative: arthroscopic Bankart procEducre assessment may become a prerequisite for American Board of Orthopaedic Surgery certification. Of these, the arthroscopic Bankart procEducre assessment is the most expensive to perform [14]. Real time scoring reliability of ASSET must be determined [22]. Ascertainment of the correct benchmarks for FAST are undetermined [22]. Scalability of the Copernicus Initiative is uncertain [22]. The American Urological Association is coordinating development of the four-module Basic Laparoscopic Urologic Skills (BLUS) and the seven-module Fundamentals of Robotic Surgery (FRS) programs [14]. Both the BLUS and the FRS provide summative assessment [14].
Internal medicine derived specialties, in particular gastroenterology and pulmonology may add simulation to MOC Barriers to simulation use in specialty board certification: Cost, personnel, physical facility, and time requirements present initial and recurring costs to specialty boards and candidates [8]. For practicing surgeons, structured on-site programs can lead to higher initial pass rates than self-study preparation [10]. COSATS costs about USD 1,000 per candidate, FLS about USD 600 per candidate, and ATLS about USD 750 per candidate [10,14]. Surgical skills simulation examinations must contain sufficient modules for broad domain assessment. The EBVS SSS has been criticized for insufficient breadth. Appropriately designed studies are necessary for SSS validation [4]. Transparency of SSS validation leaves room for improvement. Nonpublication of validating studies has led to criticism of the EBVS SSS [4]. Additional study of COSATS has been requested [20]. However, increasing the number of modules per SSS, and conducting and publishing validating studies, also add to SSS costs.
Poor inter-rater reliability and generalizability can threaten SSS predictive validity. Rubrics and checklists can be subjective, promote rote behaviors, and not adequately consider all aspects of professional expertise [19,20]. Recruitment, training, and retention of simulationbased assessment (SBA) proctors can be challenging [19]. It may be necessary to use non-specialty physicians [21]. When SBA proctor training is limited to 30-minutes preceding an examination, the proctor training session will be regarded as inadequate [20]. Despite using Downing's 12 steps for test development to develop a 45-item computer-based FLS proctor certification examination, the FLS proctor Simulation program completion may substitute for active practice. Simulation program may indicate need for return to residency or proctoring. requirements [8]. Bronchoscopy, colonoscopy, endoscopic retrograde cholangiopancreatography, sigmoidoscopy, and upper gastrointestinal endoscopy are amenable to simulation [8]. Emergency medicine, pediatrics, and radiology, particularly interventional radiology may add simulation to MOC requirements [8]. When SSS examination is not an eligibility prerequisite, but is performed in conjunction with oral and written examinations, determination needs to be made as to how best to address candidates who pass the oral and written examinations, but fail the SSS examination [4,14]. If the example of the EBVS is followed, such candidates should be regarded as failing initial certification. Parameter setting for subsequent attempts, such as repeating only the failed examination portion is needed.

Maintenance of certification:
Healthcare facilities are increasingly requesting FLS certification as a prerequisite for laparoscopy privileges [10]. Should the American Board of Surgery (ABS) move to require FLS certification as part of the MOC requirements for surgeons who initially certified before 2008, the ABS would actually be facilitating healthcare facility credentialing requirement satisfaction [10]. The ABS may also seek to add repeated FLS certification as part of the MOC requirements for surgeons who had to complete FLS as a pre-requisite for initial ABS certification. For all specialties using SSS as part of MOC requirements, determination of a minimum SSS frequency should be made. Due to logistical and financial constraints, the minimum SSS frequency will be less often than every fifth week that nuclear power plant operation teams spend in simulation [1,11]. However, given knowledge decay after 6 to 12 months, the minimum SSS may need to be at least equivalent to aviators' annual simulation [1,11,23].

Conclusion
Australia, Israel, Europe, and New Zealand seized the forefront of SSS in medical Educcation and specialty certification. However, consistent with the need for improved patient safety and health care quality, SSS has gained an intractable foothold in American medical Educcation that can only grow with time. The medical community will have to find a way to absorb the cost, personnel, physical facility, and time requirements of SSS, all of which will increase as the scope of SSS expands. Training for, and evaluation of competency in handling highstakes surgical situations will continue to need SSS.