ACCURACY AND USEFULNESS IN ASSESSING PROFICIENCY OF THE OBSERVATIONAL CLINICAL HUMAN RELIABILITY ASSESSMENT CHECKLIST OF THE OPEN INGUINAL HERNIA REPAIR PROCEDURE: A CROSS-SECTIONAL STUDY.

BACKGROUND
The Observational Clinical Human Reliability Assessment (OCHRA) can be used to score errors during surgical procedures. To construct an OCHRA-checklist, steps, substeps, and hazards of a surgical procedure need to be defined. A step-by-step framework was developed to segment surgical procedures into steps, substeps, and hazards. The first aim of this study was to investigate if the step-by-step framework could be used to construct an accurate Lichtenstein open inguinal hernia repair (LOIHR) stepwise description. The second aim was to investigate if the OCHRA-checklist based on this stepwise description was accurate and useful for surgical training and assessment.


MATERIALS AND METHODS
Ten expert surgeons rated statements regarding the accuracy of the LOIHR stepwise description, the accuracy, and the usefulness of the LOIHR OCHRA-checklist (eight, seven, and six statements, respectively) using a 5-point Likert scale. One-sample Wilcoxon signed-rank test was used to compare the outcomes to the neutral value of 3.


RESULTS
The accuracy of the stepwise description and the accuracy and usefulness of the OCHRA-checklist were rated statistically significantly higher than the neutral value of 3 (median 4.75 [5.00-4.00] with p=.009, median 5.00 [5.00-4.00] with p=.012, median 4.00 [5.00-4.00] with p=.047, respectively). The experts rated the OCHRA-checklist to be useful for the training (5.00 [5.00-4.00], p=.009), and assessment (4.50 [5.00-4.00], p=.010) of surgical residents.


CONCLUSION
This preliminary study showed that the stepwise LOIHR description constructed using the step-by-step framework was found to be accurate. The LOIHR OCHRA-checklist developed using the stepwise description was also accurate, and particularly useful for the training and assessment of proficiency of surgical residents.


Introduction
Adverse events are frequent within the surgical field. A systematic review reported surgical adverse events in approximately 14% of patients, which were potentially preventable in more than one-third of cases [1]. These adverse events are mainly due to human errors. To specifically assess surgical errors during surgical procedures, the Observational Clinical Human Reliability Assessment (OCHRA) was developed [2]. The OCHRA distinguishes executional and procedural errors [3]. Executional error concerns technical execution, for example, a skin incision placed at an incorrect location, or an incision created too long or too deep. Procedural errors concern actions during surgery, which are wrongfully not performed, partially performed, or done out of sequence [3].
For the development of a surgical procedure-specific OCHRAchecklist, the surgical procedure of choice needs to be segmented into steps and substeps in the order considered ideal for perfect execution, while potential hazards need to be identified [4]. Currently, the construction of an OCHRA-checklist is a time-consuming effort using historical technical protocols [2,5,6], expert panels [7][8][9], and textbooks and literature combined with a thorough video-analysis of the surgical procedure [10]. Typically, expert panels are also used for the identification of hazards [4]. Although extensive research concerning the usefulness of the OCHRA for the assessment of surgical trainees has been conducted [2,3,5,7,9,[11][12][13][14], the OCHRA-checklist is currently not widely used yet. The unavailability of an efficient method to segment surgical procedures might be hampering the broad implementation of the OCHRA within the surgical field. To make this process potentially more efficient, a standardized step-by-step framework has been developed to break down surgical procedures into steps and substeps with the identification of hazards [15]. A step is defined as a surgical goal that needs to be achieved and evaluated before proceeding onto the next step. Each step consists of one or more substep(s), which are based on anatomical structures or implants. The step-by-step framework used in this study allows segmentation of every surgical procedure of choice into steps and substeps in a standardized and comprehensive manner, without the need of an expert panel or other time-consuming efforts. Since the stepwise description of a surgical procedure and the OCHRA-checklist are based on steps and substeps, a surgical procedure-specific OCHRA-checklist can then be effortlessly established using the step-by-step framework.
The first aim of this study was to investigate if the step-by-step framework can be used to construct an accurate stepwise description of a surgical procedure, including its hazards. The second aim was to investigate if the developed LOIHR OCHRA-checklist based on this stepwise description was accurate and useful for surgical training and assessment of medical students, surgical residents, and surgical experts. The Lichtenstein open inguinal hernia repair (LOIHR) was used as an example surgical procedure in this study as it is a common procedure for training residents containing multiple steps with significant errors. A simulation model was used in this study to resemble the standard anatomy and pathology for the LOIHR to assess the stepwise description and the OCHRA-checklist in a standardized environment.

Stepwise description
The stepwise description of the LOIHR was constructed according to the step-by-step framework [15]. Under the direct supervision of a surgical expert, a medical doctor (MD) constructed a standardized stepwise description based on literature and available evidence-based guidelines [16][17][18][19][20]. This process for the LOIHR stepwise description consumed approximately 8 h in total. The surgical procedure concerned an indirect inguinal hernia repair (see Table, Supplemental Digital Content 1). The LOIHR stepwise description was additionally visualized in 8:00 min step-by-step video-demonstration of the surgical procedure being performed on an open inguinal hernia simulation model.

2.2 OCHRA-checklist
The OCHRA-checklist was constructed using the components of the stepwise description of the LOIHR (see Table, Supplemental Digital Content 2). A sample of the first three steps of the LOIHR OCHRAchecklist is shown in Fig. 1. The first column shows the steps of the LOIHR stepwise description, the second column shows the substeps based on the anatomical structures, and the third column describes the actions to be performed on these anatomical structures. The correct performance of a substep can be documented in the fourth column. Executional and procedural errors can be listed in the fifth and sixth columns, respectively. The hazards are stated in the final column.

2.3 Participants and design
Ten international hernia expert surgeons with significant surgical and research experience on the LOIHR were invited per email. Inclusion criteria was extensive experience in performing (more than 1000 surgical procedures) and/or researching the open inguinal hernia repair (more than 5 papers). Participants were excluded when they did not complete the survey. After confirmation of participation, an instruction letter was sent, including their login credentials to a website where they could view the stepwise description of the LOIHR, the step-by-step video-demonstration, and the OCHRA-checklist. The experts were then requested to assess the stepwise description and the OCHRA-checklist by rating statements. This study has been reported in line with the STROCSS criteria [21].

Rating of statements
Three categories of statements were made; 1. accuracy of the LOIHR stepwise description, 2. accuracy of the LOIHR OCHRA-checklist, and 3. usefulness of the LOIHR OCHRA-checklist. First, the accuracy of the LOIHR stepwise description was rated using eight statements regarding the procedure (steps, substeps and hazards). The statements regarding the accuracy of the stepwise description included two control statements ('Steps of the open inguinal hernia repair are missing' and 'Hazards of the open inguinal hernia repair are missing'). Second, the accuracy of the OCHRA-checklist was rated using seven statements. Third, the usefulness of the OCHRA-checklist for surgical training and assessment of medical students, residents, and experts was rated using six statements. All the statements were rated on a 5-point Likert-scale, varying from 1 = totally disagree to 5 = totally agree, with 3 = neutral.

Statistical analysis
The median and interquartile ranges (IQR) of the rated statements were analyzed due to the skewness of the data. One-sample Wilcoxon signed-rank test was used to compare the median of the statements to the neutral value of 3. A p-value of < .05 was considered statistically significant.
Cronbach's α was used to determine the internal consistency of each category: accuracy of the stepwise description (8 items), the accuracy of the OCHRA-checklist (7 items), and usefulness of the OCHRA-checklist (6 items). A Cronbach's α from 0.70 to 0.95 indicated an acceptable internal consistency [22]. The analyses were performed using SPSS® version 24.0 (IBM, Armonk, New York, USA).

Results
Ten surgeons from seven different countries and three different continents participated in this study ( Table 1). The average age of the expert surgeons was 55 years old (range 37-69). Eight expert surgeons had more than 20 years of post-residency experience, one surgeon had 10-20 years, and one surgeon had up to 10 years of post-residency experience. Five of the ten experts have performed individually more than 3000 open inguinal hernia repairs and have published individually more than 50 hernia-related papers.
The accuracy of the LOIHR stepwise description, as shown in Table 2, was rated statistically significantly higher than the neutral value of 3  Table 3, was rated statistically significantly higher than the neutral value of 3 (median 5.00 [5.00-4.00], Z = 2.63, p = .009), with an internal consistency of Cronbach's α = 0.960. The individual statements in this category were all rated statistically significantly higher than the neutral value of 3.
The category regarding the usefulness of the LOIHR OCHRAchecklist for training and assessment, as shown in Table 3, was rated statistically significantly higher than the neutral value of 3 (median 4.00

Discussion
The step-by-step framework is a theoretical model to break down surgical procedures into steps and substeps in a standardized manner. In this study, a stepwise description of the LOIHR was constructed in a  1 How many hernia-related papers did you publish in total? (n) The step-by-step description is a good basis for the OCHRA assessment 5.00 5.00-4.00 .009 † IQR interquartile range (Q3 -Q1); *analyzed using one-sample Wilcoxon signed-rank test, compared to a neutral value of 3; y statistically significant.
relatively short time using the step-by-step framework. The expert hernia surgeons highly rated the accuracy of the LOIHR stepwise description. Subsequently, an OCHRA-checklist was composed using the LOIHR stepwise description. This LOIHR OCHRA-checklist was found to be accurate and useful for surgical training, assessment, and monitoring of proficiency gain, particularly for surgical residents.

Stepwise description
Previous studies described other methods to segment surgical procedures into steps and substeps, for example using hierarchical task analysis (HTA) based on historical technical protocols [2,5,6], experts or expert panels [7][8][9], or HTA performed by research groups [3,14]. These methods might result in a potentially more detailed description of surgical procedures, but these methods can be time-consuming and logistically challenging. For example, Sarker et al. described an open inguinal hernia repair description using textbooks, articles, and video-analyses of the surgical procedure to draft an initial surgical procedure description [10]. Additionally, an anesthetic expert and a scrub nurse task analysis was performed and combined with this initial description. Finally, expert surgeons refined the surgical procedure description. In comparison to our LOIHR stepwise description, the open inguinal hernia repair description of Sarker et al. consisted of at least 16 tasks (equivalent to steps) in contrast to 6 steps in our LOIHR stepwise description, while the content and order of steps were similar in both descriptions [10].

OCHRA-checklist
The developed LOIHR OCHRA-checklist was assessed to be useful by the surgical hernia experts, particularly for the training, assessment, and monitoring of the proficiency gain of surgical residents. The expert surgeons assessed the LOIHR OCHRA-checklist not to be useful for surgeons, this is in contrast to the extensive use of the OCHRA-checklists in surgeons for the assessment of laparoscopic surgeries [3,9,23] and monitoring of proficiency gains [14]. In those studies, the OCHRA-checklist was found to provide surgeons objective and complete assessment of their surgical performance [3,12].
A possible explanation that our developed OCHRA-checklist was not found to be useful for surgeons might be due to that our stepwise description and subsequently, developed OCHRA-checklist described a standard approach to perform the LOIHR. Actual surgeries in patients can have variations in anatomy and pathology. In this study, we chose to exclude these variations to provide a basic outlay of the surgical procedure for inexperienced surgical residents. The addition of these variations might enhance the OCHRA-checklist usefulness for surgeons. Furthermore, experienced surgeons perform parts of a surgical procedure semi-automatically without conscious awareness [4]. When expert surgeons are asked to explain the execution of a surgical procedure in detail, it appears considerably tricky for them to identify the decisive moments [24]. The integration of which steps do and do not require conscious awareness, as in a cognitive task analysis (CTA) [25], may also make the OCHRA-checklist more suitable for surgeons.
The expert surgeons highly rated the usefulness for the monitoring of proficiency gain in surgical residents, comparable to a previous study in surgeons [14]. As the OCHRA-checklist allows supervisors to assess a surgical trainee per step of a procedure, insight will be established in which steps need more attention and the proficiency gain can be monitored.

Limitations
We acknowledge that the absence of an expert panel can be a potential weakness of the step-by-step framework, which may lead to a less detailed stepwise description. Nonetheless, the step-by-step framework provides a clear method to segment all surgical procedures in a standardized, comprehensive and time-efficient manner into steps, substeps and to identify hazards. The great advantage of the step-by-step framework is that more surgical procedures can be segmented efficiently, which may facilitate the implementation of the OCHRAchecklist more widely. A second limitation in our study was the use of a static simulation model to demonstrate a standardized LOIHR for an indirect hernia. This simulation model did not include any anatomical and pathological variations, such as adhesions due to previous surgeries, obesity, or sliding hernia.

Future perspectives
The OCHRA checklist could be used for feedback to facilitate the learning curve. Based on the checklist, the proficiency of the resident can be evaluated and measured over time. The supervisor can decide to include more difficult cases over time and continue to assess the proficiency with the OCHRA method.
To further improve the stepwise descriptions and associated OCHRAchecklists, a system could be developed to continuously integrate clinically encountered anatomical and pathological variations of the patient during the surgical procedure. This system could also use clinical postoperative patient outcomes to improve the hazards in the stepwise description and OCHRA-checklist. If the postoperative adverse events were caused during surgery, these could be implemented as new hazards. In previous studies, the OCHRA was considered to be useful to pinpoint the potential hazard zones for a specific error [8,12,26].
Finally, further research is needed to determine the actual usefulness and compliance of the OCHRA-checklist in the operating room with surgical trainees and their supervisors. Also, research concerning the comparison of the effects, usefulness, and compliance between the OCHRA-checklist and other surgical assessment tools, such as the Objective Structured Assessment of Technical Skills (OSATS) or the The OCHRA checklist is useful in the training of a surgical resident 5.00 5.00-4.00 .009 † IQR interquartile range (Q3 -Q1); *analyzed using one-sample Wilcoxon signed-rank test, compared to a neutral value of 3; † statistically significant.
Surgical Quality Assurance (SQA) should be carried out [27,28]. We are testing this hypothesis in a next study.

Conclusion
In summary, the step-by-step framework was used to construct a stepwise description for the open inguinal hernia repair and OCHRAchecklist. The international experts highly rated the accuracy of the stepwise description, and the accuracy and usefulness of the OCHRAchecklist. The OCHRA-checklist was found to be particularly useful for surgical residents in terms of training, assessment, and monitoring of proficiency gain.

Ethical approval
Approval from the institutional review board was not required for this study (based on anonymous questionnaires to surgeons).

Sources of funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author contribution
T. Nazari: study design, data collections, data analysis, writing. M.P. Simons

Research data
We state that our research date will be made available on request.

Informed consent
For this type of study, formal consent was not necessary.

Provenance and peer review
Not commissioned, externally peer-reviewed.

Declaration of competing interest
Authors declare no conflict of interest.