Assessment of Anaesthesia Teams' Non-Technical Skills in Clinical Practice before and after Simulation-Based Team Training: A Quasiexperimental Study

Background In situ simulation-based team training of non-technical skills is considered an important initiative for preventing adverse events caused by poor teamwork among healthcare personnel. This study aimed to assess the non-technical skills of anaesthesia personnel before and after in situ simulation-based team training in a clinical setting. Methods The study was conducted from October 2020 to June 2021 using a quasiexperimental before and after design based on video-recorded observations and ratings of anaesthesia teams' non-technical skills during anaesthesia induction in the operating room before and shortly after in situ simulation-based team training. Anaesthesia personnel were divided into 20 teams and video recorded during anaesthesia induction. The Anaesthetists' Non-technical Skills (ANTS) system was used to score the teams' non-technical skills. A paired-sample t-test was used to assess the impact of the intervention on the anaesthesia teams' scores on the various ANTS categories. Interrater agreement between the two ANTS raters was assessed using weighted kappa. Results At the category level, the overall scores had a statistically significant increase in performance after simulation-based team training (3.48 vs. 3.71; p < 0.001). Furthermore, scores of five of the 15 elements were significantly different. Interrater agreement revealed moderate agreement between the two raters (weighted kappa = 0.51, p value <0.001). Conclusion The anaesthesia teams' increased non-technical skills after simulation-based team training may indicate the transfer of knowledge from training to clinical practice. The moderate agreement between the raters could be attributed to the subjective nature of the evaluation procedure. The ANTS was originally used as an individual assessment tool; however, this study has demonstrated its potential as a team assessment tool.

Background.In situ simulation-based team training of non-technical skills is considered an important initiative for preventing adverse events caused by poor teamwork among healthcare personnel.Tis study aimed to assess the non-technical skills of anaesthesia personnel before and after in situ simulation-based team training in a clinical setting.Methods.Te study was conducted from October 2020 to June 2021 using a quasiexperimental before and after design based on video-recorded observations and ratings of anaesthesia teams' non-technical skills during anaesthesia induction in the operating room before and shortly after in situ simulation-based team training.Anaesthesia personnel were divided into 20 teams and video recorded during anaesthesia induction.Te Anaesthetists' Non-technical Skills (ANTS) system was used to score the teams' non-technical skills.A paired-sample t-test was used to assess the impact of the intervention on the anaesthesia teams' scores on the various ANTS categories.Interrater agreement between the two ANTS raters was assessed using weighted kappa.Results.At the category level, the overall scores had a statistically signifcant increase in performance after simulation-based team training (3.48 vs. 3.71; p < 0.001).Furthermore, scores of fve of the 15 elements were signifcantly diferent.Interrater agreement revealed moderate agreement between the two raters (weighted kappa = 0.51, p value <0.001).Conclusion.Te anaesthesia teams' increased nontechnical skills after simulation-based team training may indicate the transfer of knowledge from training to clinical practice.Te moderate agreement between the raters could be attributed to the subjective nature of the evaluation procedure.Te ANTS was originally used as an individual assessment tool; however, this study has demonstrated its potential as a team assessment tool.

Background
In hospitals, teamwork can cause intraoperative errors, adverse patient outcomes, and mortality.Non-technical skills are important in prevention of adverse events [1,2].Non-technical skills are defned as "the cognitive, social, and personal resource skills that complement technical skills" [3].Te World Health Organization (WHO) patient safety action plan 2021-2030 deems the integration of patient safety competencies including non-technical skills, in simulation-based team training to be critical [4].A metaanalysis [5] stated that there is substantial evidence for simulation-based education to improve non-technical skills.However, there is still a lack of research regarding transfer of learning to anaesthesia teams' clinical practice.
High-fdelity human patient simulation was introduced in healthcare by anaesthesiologist David Gaba [6], with anaesthesia personnel as pioneers in the implementation of human factor-based simulation-based team training [7,8].Among anaesthesia personnel, simulation-based team training including task management, teamwork, situation awareness, and decision making prepares teams to manage challenging situations and is considered essential for safe clinical practice [9,10].
In team training involving anaesthesia personnel, improved team performance [11,12], cultural attitudes, and communication [13] after simulation-based team training of non-technical skills have been reported.A recent study by Finstad et al. exploring anaesthesia personnel's experience from interprofessional in situ simulation-based team training of non-technical skills reported that training contributed to enhanced anaesthesia practice [14].Te study focused on transfer of skills learnt from simulation-based team training to clinical practice with respect to level one (trainee reactions), level two (trainee learning), and level three (transfer of learning to clinical practice) of Kirkpatrick's four-level model [14,15].As the research so far has been mainly focused on Kirkpatrick levels one (reaction) and two (learning), further research on levels three (behaviour) and four (results) is needed [16].A systematic review concluded that further research on the retention and transfer of human factors including non-technical skills from simulation-based team training to clinical practice is essential to characterize its efects on patient safety [17].
Boet et al. concluded in their systematic review, with presentation of assessment tools to assess teamwork, that there is a need for tools to assess intraoperative team performance and that selection of tools depend on the specifc context [18].
Fletcher et al. highlighted the importance of both technical and non-technical skills in all circumstances that arise during diferent healthcare procedures.In addition, they introduced the Anaesthetists' Non-technical Skills (ANTS) system as a framework for both observing and rating non-technical skills for development of overall good practices in anaesthesia [19,20].Te ANTS system is a tool designed for experienced anaesthetists to rate non-technical skills of another anaesthetist with at least basic technical competence [21].Table 1 provides an overview and description of the ANTS system categories.Te framework was introduced in aviation for assessing non-technical skills of pilots [22], and adaption to the health sciences led to behavioural rating systems for several professionals including anaesthetists [19], nurse anaesthetists [23,24], scrub practitioners [25], and surgeons [26].Even though the ANTS is used extensively [27][28][29][30], it is limited to skills that can be identifed through observable behaviour.
Despite widespread implementation of simulation-based team training in anaesthesia, we do not know to what extent non-technical skills in anaesthesia teams is learned during this training.Assessment and feedback using the ANTS system can be used in both clinical and simulation environments to identify strengths and weakness [19].Te system is used to evaluate performance and is an indicator of patient safety [31].Moreover, it is used when anaesthesia is normally administered in the operating room.
Assessment of non-technical skills during anaesthesia induction in clinical practice before and after simulationbased team training could help in evaluation of the transfer of learning from simulation to clinical practice [14,17].Terefore, this study aimed to assess the non-technical skills of anaesthesia personnel before and after in situ simulationbased team training in a clinical setting.

Design.
Te study used a quasiexperimental before and after design [32] using video recordings of the anaesthesia teams' non-technical skills during anaesthesia induction in the operating room.All videos showed two anaesthesia personnel working with other members of the surgical team.Overall, 20 videos before and 20 videos after simulationbased team training were viewed and evaluated by experienced nurse anaesthetists.

Setting and Sample.
Te impact of simulation-based team training was assessed during anaesthesia induction with endotracheal intubation in patients undergoing earnose-throat surgery.A total of seven anaesthesiologists and nine nurse anaesthetists (total n � 16) employed in a surgical department of a university hospital in Norway were recruited for the study.Nearly all participants had substantial work experience but varying experience with simulation training.Te participants were paired into 20 diferent teams consisting of either an anaesthesiologist and a nurse anaesthetist (n � 17) or two nurse anaesthetists (n � 3).Te same teams were assessed before and after simulation-based team training.

In Situ Simulation-Based Team Training Programme.
Te in situ simulation-based team training of non-technical skills was developed based on categories in the ANTS system (Table 1) to defne learning objectives and scenario (Table 2).Te simulation-based team training programme was structured based on the simulation setting model by Peter Dieckmann [33,34] (Table 3).4, Figure 1).Te video recordings were evaluated (box 2 and 5, Figure 1) by two experienced nurse anaesthetists with respect to non-technical skills based on the framework for observing and rating tool, ANTS system [20].Figure 1 shows an overview of the study procedure.

Instrument.
In this study, the non-technical skills of anaesthesia personnel with respect to teamwork were assessed using the ANTS System Handbook [20].Te ANTS  [20].
Task management: Skills for organizing resources and required activities to achieve goals including individual case plans or longer-term scheduling issues Team working: Skills for working in a group, in any role, to ensure efective completion of collaborative tasks and team member satisfaction; the focus is particularly on the team rather than the task  2.4.2.Procedure.Two experienced nurse anaesthetists familiar with the language and structure of the ANTS system participated as raters [20].In the preparation phase, the two raters underwent training in rating organised and conducted by the frst author (ASF) to become familiar with the instrument and to increase the non-technical skills rating agreement between themselves.Video fles from a pilot test in clinical practice and video recordings from anaesthesia students' simulation settings [36] were used for rater training.After watching the training videos, each of the two raters individually rated each of the two team members-the nurse anaesthetist and the anaesthesiologist-and then jointly re-evaluated their ratings to develop a common understanding of the rating procedure.After the training, the two raters independently viewed each of the 20 before and 20 after videos of the anaesthesia team during anaesthesia induction, rating each team member individually according to the non-technical skills using the ANTS system framework.Team scores were calculated as the mean of the two individual team members' scores, as the ANTS system was not developed for assessing an entire anaesthesia team but only the anaesthesiologist, one of the two team members.

Data Analyses.
For each of the 16 participants in the study, the scores for each of the 15 skill elements were calculated as the mean of the two raters, while the team score was calculated as the mean of the two team members.When an element for one team member was not observed, the observed team member's score represented the team score.One non-technical skills' score for one team was missing.
Te analyses did not include the performance of the present operating room nurses.
Te overall ANTS scores were summarised as the mean (SD) across all teams.Te scores before and after simulationbased team training were compared using paired-sample ttests.Statistical signifcance was set at p < 0.05.
Te study adheres to the Transparent Reporting of Evaluations with Nonrandomized Designs (TRENDs) guidelines [40].

Anaesthetists' Non-Technical Skills' Scores.
Te overall mean (SD) ANTS team scores before and after simulationbased team training were 3.48 (0.56) and 3.71 (0.45), respectively (p � < 0.001), with a corresponding mean (95% CI) increase in ANTS scores of 0.23 (0.16-0.30) (Table 6).At the category level, the mean (SD) ANTS team scores had a statistically signifcant increase after the simulation-based team training intervention in three of the four categories.At the element level, the mean (SD) team ANTS scores had a statistically signifcant increase in fve out of 15 elements (Table 7).While not statistically signifcant, the scores increased slight increases also in fve other elements after simulation-based team training (Table 7 and Figure 2).

Interrater Reliability.
Assessment of the agreement between the two raters yielded a weighted kappa value of 0.51, indicating moderate agreement.

Discussion
Te overall ANTS team score had a statistically signifcant increase after simulation-based team training as did three of the four ANTS categories and fve out of 15 ANTS elements.

Task Management.
In the task management category, only the element "providing and maintaining standards" had a statistically signifcant increase in score after simulationbased team training, which may indicate that this element was more observable than the elements with minor increase or even decrease in performance.Kirkpatrick  4 Anesthesiology Research and Practice [15]."Providing and maintaining standards" concerns safety and quality with measures including following protocols, cross-checking medication, checking machines, and maintaining anaesthesia journal.Tese behaviour markers may provide insights into elements of "planning and preparing" including preparing drugs and equipment.Te elements are connected as they may have certain overlap with each other [20].Moreover, the element "planning and preparing" was  Anesthesiology Research and Practice  [41].Tis may explain the possible diference between measurement in simulation and clinical practice.Nevertheless, our results reveal a minor increase in task management, indicating transfer of learning from simulation to clinical practice.

Team Working.
Te teamwork category had a statistically signifcant improvement in scores after simulationbased team training with signifcant improvement in the elements "exchanging information" and "using authority and assertiveness."Tese elements are crucial for team coordination and task completion (e.g., updating, confrming understanding, and maintaining documentation) and team leading and clear communication (e.g., state cases and providing justifcation) [20].Tis improvement may confrm that teamwork is an important category for simulation-based team training for transfer of learning to clinical practice [42].Even though the elements "coordinating activities with team members" and "supporting others" only had non-signifcant increase in performance, these elements may contribute to the improvement in the category with the other elements.No behavioural markers were observable in the element "assessing capabilities," which may be due to the situation and behaviour observability [15].Moreover, four of the fve elements had team improvement, despite statistically signifcant increase in just two elements.However, we suggest that rater training should be prioritized beforehand, as the score system may seem simple during training but difcult in practice [41].

Situation Awareness.
Te team scores concerning situation awareness had statistically signifcant improvement at the category level and in the element "gathering information," referring to collecting data about the situation by observing the environment including measures such as frequently scanning and cross-checking patient, surgery, and team information.Tis category is essential for patient safety [43] and to understand the observed aspects and predictions based on these observations [20].Te elements "recognizing and understanding" and "anticipating" had minor improvement in scores and contribute to the improvement in the scores of the category.Tis category's correlation to the increased teamwork scores is not surprising as these skills are likely overlapping.Focusing on these essential skills in the simulation-based team training may have contributed to improved performance in clinical practice.

Decision
Making.Tis category concerns judgement for further action and the team scores had statistically signifcant increase in this category and the element "reevaluating," refereeing to continuous review of identifed options, reassessment of diferent parameters including Anesthesiology Research and Practice patient characteristics, situation, conditions evolved, and efectuated action [20].No change was observed in the element "identifying options" with maximum scores before and after simulation-based team training.Tis element may be the strength of this category although it did not result in a statistically signifcant increase in performance."Balancing risk and selecting options" was not observable most likely due to challenging observability [15] and/or may be the fact that some skills may take longer time to learn than was allowed in this study.Re-evaluation could be essential for all other categories concerning anaesthesia personnel's control of the situation, cooperation in the team, and task management for optimal patient care and safety.

In Situ Simulation-Based Team Training.
A metaanalysis has shown that simulation-based education is favourable for improving non-technical skills [5].Yet, simulation-based team training of non-technical skills for anaesthesia personnel with a view to transfer of learning to clinical practice seems to be lacking.Terefore, the results of our study might contribute to future simulation-based team training.In our study, the ANTS system categories [20] were used to defne learning objectives for the in situ simulationbased team training program.To a certain extent, learning objectives were achieved with respect to calculated team score assessment in clinical practice after simulation-based team training.In situ simulation-based team training seems to provide the greatest opportunity to achieve improvement goals in team functioning [44].Moreover, a single in situ simulation-based team training of the present learning objectives could be sufcient for certain goals.More frequent simulation-based team training could be needed to obtain even more transfer of learning to clinical practice, as recently reported [14].Te elements in ANTS may also be modifed for specifc clinical situations, which may have contributed to the statistical signifcant results in our study.Tis can be implemented for training but not for research, as the conditions can change suddenly [45].
4.6.ANTS System as a Team Score Instrument.All 15 ANTS elements were scored in this study.Te "not observable" scores could be due to various factors including videotechnical issues, difculty of the raters to observe the real situation, or the subjective nature of the evaluation.Te results depend on the noticeability and measurability of the behaviour [15], and a suitable assessment tool may be crucial.Abildgren et al.'s systematic review (2022) reports that non-technical skills including communication and teamwork, could be recognisable, and have valid interpretations for training personnel, but these elements may be difcult for the raters to assess.Rating non-technical skills is easier for passive observers, especially using prerecorded videos [17].Tis observation agrees with the study by Ballangrud et al., describing self-report as important in assessing unobservable components [46].
Te ANTS system has a high level of acceptability and a reasonable level of reliability [19]; in our view, this is a valuable assessment tool for anaesthesia teams in the operating rooms (ORs) even though the team scores were calculated as the mean of two individual scores [9,18,28].
Some studies suggest the need for development of robust team-based metrics [9,18,47,48] and standards and recommend future research to determine if existing tools are suitable for OR teams [18,47,48].Te ANTS system has been suggested as basis for developing a team assessment tool in intensive care units [28].Tis could also be considered for anaesthesia teams, as ANTS was originally developed for assessing individual anaesthetists.Boet et al. concluded that a number of team assessment tools are available for diferent teams but suggest more research into the topic of developing tools for intraoperative crisis situations, where anaesthesia teams are an integral part.Hence, we currently consider ANTS, and corresponding team scores calculated from individual assessments, to be a promising tool for team assessment of anaesthesia teams in clinical practice.
If the learning gained from simulation-based team training cannot be transferred to clinical practice, good reaction (level 1) or increased knowledge (level 2) in Kirkpatrick's evaluation model [15] are not relevant.Our results revealed increased performance in all the ANTS system categories and certain elements.Tus, the study revealed learning at level 3 in Kirkpatrick's model.Moreover, the study attempted to describe the change in clinical practice of experienced anaesthesia personnel following simulation-based team training.Teir extended experience may have contributed to the moderate efect of simulation.
Although we used a team score calculated as the mean of two individual scores, further research is needed to develop a suitable team assessment tool based on the ANTS system.

Strengths and Limitations.
Te study had several strengths including the uniform team composition before and after simulation-based team training.Moreover, video recordings were used instead of raters being present in the OR, which helped to avoid disturbing the participants in the clinical setting.Furthermore, the raters could observe the video recordings multiple times.
However, our study has some limitations.Te weighted kappa, a marker of rater agreement, showed only a moderate strength.Due to ethical approval and participants' consent in our study, the video recordings were deleted after three weeks, thereby limiting the raters available assessment' time.Tis could afect the rating owing to rater stress and fatigue [36].Te raters were aware whether they were rating before or after videos, which may be a weakness according to expectation of improved result after simulation-based team training.Te raters decided their own training time.Terefore, the training period may vary from individual to individual and the readiness of each rater.Despite using a validated instrument, our participants were experienced clinicians which may have contributed to a ceiling efect with high score both before and after the simulation-based team training.Tis could also be the cause for the small variance in our data.ANTS is an individual assessment tool, and we calculated the team score from the mean scores of the two individuals.A team assessment tool suitable for the anaesthesia team would be preferable.However, since such a tool does not yet exist, the ANTS system was the best option available.Tus, good team performance could be achieved not only by one person performing well but also by the strength of the team itself.

Conclusion
Te performance of anaesthesia teams improved in clinical practice after in situ simulation-based team training.Tis may indicate transfer of learning from simulation-based team training to clinical practice and aligns with level 3 (transfer of learning to clinical setting) in Kirkpatrick's evaluation model.Te ANTS system categories may be helpful to prepare anaesthesia teams to cope with challenging situations and are essential for safe clinical practice.After simulation-based team training, the three categories, team working, situation awareness, and decision making, had statistical signifcant improvement due to the elements "providing and maintaining standards," exchanging information", "using authority and assertiveness," gathering information" and "re-evaluating."Observability may be low for certain elements that require more attention in future rater training; hence, there is a need to develop a more specifc rating tool for team performance of NTS for intraoperative anaesthesia teams.

2 Anesthesiology Research and Practice 2 . 4 .
Te in situ simulation-based team training programme was conducted in March and April 2021.Te anaesthesia team (one anaesthesiologist and one nurse anaesthetist) on duty participated in the simulation-based team training with the simulated case of anaesthesia induction.Te focus was only on the anaesthesia team, but two operating room nurses participated as support during the anaesthesia induction simulation to create realism.Before the simulation-based team training, the participants received an information leafet regarding healthcare simulations and non-technical skills.Each simulation-based team training session lasted for one hour.Simulation-based team training scenarios were led by an educated facilitator (ASF) in an operating room.Te facilitator directed the debriefng and concluded with a summary and evaluation.Data Collection.Video recordings of the anaesthesia team in clinical practice in the operating room were conducted before and after in situ simulation-based team training.Overall, 20 videos were recorded before the simulation-based team training and 20 videos were recorded after the simulation-based team training (box 1 and Situation awareness: Skills for developing and maintaining an overall awareness of the work setting based on observing all relevant aspects of the theatre environment (patient, team, time, displays, and equipment), understanding what they mean, and thinking ahead about what could happen next Decision making: Skills for selecting a course of action or diagnosis, in both normal conditions and in time-pressured crisis situations Permission provided by the copyright holder.

Figure 2 :
Figure 2: Radar diagram of ANTS team scores for each element before and after simulation-based team training.

Table 1 :
Te ANTS system categories

Table 2 :
In situ simulation-based team training programme's learning objectives and simulation scenario.
(ii) Te participants perform teamwork by displaying behaviour: cooperating and communicating with team members, using authority and assertiveness, assessing capability, and supporting others (iii) Te participants achieve situation awareness by displaying behaviour according to: gathering information, recognizing the environment, understanding what happens, and being anticipatory (iv) Te participants perform decision making by displaying behaviour according to: identifying options, balancing risks and selecting options, and re-evaluating Simulation-based team training scenario Anaesthesia induction of elective patient for operation: septoplasty.Te nurse anaesthetist prepares for anaesthesia induction: electronic documentation, monitoring equipment, anaesthesia machine and other required equipment, patient (simulated patient) preparations.Te anaesthesiologist arrives in the operation room and coordinates with the nurse anaesthetist and patient.Te anaesthesia induction starts (a manikin is used for intubation).Te scenario ends when the endotracheal tube is inserted and its' position verifed
and Kirkpatrick claimed that observation assessment depends on the noticeability and measurability of the behaviour markers

Table 5 :
[20]ples of ANTS elements with behaviour markers for good and poor practice[20].

Table 6 :
Anaesthetists' ANTS category scores for teams before and after simulation-based team training.
Permission provided by the copyright holder.

Table 7 :
Anaesthetists' ANTS element scores for teams before and after simulation-based team training.