A cost-effectiveness analysis of self-debriefing versus instructor debriefing for simulated crises in perioperative medicine in Canada

Purpose High-fidelity simulation training is effective for learning crisis resource management (CRM) skills, but cost is a major barrier to implementing high-fidelity simulation training into the curriculum. The aim of this study was to examine the cost-effectiveness of self-debriefing and traditional instructor debriefing in CRM training programs and to calculate the minimum willingness-to-pay (WTP) value when one debriefing type becomes more cost-effective than the other. Methods This study used previous data from a randomized controlled trial involving 50 anesthesiology residents in Canada. Each participant managed a pretest crisis scenario. Participants who were randomized to self-debrief used the video of their pretest scenario with no instructor present during their debriefing. Participants from the control group were debriefed by a trained instructor using the video of their pretest scenario. Participants individually managed a post-test simulated crisis scenario. We compared the cost and effectiveness of self-debriefing versus instructor debriefing using net benefit regression. The cost-effectiveness estimate was reported as the incremental net benefit and the uncertainty was presented using a cost-effectiveness acceptability curve. Results Self-debriefing costs less than instructor debriefing. As the WTP increased, the probability that self-debriefing would be cost-effective decreased. With a WTP ≤Can$200, the self-debriefing program was cost-effective. However, when effectiveness was priced higher than cost-savings and with a WTP >Can$300, instructor debriefing was the preferred alternative. Conclusion With a lower WTP (≤Can$200), self-debriefing was cost-effective in CRM simulation training when compared to instructor debriefing. This study provides evidence regarding cost-effectiveness that will inform decision-makers and clinical educators in their decision-making process, and may help to optimize resource allocation in education.


Acknowledgement
The ANTS System was developed and evaluated in a collaborative project between the University of Aberdeen Industrial Psychology Research Centre and the Scottish Clinical Simulation Centre. The first phase of the study (1999)(2000)(2001)(2002)(2003) was funded by NHS Education for Scotland. We particularly wish to acknowledge the significant contribution made by Georgina Fletcher to the design of this system.
The copyright of this publication is owned by the University of Aberdeen. It may be photocopied or electronically reproduced by downloading from the website without further permission for personal, organisational or 'not for profit' use. No reproduction by or for commercial organisations is permitted without the express permission of the copyright holder.

Background
The training programme in Anaesthesia has been developed to help trainees acquire the necessary knowledge, skills and values that will enable them to meet the challenges of consultant practice. The most recent major development in the UK has been the introduction of the competency based training scheme, which recommends that progress through and completion of training be based on competence. This in turn places the emphasis on teaching and assessment in the workplace, and is encouraging greater emphasis on those components necessary to provide effective management of patients. The competency based approach can be thought of in terms of not only acquiring the individual components but being able to integrate them effectively in providing solutions to clinical challenges. Another important development in medical education has been the increasing recognition of the importance of reflection in the training of professionals. In a time of reduced working hours and exposure to fewer clinical challenges, it is important that clinicians, both in training and career grades, make the most of their clinical experience. Feedback on strengths and weaknesses or self-reflection are more likely to be effective when there is a terminology or vocabulary that permits analysis of performance. The tool described in this booklet addresses the area of non-technical skills. It provides both a framework and common terminology that allows anaesthetists to communicate effectively with each other in this area of practice, helping trainees (and others) develop abilities in both the real workplace or simulated work environment.
This handbook provides a condensed guide to the Anaesthetists' Non-Technical Skills System and includes suggestions on how the system can be used. Part 1: Information for Users is written in the form of answers to frequently asked questions. These have been based on research for the ANTS project and literature on the use of behavioural marker systems to support non-technical skills training. Part 2: System Details provides the full contents of the ANTS System, the rating scale, and the rating form. Further information and rating forms can be found at the ANTS website: www.abdn.ac.uk/iprc/ANTS

Part 1 Information for users
What are Non-Technical Skills? As in other industrial domains, accidents and incidents in anaesthesia are usually caused by a combination of organisational and operational factors. Investigations into adverse events or 'human errors' have shown that as many as 80% of them are the result of human factors breakdowns such as poor communication, inadequate monitoring, failures to cross-check drugs and equipment, rather than lack of technical knowledge or equipment problems. Research observing medical teams in operating theatres has also highlighted difficulties arising from loss of situation awareness and poor team interactions*.
Reducing the likelihood of such problems requires anaesthetists to have an additional set of skills, known as non-technical skills, that are used integrally with medical knowledge and clinical techniques. These non-technical skills can be defined as behaviours in the operating theatre environment not directly related to the use of medical expertise, drugs or equipment. They encompass both interpersonal skills e.g. communication, team working, leadership, and cognitive skills e.g. situation awareness, decision making. Such skills are not new in anaesthesia, good anaesthetists have always demonstrated these competencies. In the past, these skills have not been explicitly addressed through any formal education and trainees have had to acquire them along the way. However, with an increased focus on reducing adverse events and the introduction of competency based training and simulators, this is now changing.
To achieve successful non-technical skills training, it is first necessary to identify the requisite skills for the job in the given operational environment and culture. It is also important to be able to assess these skills, to provide feedback about performance and to allow training to be evaluated. To structure the training and assessment of pilots' non-technical skills, the aviation industry uses behavioural marker systems. Behavioural marker systems are empirically derived taxonomies of the principal non-technical skills required for the job, with an observation-based rating system for assessing their component behaviours*. Their explicit nature and the reliance on assessment of observable behaviour by trained instructors means behavioural marker systems can provide a structured tool for making reliable assessments. At a more basic level, they also provide a common language for discussing non-technical skills. Such systems can play an important role in supporting non-technical skills training in anaesthesia, in both simulator and onthe-job training.
It is important to remember that non-technical skills should not be considered in isolation to other aspects of anaesthetic competence. The purpose of examining these skills is to support the development of overall good practice. Successful task performance depends on the effective integration of both technical and non-technical skills for any given situation.
*For more information see the reports and journal articles on the ANTS website

What is the ANTS System?
The Anaesthetists' Non-Technical Skills (ANTS) System is a behavioural marker system developed by industrial psychologists and anaesthetists during a four year collaborative research project in Scotland. Used integrally with medical knowledge and clinical skills, non-technical skills should help to support safe and effective performance in everyday tasks and emergency situations. ANTS describes the main observable non-technical skills associated with good anaesthetic practice. The purpose of the system is to provide the anaesthetic community with a framework for describing non-technical skills and a tool to guide their assessment in an explicit and transparent manner. In short, the ANTS System supplies consultants and trainees with a language for discussing the 'behavioural aspects' of performance. It can be used for assessing an individual's behaviour, to provide input for the training process and for structuring feedback on skills development. Until a fuller understanding of the in-theatre validity and reliability of the ANTS System has been achieved, it is not recommended for formal summative assessment.
The ANTS System comprises a three level hierarchy. At the highest level are four skill categories and beneath these are fifteen skill elements (see table below). Each element has a definition and some examples of good and poor behaviours that could be associated with it. These are the behavioural markers, as they help to indicate the presence or absence of the skill elements. They have been derived from real examples given by consultant anaesthetists during interviews describing their experiences on a variety of cases. The ANTS System is not intended to provide a totally exhaustive list of all non-technical skills used by anaesthetists. It is limited to the principal skills that can actually be identified through observable behaviour. This advice is based on feedback from consultant anaesthetists who trialled the ANTS System with trainees in the operating theatre.

How should I implement the ANTS System?
User selection and training • In order to use the ANTS System effectively, training is required. This should include: -background knowledge on human performance, error management and non-technical skills, so constructive, directive feedback can be given to trainees; -principles of using psychometric tools for rating performance; -the contents of the ANTS System and how they relate to everyday activities; -practice in observing non-technical skills and making ratings with the ANTS System. • If the ANTS System is to be used for assessment, trainers should undergo calibration to ensure that they can provide standardised judgements. • Recurrent training and calibration programmes should be developed. • It is recommended that a small group of consultants is selected in each department to become ANTS trainers/assessors.
Trainee selection and training • Trainees should receive training on human performance and error management to support development of their non-technical skills. In future, this may begin at medical school and then be further developed throughout postgraduate training • Trainees should receive their own copy of this ANTS System Handbook for reference. • The ANTS System should be used appropriately for the level of experience of the trainee: with junior trainees, the focus of training is on developing basic anaesthetic expertise; the ANTS System can be used for general discussion of non-technical skills and their importance to clinical practice; -for more senior trainees, the ANTS System can be used to rate skills and provide feedback during increasingly challenging cases; -towards completion of training, it can also be used to help senior trainees learn how to assess ANTS in others. • Consultants should explain to trainees why it is important to provide feedback on non-technical skills during training, highlighting that the ANTS System has been designed to aid the development of professional skills.

How should the ANTS System be used?
Ratings can be made at both the element and category levels. The recommended method is first to observe performance, making notes of any specific behaviours or omissions. Any assessment should be based only on behaviours observed directly. Using these observations, the rating can then be carried out, first at element level, then at the more general category level. A four point scale is used to describe the level of performance demonstrated (with an option to record skills that were not observed) -see page 14. A copy of the rating form is shown on page 15. Before using the ANTS System for teaching and assessment in training, it is important that you have received training on the system. This should consist of practice in observation and rating, receiving feedback on your scores, and discussion of appropriate use of the system.

5
Time scheduling • Formative assessment using the ANTS System requires feedback to be given to trainees.
• Trainers/assessors will require to have time available for feedback sessions.
• Trainees who are taking part in a non-technical skills review should be given time out of their in-service commitment to attend a feedback discussion.
General recommendations • It may take some time for users to become familiar with the language and structure of the ANTS System; training should help facilitate this process. • As with other in-theatre training, teaching and assessment should not interfere with clinical care; if circumstances in theatre dictate, concurrent use of ANTS should be abandoned. • Formative assessment and feedback on ANTS should occur routinely in both clinical and simulator environments.
Suggested functions • To assess/review trainees' non-technical skills on a periodic basis to identify strengths and weaknesses and support skills development -use in a case or list where the trainee can manage the patient(s), as taking the lead, with consultant observing and providing assistance as requested/required, this can be as second anaesthetist or just stepping in if a problem occurs. • To guide general discussion of ANTS and their role in case management -consultant and trainee work together more as a team and discuss with case/list issues being considered from a non-technical perspective e.g. role of situation awareness -what is it for, how is it to be developed and maintained, how can it be lost or why good team working is so important?; -this more informal use is appropriate with new users, junior trainees when numerical ratings are premature, and senior trainees in more complex cases. • As a framework for self-reflection both by trainees and other grades -questions could be asked about the categories and elements either following or in advance of a case, e.g. what resources would be needed for a vascular emergency case, what are the situation awareness requirements in this case? • As above but during simulator-based training -videos of scenarios could be reviewed by the trainees with their instructors for more focussed feedback sessions.
Practical issues • Use ANTS System in a variety of different cases as appropriate for the list type, health of patient, trainee level and consultant load. • New users are recommended to work at the element level, as ratings can be more directly related to observed behaviours. • If using the ANTS System for skills assessment, make brief notes about observations on the form during the case if possible e.g. of things seen, not seen, key events. Following the case/list make ratings based on these observations. • Consultants and trainees should have a feedback and discussion session after the case or list being reviewed -use element level observations/ratings to give specific feedback on skills.
-use category level to describe more general performance. • Use whole ANTS System during training and assessment but focus on areas relating to weakness or of particular importance for type of case, e.g. co-ordinating with team in shared airway work. Prioritising -scheduling tasks, activities, issues, information channels, etc., according to importance (e.g. due to time, seriousness, plans); being able to identify key issues and allocate attention to them accordingly, and avoiding being distracted by less important or irrelevant matters. Team Working: Skills for working in a group context, in any role, to ensure effective joint task completion and team member satisfaction; the focus is particularly on the team rather than the task. It has five skill elements: co-ordinating activities with team members; exchanging information; using authority and assertiveness; assessing capabilities; supporting others.

Behavioural markers for good practice
Co-ordinating activities with team members -working together with others to carry out tasks, for both physical and cognitive activities; understanding the roles and responsibilities of different team members, and ensuring that a collaborative approach is employed.
Behavioural markers for good practice • confirms roles and responsibilities of team members • discusses case with surgeons or colleagues • considers requirements of others before acting • co-operates with others to achieve goals Behavioural markers for poor practice • does not co-ordinate with surgeon(s) and other groups • relies too much on familiarity of team for getting things done -makes assumptions, takes things for granted • intervenes without informing/ involving others • does not involve team in tasks Exchanging information -giving and receiving the knowledge and data necessary for team co-ordination and task completion. Using authority and assertiveness -leading the team and/or the task (as required), accepting a non-leading role when appropriate; adopting a suitably forceful manner to make a point, and adapting this for the team and/or situation.

Behavioural markers for good practice
Behavioural markers for good practice • makes requirements known with necessary level of assertiveness • takes over task leadership as required • gives clear orders to team members • states case and provides justification Behavioural markers for poor practice • does not challenge senior colleagues or consultants • does not allow others to put forward their case • fails to attempt to resolve conflicts • does not advocate position when required 11 Team Working: continued Assessing capabilities -judging different team members' skills, and their ability to deal with a situation; being alert to factors that may limit these and their capacity to perform effectively (e.g. level of expertise, experience, stress, fatigue). Gathering information -actively and specifically collecting data about the situation by continuously observing the whole environment and monitoring all available data sources and cues and verifying data to confirm their reliability (i.e. that they are not artefactual).

Behavioural markers for good practice
Behavioural markers for good practice • obtains and documents patient information pre-operatively • conducts frequent scan of the environment • collects information from team to identify problem • watches surgical procedure, verifying status when required • cross-checks information to increase reliability Behavioural markers for poor practice • reduces level of monitoring because of distractions • responds to individual cues without confirmation • does not alter physical layout of workspace to improve data visibility • does not ask questions to orient self to situation during hand-over Recognising and understanding -interpreting information collected from the environment (with respect to existing knowledge) to identify the match or mis-match between the situation and the expected state, and to update one's current mental picture. Balancing risks and selecting options -assessing hazards to weigh up the threats or benefits of a situation, considering the advantages and disadvantages of different courses of action; choosing a solution or course of action based on these processes.
Behavioural markers for good practice • considers risks of different treatment options • weighs up factors with respect to patient's condition • assesses time criticality associated with possible options • implements chosen option Behavioural markers for poor practice • does not find out about the risks associated with an unfamiliar condition/drug • does not preview courses of action with relevant people to assess their suitability • fails to review possible options with the team Re-evaluating -continually reviewing the suitability of the options identified, assessed and selected; and re-assessing the situation following implementation of a given action.
Behavioural markers for good practice • re-assesses patient after treatment or intervention • reviews situation, if decision was to wait and see • continues to list options as patient's condition evolves Behavioural markers for poor practice • fails to allow adequate time for intervention to take effect • fails to include other team members in re-evaluation. • is unwilling to revise course of action in light of new information 14

Rating Anaesthetists' Non-Technical Skills
The scale below can be used for rating non-technical skills based on observed behaviour. If it is not relevant for a particular element to be demonstrated in a situation, the 'not observed' rating should be used.

ANTS System Rating Options
Rating Label

Description
Performance was of a consistently high standard, enhancing patient safety; it could be used as a positive example for others Performance was of a satisfactory standard but could be improved Performance indicated cause for concern, considerable improvement is needed