Respective value of the traditional clinical rotation and high fidelity simulation on the acquisition of clinical reasoning skills in medical students – A Randomized Controlled Trial

To assess the respective value of Traditional clinical rotation and High Fidelity Simulation (HFS) on the acquisition of clinical reasoning (CR) skills in Medical Students. METHODS A randomized controlled trial was conducted. Forty medical students were exposed to a full day of HFS, either during their Paediatric Term (PT) (Experimental group = PT+, HFS+) or, after completion of their PT (Control group = PT+, HFS-). CR skills were assessed by a Script Concordance Test (SCT) prior to group allocation and at the completion of PT. RESULTS 39 out of 40 students completed both SCT. Scores before (mean/SD 57.4/6.7) and after (mean/SD 61.1/7.0) the PT were significantly improved (mean increase [95% confidence interval (CI)]: 3.6 [2.05.2]; p CONCLUSION Exposure to PT significantly improves the medical students’ CR acquisition with no further enhancement by exposure to HFS. Thus potential improvements in students’ knowledge and skills due to exposure to HFS may not affect CR skills, and thus may not be assessable by SCT. Lovett S, Roche J, Hunter S, Symonds I, Tomlinson N, Gagnon R, Charlin B, Mattes J MedEdPublish http://dx.doi.org/10.15694/mep.2016.000037


Introduction
In clinical settings, the challenge for educators today, is to develop educational interventions that not only improve the knowledge, professional development and technical skill of our medical graduates, but that improve the core clinical competency of clinical reasoning (CR) skill acquisition.The added challenge for universities and teaching hospitals is to provide these educational interventions that improve CR, to a growing number of medical trainees and graduates, in a cost-efficient and evidence-based manner.
CR is a complex process and clinicians analyze large amounts of information to arrive at the correct diagnosis or management plan for each patient.The organization of knowledge used in that endeavor is termed a "Script" (Charlin, Roy, Brailowsky, Goulet, & Vleuten, 2000).The CR process is extremely intricate, yet fast (Charlin, Boshuizen, Custers, & Feltovich, 2007), and different clinicians often come to the same diagnostic decision via varying paths (Grant & Marsden, 1988).Activation of the illness scripts, when dealing with a patient, helps the clinician to efficiently rule in or out hypotheses, and helps decide on management strategies for them (Charlin, Roy, Brailowsky, Goulet, & Vleuten, 2000;Charlin, Boshuizen, Custers, & Feltovich, 2007;Feltovich & Barrows, 1984).
The Script Theory aims to describe how humans understand real world events and why this understanding occurs almost effortlessly (Charlin, Boshuizen, Custers, & Feltovich, 2007).It tries to explain how our non-analytical (or pattern recognition) thought process can produce accurate decisions so efficiently, and suggests that for non-routine situations, deliberate script activation occurs (Coderre, Mandin, Harasym, & Fick, 2003) to help solve new problems.Using the Script Theory (Charlin, Roy, Brailowsky, Goulet, & Vleuten, 2000;Charlin, Boshuizen, Custers, & Feltovich, 2007;Charlin, Tardif, & Boshuizen, Scripts and medical diagnostic knowledge: Theory and application for clinical reasoning instruction and research, 2000), Charlin and colleagues have developed the Script Concordance Test (SCT) assessment that is able to give quantitative and reproducible results when assessing individual's CR capabilities.
The complex CR process is at the core of medical practice, and it is the clinician's most critical competence (Pelaccia, Tardiff, Triby, & Charlin, 2011).Doctors who can clinically reason efficiently and accurately will benefit society (Askew, Manthey, & Mahler, 2012) (Forsberg, Aronsson, Keller, & Linbald, 2011).If CR is a skill of utmost importance to develop competent clinicians of the future we must ask ourselves: Can we design learning activities to promote the CR acquisition of novices? 1.
Can we assess the effectiveness of these learning activities in terms of CR acquisition?2.
The challenge for educators is to support clinical students in the acquisition of CR ability, by providing educational interventions that teach these skills (Newble, Norman, & van der Vleuten , 2000).The assumption that CR ability is a product of experience has meant that the default method of teaching CR to medical students revolves around the exposure to traditional content curriculum provided at university and to patients and cases while on the wards or in tutorial groups.More recently HFS has been used as a surrogate for providing clinical experience to students and graduates.However, HFS is expensive and labor intensive teaching method.Although improvement in technical clinical skills has been proven (McGaghie, Issenberg, Cohen, Barsuk, & Wayne, 2011;Cook, et al., 2011;Cheng, Lang, Starr, Pusic, & Cook, 2014), further research is required to validate its use to improve CR acquisition.
In this randomized controlled trial we assessed the effect of HFS on the acquisition of CR skills, as measured by SCT, in medical students during the traditional PT.

Assessment Tool
The SCT is largely used as a measure of clinical reasoning.The SCT measures one of the most crucial aspects of CR, which is the ability to weight information in ill define scenarios to make decisions (Charlin, Roy, Brailowsky, Goulet, & Vleuten, 2000;Charlin & van der Vleuten, Standardised assessment of reasoning in the context of uncertainty: The Script Concordance approach., 2004), thus making the SCT the best available assessment tool for the very complex process of CR (Charlin, et al., 2012).Hence we can measure the effect an intervention has on an individual's CR abilities, and thus the SCT can guide development of ways to teach CR.Our study chose to utilize the SCT to determine how effective the traditional PT alone (control group = (PT (+) HFS (-)) versus traditional PT with one day HFS exposure (experimental group = (PT (+) HFS (+)) was at improving clinical reasoning acquisition in medical students.

Clinical Rotation
Students from the Joint Medical Program (JMP) of the University of Newcastle (UoN) and University of New England (UNE), commencing their fourth year of the five years medical program were recruited to participate in the project.Involvement in the project occurred during the six week PT that comprised traditional paediatric curriculum delivered by problem based learning tutorials, case based discussions, lecture series, paediatric clinical placements and skills sessions.This traditional format is aimed at providing knowledge and skills to the students so they are competent and safe interns who are versed in the common acute and chronic medical presentations of children.The PT assessment includes summative (multiple choice examination and Objective Structured Clinical Exam (OSCE), student reflective portfolio) and formative (professional and clinical skills) assessment tools.All participating students entered the study with similar baseline technical and non-technical skills attained throughout the previous years of the JMP.No students have had previous exposure to paediatric medicine prior to involvement in study.Students did not miss out on any of the traditional PT to be involved in this study.

High Fidelity Simulation
Eight paediatric simulation scenarios were designed of common and acute paediatric presentations.These scenarios were designed to develop clinical reasoning and decision making skills rather than technical or procedural skills.Scenarios were trialed on Paediatric Registrars and Consultants for validity, prior to exposing medical students.Each student performed individually one simulation in a random order followed by peer feedback and instructor debriefing.Each student also observed an additional seven live simulations with instruction to provide peer feedback and attend debriefing immediately after the simulation.Sending the students into the scenarios as individuals forced them to face a situation of uncertainty, critically analyze the information provided, and come to a clinical decision for their patient in the scenario.The aim of this was to mobilize illness scripts of individual students' and thus promote CR (Charlin, Tardif, & Boshuizen, Scripts and medical diagnostic knowledge: Theory and application for clinical reasoning instruction and research, 2000).Scenarios included acute presentations of croup, asthma, pneumonia with sepsis, head injury, abdominal trauma, anaphylaxis, sepsis and seizures.Each scenario involved a senior paediatric nurse, a high fidelity child or infant mannequin, consultant phone advice, access to NSW Health practice guidelines, a paediatric pharmacopoeia, and a resuscitation trolley with standard airway and circulation support equipment.The students were aware that during the scenario, they were only expected to function at the intern level (junior medical officer), and under no circumstances were they expected to perform procedures that they would otherwise not for their skill level.The focus of the simulation experience was on clinical decision making, not on acquiring new technical skills.Students were expected to be able to perform basic life support and all students had completed the Resus4Kids (Resus4Kids, 2013) program prior to participation.
On the day of exposure to HFS, each group was oriented to the space, equipment and expectations of the day.All students were debriefed following their scenario.A confidentiality contract was signed by each member of the staff and student cohort to ensure confidentiality of students' performances.

Script Concordance Test Development
Utilizing published SCT guidelines for construction (Dory, Gagnon, Vanpee, & Charlin, 2012;Fournier, Demeester, & Charlin, 2008) a script concordance test was developed comprising 51 cases, each with three questions relating to the case, totaling 153 questions.The clinical scenarios comprised of common paediatric cases that reflect the goals of Australian medical educational objectives.
We approached nineteen experts to complete the SCT (Dory, Gagnon, Vanpee, & Charlin, 2012); all were certified by the Royal Australasian College of Physician as Pediatricians or Fellows.Fifteen accepted and returned completed tests, complying with the recommendations made by Gagnon et al (Gagnon, Charlin, Coletti, Sauve, & Van der Vleuten, 2005) to get ten or more panel members to achieve adequate reliability estimates and thus created the scoring keys.
The SCT comprising 51 cases and 153 questions (see Figure 1 for example SCT question), which the experts completed, was then assessed to determine the questions that best detected clinical experience.Questions whose responses lacked variance among experts, are thought to measure factual knowledge rather than reasoning in context of uncertainty.These questions are discarded as deemed poor discriminators of clinical expertise and reasoning.Utilizing the published guidelines by Charlin et al (Charlin, et al., 2006) we computed the variance among the expert panel members.Questions that showed a variance above 0.4 on the indicator were chosen for the final SCT that was delivered to the students.This final SCT comprised of 41 cases and a total of 90 questions.Students had no previous experience with the SCT format and instructions on how to answer the paper were given prior to sitting SCT.The SCT was completed by the students on a paper copy and the results were scored and computed with the excel program available on the SCT website of the University of Montreal, Canada (University of Montreal -Faculty of Medicine, 2014).

Study Populations
Information regarding the project was provided during their introductory week for the rotation in JMP Women and Childs' Health (WACH) course.All students were asked for participation, 64% accepted to take part (n=40).The Human Ethics Committee of the UoN and UNE approved the study and written informed consent was obtained from students and experts before inclusion into the study.

Study Design
40 students were randomly assigned to either experimental (PT (+) HFS(+)) or control (PT(+) HFS(-)) arm.After randomization, 22 students were assigned to the control arm and 18 to the experimental arm.Simple randomization was performed and was completed independent to researchers.

Statistical Analysis
This study was powered to detect a difference in mean SCT score of one standard deviation between groups in the second SCT result with a power of 80% and alpha value of 0.05.Regarding analysis, we confirmed that the SCT data follows a Gaussian distribution employing the Kolmogorov-Smirnov test.We used the t-test for paired or unpaired observations (as appropriate) to test for statistically significant differences in SCT scores if normally distributed, and the Wilcoxon matched pairs test (paired) or Mann-Whitney test (unpaired) for data that was not normally distributed.We used the Spearman correlation coefficient (r s ), to quantify the association between two variables.We used Prism 5 GraphPad Software Inc. for analysis.

Results
There was no significant difference between the experimental and control group's SCT-1 result, indicating that the participating students all enter the PT with similar base line CR capabilities (Fig 3).Both the experimental and control group showed improvement in their SCT scores following being exposed to the traditional PT (Fig 3).
There was no significant difference between the SCT-2 scores between the experimental and control arms, indicating that exposure to one session of simulation did not affect SCT scores (primary outcome).The improvement in mean SCT score [95% CI] observed in the experimental group was 3.2 [0.5-5.9] and in the control group 4.0 [1.9-6.1](p=0.61).Merchant, 2012;Issenberg & Scalese, 2007).However it is unknown whether HFS directly impacts on CR capabilities.Our study was designed to assess the impact of HFS on CR skills acquisition of students during their PT as part of the JMP medical curriculum.We have shown that HFS implemented into the PT in a way which was thought to be deliverable and sustainable within the JMP medical curriculum in the future, had no significant effect on CR skills assessed by SCT.Importantly, both experimental and control groups did show an overall improvement in their CR skill acquisition, demonstrating the educational worth of the traditional PT.Thus SCT is sensitive to an educational intervention, in this case PT, which has not previously been presented in the English literature.
It is unlikely that this result is a consequence of the study size because our study was powered to detect a difference of one standard deviation in SCT score between groups.It is possible that exposure to HFS needs to be increased significantly to affect CR, at least in novices.However, the additional face-to-face teaching time for HPS in this study was greater than 50% of the time committed to structured professional skills or problem-based learning tutorials during the PT.Thus, we and others would find it challenging to offer more HPS teaching in a six-week rotation, due to competitive teaching activities thought to be a integral component of a problem-based medical curriculum.It is possible that the HPS modules used in this study require modification if the goal is acquisition of CR skills.This would require future prospective studies.
HFS was well received by our students (data not shown), which is in line with previous reports (Ten Eyck, Tews, & Ballester, 2009).The student satisfaction surveys showed that the experience was enjoyable, approachable and they felt more confident about dealing with acute paediatric situations after the simulation.We acknowledge that there is a significant role for the use of HFS in training medical students and graduates, especially when developing technical and non-technical skills such as communication and teamwork.
Potential confounders that occurred during our study must be recognized.The difficulty in creating a large bank of SCT questions from which to validate and produce two different tests of reliable value meant the first SCT sat by the students was the same as the second SCT.There was the concern that any improvement in the second SCT over the first SCT scores may be due to recall bias.It was felt by the research team that this effect would be small, due to the six-week gap between sittings, and the complexity of the questions.Furthermore the actual design of the SCT relies upon there not being one correct answer, but a spectrum of "correctness" for each hypothesis.Most importantly our study was controlled, and any potential recall effect would have resulted in an improvement in both groups without compromising the primary outcome.
The impact that the HFS sessions had on the students was obviously a key concern to how it affected the students' CR skills.We relied on the students' respecting the need for confidentiality amongst each intervention group to not discuss HFS scenarios with students still yet to experience the HFS to maximize the teaching power of the HFS scenarios.The importance of not revealing information about the intervention day was highlighted by the confidentiality clause all students had to sign to be eligible to participate.

Conclusion
We were able to show that during exposure to the traditional paediatric curriculum found in the JMP, there is a significant improvement in CR acquisition skills in our population and this was detected by the SCT.
HFS is a well-received teaching method that provides experiential learning opportunities for students, however it is very expensive and labor intensive.
HFS does provide opportunities to attain technical and non-technical skills but not for the development of clinical reasoning when delivered as it was in our study.
The effect of HFS on CR has been previously difficult to analyze, but the SCT offered us a unique and reliable method to do so.Exposure to one HFS session did not improve CR acquisition, above what was achieved through exposure to the traditional medical curricula of the JMP.
These findings provide the first level II evidence that SCT is sensitive to an educational intervention and suggest that HPS may not promote CR skills in medical students when assessed by SCT.It also highlights the potential use of the SCT to identify and design learning activities in our JMP curriculum that can improve CR acquisition of our future students in an efficient, enjoyable and economical way.

Figure 1 .
Figure 1.Example of the components of a script concordance question.Image modified from Charlin B and van der Vleuten C (Charlin & van der Vleuten, Standardised assessment of reasoning in the context of uncertainty: The Script Concordance approach., 2004) with permission.

Figure 3 :
Figure 3: SCT Scores for Experts and Students