Deliberate practice model effectively directs student learning in the critical care rotation: The impact of assessment on medical student learning

Introduction: Australian Residents (Junior Doctors) identified their clinical experience for acute clinical episodes as insufficient. Absence of clinical assessments during their critical care experience, within their medical degree, was thought to be a reason. The ‘deliberate practice’ model was used to explore medical student learning, during the Critical Care Medicine Rotation (CCMR) at Sydney Medical School, Nepean. Method: The interpretative-constructive theoretical study utilized theory, practical (simulation/clinical) standardized assessments and focus groups. Focus group questions directed student exploration of their learning during the rotation, and use of an assessment of basic airway management skills. The mixed method study design was conducted over 18 months (Jan 2014-May 2015), 96% (46) of possible cohort (27 male,19 female) enrolled during their clinical years. The ‘Biggs’ Model of ‘knowledge’ was used to understand student declarative, procedural and conditional knowledge. Results: Analysis of pre/post declarative knowledge n=37(80%) demonstrated a significant p value < 0.00001. From within the total cohort, n=12(26%) students were assessed for both procedural and conditional knowledge. This change was not significant at p ≤ 0.05. In this complex clinical environment, knowledge did not automatically translate directly into practical skill attainment. Students (focus groups n=36(78%))identified the model of ‘deliberate practice’ effective, indicated a desire for more simulations and clinical experience, clear faculty direction for required learning activities and valued peer support. Discussion: Application of ‘deliberate practice’ model in the CCMR was effective. The pre-and-post assessments provided students the knowledge their own skill level and was effective in encouraging individual skill development. Whereat S, Mclean A, Cole L, Suharto T, Hendry G, Lane S MedEdPublish https://doi.org/10.15694/mep.2020.000169.1 Page | 2 It highlighted a need to review other aspects of critical care clinical skills education. A follow up study, examining the role of peer/near peer assessment in this area has been conducted, with studies in different complex clinical environments, examining workplace learning contrasted with simulation, in post graduate trainees, a new area of research.


Introduction
Postgraduate residents (Junior Medical officers (JMO) also known as Postgraduate Year 1(PGY1) and PGY2 in Australia), have been identified as not adequately prepared to manage acute patients, they lack effective clinical experience in Critical Care Medicine (CCM) (Tallentire et al., 2012;Duns et al., 2008), however 30% of clinical decisions regarding sick deteriorating patients are made by residents in their first two years' postgraduate. Junior doctors self-identify a lack of competence, safety, and procedural skill preparation (Sullivan et al., 2010;Wong, 2005;Grover et al., 2009) and lack confidence in their ability to act appropriately in emergency situations (Tallentire, Smith, Skinner et al., 2012;Duns, Weiland, Crotty et al., 2008;Weller, 2004;Ellison et al., 2008;Nicol et al., 2011). The Critical Care Rotation (CCMR) during the Sydney Medical Program (SMP) provides the medical students with a unique but limited experience in the complex management of the critically ill in Emergency, Anaesthetic and Intensive Care Medicine departments.

Factors identified:
A lack of research within medical student groups during their CCMR, of their educational needs and the little clinical assessment of medical students, within the clinical environment in Australia and New Zealand CCMR programs was identified (Whereat, McLean, 2012). Simulation is used more often, provides a safe learning environment for students, however simulation, students indicate (Whereat, 2016;Bell et al., 2009;Freund et al., 2013), it does not replace explicit clinical experience, nevertheless is identified as a useful step toward and in preparation for clinical practice.

Exploration:
The literature clearly identifies that students are assessment driven and that assessment can be used to drive learning (Phillips, Smith, Straus, 2013). Consequently, the question considered, was 'Could assessments be a suitable tool to ensure students gained effective clinical skills, supervised practice and a range of core skills from this rotation?'.
Theoretical and clinical knowledge is vertically integrated throughout the four years of the Sydney Medical Program with clinical immersion in the last two years. Prior to clinical years, the students experience of basic airway management is conducted in the simulation setting, and few if any students had any practical experience in the clinical setting. The CCMR therefore, provided a real opportunity to explore the translation of this experience into the clinical environment.
Basic airway management is a core professional skill (Keogh et al., 2012) and while the basics of this skill are learnt pre-clinically, the application and understanding consolidated during in this rotation. Two benefits of an effective program would be that, prior knowledge would be unlocked at the beginning of the rotation, instigating the opportunity for additional scaffolding of student learning in this area. It would also effect a 'deliberate practice' learning experience, in the clinical environment (Ericsson, 2008;McGaghie et al., 2011). A mixed method study design, utilized theory and practical (simulation and clinical) assessments in addition to normal program, with focus groups exploring the effect of the assessment on their learning within CCM, and the effect of a practical skill learnt Whereat S, Mclean A, Cole L, Suharto T, Hendry G, Lane S MedEdPublish https://doi.org/10.15694/mep.2020.000169.1 Page | 3 in a simulation environment and its application in the clinical environment.
In studies examining clinical decision making (Schwartz et al., 2010;Schmidt, Norman, Boshuizen, 1990), two skills were identified in this process, in direct patient care, first diagnosis and management skills were required, and then the ability to apply that skill to the context or the individual. These two studies used standardized patients to train medical students to think about the application in context, this study went a step further, by utilizing both simulation and the clinical environment with real patients to support further contextual learning.

Methods
An interpretative/constructive theoretical study approach to student learning was conducted (Whereat, 2016). The mixed method 'basic airway management study' used quantitative methods to analyse the written, simulation and clinical assessments, feedback was provided following both the assessments. The qualitative method of focus groups, was utilized, to explore the students' perceptions of learning throughout this rotation and the impact of the assessment on their learning.
The (Biggs, 1999;Walsh, 2007) theoretical model describes the combination of the three distinct types of knowledge. Declarative knowledge is relevant theory. Procedural knowledge comprises the skills required to apply this knowledge, Conditional knowledge is the use of appropriate conditions to apply this knowledge within the workplace. Students' learning during the critical care rotation, is focused in the workplace, therefore utilises conditional knowledge application, along with prior procedural and declarative knowledge. Assessments then examined both the pre-existing levels of theoretical (declarative) knowledge and practical skill (procedural learning) with the post assessment in the clinical assessment exploring the changes gained throughout the rotation measuring 'conditional' or workplace learning.

Data Analysis:
A T-Test Calculator for 2 Dependent Means was applied to the pre and post results, with a thematic analysis of recorded focus groups conducted.

Results/Analysis
Study Participation: The study was conducted over 18 months (Jan 2014 -May 2015) with 46 (27 M,19 F) clinical medical students participating from the Sydney Medical School, Nepean (see Table 1, 'Student Demographics' below). The students were grouped according to the students' regular CCM rotation allocation and all were offered the opportunity to participate in the study. participation in study' below). Participant numbers involved in the pre-testing both written (93%) and practical assessments (96%) was high, post-written testing (80%) and focus group participation (78%) decreased a little, while participation (26%) in the post-clinical assessment was lower. The expectations of medical student knowledge and practical skill level, was based on senior clinicians (Intensivist and Anaesthetist) assessment, they agreed that students should be proficient in principles, and adequate in practice by the end of the rotation. The expected knowledge standard was based curriculum content delivered prior to clinical rotation and an analysis of new knowledge in curriculum delivered during rotation. The students completed the same written assessment, the minimum written knowledge level expected was 48% prerotation, and 64% (due to new knowledge introduced during rotation) post rotation, the practical skill expectation for a novice medical student was 60%.
Analysis theory component: Statistical analysis of written assessment results indicated a significant p value of < 0.00001. Figure 1 'Written comparison, pre-post-test results as against expected standards' see below, demonstrates that students' knowledge in this area was better than expected with 4 (9%) students below the accepted level (48%) at start of the rotation. Post block results, 2 (4.5%) students were below the expected minimum knowledge level of 64%, with the class average 83%. These results were expected (Larsen, Butler, Roediger Iii, 2009;Larsen, Butler,Roediger Iii, 2013;Karpicke, 2012;Karpicke, Roediger Iii, 2007), the authors indicate that testing not only measures content, but that tests enhance learning with repeated retrieval of knowledge. Repeated assessment ensures that long term retention occurs in this environment. Overall the level of knowledge change for 26 (59%) of the students was 11-30%, while 3 students level demonstrated no change. Likewise, this result was consistent with a clinical scenario study (Galvagno Jr, Segal, 2009). Two students demonstrated a negative outcome, one student's knowledge level deteriorated, the second student improved but did not meet expected standard.

Analysis practical assessments:
The practical assessment was based on the standard Basic Life Support assessment tool, modified to focus on the airway skill component. The specific criteria that were assessed are detailed below, as is, which of these criteria were completed more accurately by the students is detailed in Table 3 below. Achieves effective BVM seal and ventilates the patient with a good seal, duration >>one second for each breath 71% Positions patient correctly 84% Avoids trauma to patient's airway 89% Observes chest rise and adjusts technique accordingly (i.e. stops inflating stomach) 80% Identifies apnoea /airway problem 98% There were four criteria that highlighted issues and comparison with a similar study (Kidner, Laurence, 2006), identified that while the medical students performed better, both studies identified poor completion of similar criteria. These were attaching oxygen, calling for help, positioning the patient head and thinking to use airway devices. When 40 % of the students, still do not complete these criteria well, it identifies the potential for a serious problem, with direct consequence for patient safety, which considering the proximity of these students to graduation and clinical practice is an issue. It demonstrates what both senior and junior medical officers had indicated previously.
When applying the T-Test Calculator to both the practical assessments, the value of p was 0.754811 and the results were not significant at p ≤ 0.05. While this doesn't pass the test for significance or demonstrate substantial change, students were able to complete most tasks equally or better in the clinical setting. Seven (54%) students improved their personal results in the clinical setting, two (23%) performed consistently in both settings, therefore 9 (77%) students transferred the simulation experience to the clinical setting effectively, (see Figure 2 below).  There were clear improvements, in the use of airway adjuncts and bag valve mask. These students would have gained this practical experience in the anesthetics rotation, demonstrating the benefit of practical experience and the positive effect on their skill level. Several criteria "Calls for help" (in both settings), thinking to attach oxygen, checking pulse oximetry, checking chest rise remain criteria poorly completed as a group and were less well completed in the clinical setting. The criteria 'calling for help' in the clinical setting was complicated by the assessor standing by their side, therefore a confounding factor making the comparison of this criteria difficult.
Analysis of the relationship of theory to practical skill learning was considered. Demonstrated in Figure 4 'Comparison of students' rate of learning theory to practical skill' below, it was identified that there was no relationship between an individual's ability to learn theory and their corresponding ability to learn the practical skill. The figure below demonstrates the comparison each student's Pre-and-Post theory and practical result. No one student gained both types of knowledge at the same rate. During the eight weeks, some students gained more theoretical knowledge, others more practical skills, some even went backwards in knowledge. Limitations Participation (26%) in the post-clinical assessment was comparatively low due to several limitations. The two key factors that negatively impacted were core curriculum activities and the study design, both influenced student ability to participate. Students while completing these rotations were required to complete other curriculum assessments, reducing student focus and availability for core rotational activities. 'I really hated that we had so many assignments due this week…. It ruined my anesthetics experience and I liked anesthetics the best…. (F, Year 3, FG5).' Clinical assessment participation was also effected by study design. The poor numbers were related to three aspects of the study assessment that were required to align within one week. Assessors' availability (three were recruited), availability of consented low risk patients in the last week of the rotation, and participants' ability to leave their other clinical placement to attend the assessment.
The difficulty in designing and executing this type of study is recognized (McGaghie et al., 2010), with the number of clinical assessments conducted, consistent with other studies conducted within the clinical environment (Kidner, Laurence, 2006;Russo et al., 2013;Deakin et al., 2010) however it is indicated (Rosenstock et al., 2006) that 10-20 participants is sufficient to understand a phenomenon in a group of people.

Discussion
Overall the pre-post assessment analysis demonstrated, that the formative assessments, provided direction, resulted in effective feedback, and delivered improved student skill in the clinical environment. The knowledge and practical skills improvement indicates the positive effect of the modified CCMR, affording credence to student perceptions (Halamek, 2007) of their learning within CCM described within the focus groups. Focus group exploration of student perceptions of their learning during the CCMR (Table 4 'Focus group themes' below), identified two discrete areas of discussion, the impact of assessment on student learning and specific aspects of the CCM rotation that enhanced their learning more generally. Four themes were identified, with the first one 'Goal setting' exploring the use of the assessment in greater depth. The content of the other themes 'Active learning', 'Peer continuum' and 'Conflict' are explored in more depth in another forth coming paper.  (F,Year 4,FG4). Suggesting, that where there is important skill knowledge, the written assessment should not provide hints. Clinicians don't get that option in an emergency. This was an important aspect of the theme of "Goal setting". While students prefer directed guidance, they demonstrated that can be in the form of assessments they complete independently. 'I think doing the same test twice shows that because when you've done it before and you didn't know it and then you learned it' (F,Year 4,FG3).
Students identified that the practical assessment provided direction, a base for deliberate practice and understanding of expected level, from which they could build their knowledge and experience in this rotation. They also highlighted the difference and difficulty of managing a real patient. Points they made included that 'Learning's different on patients' (M,Year 4,FG2). 'When. l actually go see a patient and you're like oh, this is for real. I think the way that you're consolidating [your] memory is different in those two different situations' (F,Year 4,FG2).
Students fear that their first encounter requiring airway skills will be at an emergency, and that they will be ill prepared for this moment and therefore the 'focus on airway management' was important, 'I think it's probably pretty unanimous. When it comes to primary care, resuscitation, airway management is something that is obviously fundamental, but also really intimidating for students and young doctors....' (M,Year 4,FG2). The students first preference was to make mistakes in simulations rather than on a real patient. The practical and clinical assessments including supervision with both activities, created safety for both students and patients.
A core features of student learning in this environment, was the processes of observation, guidance and participation, 'Having the assessment first when we knew absolutely nothing was good ...... and so I went over it in my head a lot throughout the next eight weeks, how I would manage that differently as I was learning more and more, going through it. No, I wouldn't do what I did then, I would do it this way.' (F,Year 4,FG2). This student used the knowledge of her performance to provide a baseline to work from, she then spent time going over how to do the task and preparing for the second assessment.
Our survey of ICM rotations across Australia and New Zealand, identified that the use of practical assessments was limited (Whereat, McLean, 2012), in addition, most studies that utilize the pre and post method of testing, repeat assessments in the same simulation environment for consistency and to demonstrate effectiveness. In this study, the students were assessed in the clinical setting, to explore the use of context and impact on students learning. It has been described previously (Kneebone et al., 2006;Schuwirth, Van Der Vleuten, 2003) that contextual learning is easier, and which in turn, promotes greater knowledge retention, which the students in this study demonstrated clearly.
Students identified support, feedback and motivation as guiding their learning. 'So, it's just like using your practical knowledge and applying that to a real patient. You have a consultant there who can guide you if you get a little bit lost and they can give you feedback (F,Year 4,FG3).'The consultant provided support, guidance and immediate feedback to the students and importantly patient safety. The students felt that they learnt differently in the two different contexts (procedural and conditional) which confirmed the value in identifying the three methods of professional learning (Walsh, 2007), and the need for two different contexts for the practical assessment.
There was a strong theme indicating the value of focused feedback and supervised practice for students which identified the theme "Active learning". They confirmed that if supervised and staged opportunities for practice were provided, participants were able to change aspects of delivery of that skill immediately (Ericsson, 2008). The literature clearly identifies that students prefer 'supported participation' rather than 'adult learning' (Bell, Boshuizen, Scherpbier et al., 2009;Hay et al., 2013) and this was confirmed by our students within in the CCMR. The need for further research was identified with a peer assessment study analysis, in progress. The study of different complex clinical environments examining cohorts of both medical and junior surgical trainees are identified as areas for potential research and under consideration.

Conclusion
The use of assessment, in this environment was very positive learning tool for students. It provided a basis for deliberate practice, from which students, would build knowledge and experience. The students very clearly used the simulated assessment to create an understanding of their current level of expertise and directed their individual goals for supervised practice. The knowledge of the second assessment in the clinical context providing a stimulus, to deliberately go and seek practice, and directed their focus on basic airway management skills in CCM.
The middle of an emergency is not the time to be practicing or learning these skills. The improvements in student results confirm that there would be great benefit to add this clinical assessment as part of the curriculum. The assessors suggest this structure would be possible as part of the Anaesthetic rotation. Students noted the need for real experience, reinforcing the idea that being assessed in a real setting prior to graduation is far better than when they may be called to do this as a resident.
The curriculum aims to be a vertically integrated program, so the addition of planned assessments would aid in skill retention, and a potential major problem if skills are not practiced regularly. Therefore, if the educators/tutors design appropriately staged, supervised opportunities for clinical practice in this environment, the students will then be well prepared for clinical encounters into the future.
Improved knowledge and understanding of 'transfer' of knowledge into clinical settings is vital in today's educational setting. The use of a non-formalized assessment was seen to be valuable in the unlocking of prior knowledge, the additional impact of clinical learning opportunities, aided in improved student knowledge, and application of clinical skills.