Junior doctors’ preparedness to prescribe, monitor, and treat patients with the antibiotic vancomycin in an Australian teaching hospital

Purpose We aimed to assess the preparedness of junior doctors to use vancomycin, and to determine whether attending an educational session and being provided pocket guidelines were associated with self-reported confidence and objective knowledge. Methods This was a 2-component cross-sectional study. A 60-minute educational session was implemented and pocket guidelines were provided. Preparedness was evaluated by a self-reported confidence survey in the early and late stages of each training year, and by continuing medical education (CME) knowledge scores. Results Self-confidence was higher among those later in the training year (n=75) than in those earlier (n=120) in the year for all questions. In the late group, vancomycin education was associated with higher self-confidence regarding the frequency of therapeutic drug monitoring (P=0.02) and dose amendment (P=0.05); however, the confidence for initial monitoring was lower (P<0.05). Those with pocket guidelines were more confident treating patients with vancomycin (P<0.001), choosing initial (P=0.01) and maintenance doses (P<0.001), and knowing the monitoring frequency (P=0.03). The 85 respondents who completed the knowledge assessment scored a mean±standard deviation of 8.55±1.55 on 10 questions, and the interventions had no significant effect. Conclusion Attending an educational session and possessing pocket guidelines were associated with preparedness, as measured by higher self-reported confidence using vancomycin. High knowledge scores were attained following CME; however attending an educational session or possessing pocket guidelines did not significantly increase the knowledge scores. Our findings support providing educational sessions and pocket guidelines to increase self-confidence in prescribing vancomycin, yet also highlight the importance of evaluating content, format, and delivery when seeking to improve preparedness to use vancomycin through education.

In Australia, junior doctors (medical postgraduates 1-2 years and above who have not completed specialist training) undergo semistructured teaching in public hospitals guided by the Australian Curriculum Framework for Junior Doctors. Under the domain of clinical management, this framework lists prescribing, therapeutics, and treating infections as core areas for junior doctors [8]. There is modest evidence supporting educational interventions to improve antibiotic prescribing in hospitals, which can be considered as belonging to the field of antimicrobial stewardship [9], and in general to improve prescribing by junior doctors [10]. However, no current study, to our knowledge, has evaluated the impact of educational interventions on junior doctors' preparedness to prescribe and treat patients with vancomycin. As junior doctors perform the great majority of prescribing in teaching hospitals, the aim of this study was to assess the preparedness of this group to prescribe and monitor vancomycin, and to determine whether an educational program and the provision of pocket guidelines were associated with self-reported and objective knowledge of vancomycin prescribing.

Setting
The study was conducted at Flinders Medical Centre (FMC), a 580-bed government teaching hospital in Adelaide, Australia.

Subjects
The participants of the study were junior doctors identified from the register of the Trainee Medical Officer (TMO) Unit, FMC. The potential cohorts of junior doctors available to participate comprised 72 doctors in 2012, 73 in 2013, and 74 in 2014.

Study design
This was a cross-sectional study assessing confidence and knowledge about prescribing and monitoring vancomycin conducted between 2012 and 2014. The study comprised 2 components each year. Component 1 was a self-reported confidence survey (Supplement 1), and component 2 was an online continuing medical education (CME) module on vancomycin with knowledge assessment questions (Supplement 2). The 8 survey questions relating to self-reported confidence were analysed individually, but were first subjected to a content validity assessment to assess topic coverage via a factor analysis to determine dimensionality and an analysis using the Cronbach alpha to assess internal reliability. Content validity was assessed by 4 experts (2 pharmacists and 2 physicians). Following the adaptation of several questions, agreement was reached that the questions covered all relevant aspects of the construct. We used factor analysis, with maximum likelihood used to determine whether the ques-tions could be considered as all relating to a single confidence domain. The 8 questions provided solutions with between 1 and 4 factors. The lowest Bayesian information criterion (BIC) was obtained for that with 2 factors (BIC= 109.7) but the solution with a single factor was very similar (BIC = 110.0). In addition, the first factor was the only one of the 4 factors with an eigenvalue greater than 1, and it alone explained 69.4% of the variability in the data, indicating that the 8 questions could be thought of as relating to a single domain. The internal reliability as assessed by the Cronbach alpha was alpha = 0.929. The knowledge component questions were also assessed for content validity by the same panel of 4 experts. The knowledge questionnaire comprised 10 multiple-choice questions, which were given equal weighting for a total score ranging from 0 to 10. Agreement on the correct answer for each question was also assessed by the experts who each took the test alone before obtaining concurrence on the correct answer.
Component 1: the self-confidence survey required respondents to use a 5-point Likert scale of (1) strongly agree, (2) agree, (3) not sure, (4) disagree, and (5) strongly disagree for a series of questions. The survey was disseminated to doctors in both the early and late part of each year. In Australia, the hospital teaching year is January to December, and the survey was disseminated and completed by participants in a 4-week period commencing in January (early in the training year) and a 4-week period running from November to December (late in the training year) in 2012, 2013, and 2014. The self-confidence survey was available to complete on paper and electronically via Survey Monkey (San Mateo, CA, USA). Component 2: the CME and knowledge assessment were provided over 3 consecutive years during the period of June to December. The CME module was disseminated to doctors via email with a link to the online knowledge assessment, also hosted via Survey Monkey. No incentives were offered to complete the self-confidence survey or CME knowledge assessment.
The study included educational support in the form of a 60-minute, face-to-face, non-compulsory educational session and the provision of pocket guidelines (Supplement 3). The educational sessions contained core information and practical advice on prescribing and monitoring, with content selected by a multidisciplinary group of local experts. The sessions were delivered 3 times each year in an effort to capture rotating doctors. Sessions began early in participants' training year, with repeat sessions offered mid-way through the year. The laminated pocket guidelines (6 × 10 cm) contained the essential features of the institutional guidelines. The pocket guidelines were disseminated at educational sessions and via the TMO Unit.

Statistical analysis
Differences in the mean Likert scale confidence scores between the early and late training groups were assessed using the independent t-test. In addition, amongst the late in training year group alone, http://jeehp.org we also compared the difference in mean confidence scores for those who attended an educational session and those who did not, and also the difference in mean confidence scores for those who possessed pocket guidelines and those who did not. The differences in the proportion of respondents correctly answering CME knowledge questions according to vancomycin prescribing experience, attendance at an educational session, and possession of pocket guidelines were assessed in a univariate analysis using the Fisher exact test. In addition, we also assessed whether these 3 factors were independent predictors of a correct response using multivariate binary logistic regression. Finally, multivariate linear regression was used to assess whether any of these 3 factors predicted total knowledge scores, as defined by the number of correct responses across the 10 questions.

Sample power
We had 80% power to detect a difference in Likert scale confidence scores of 0.4, assuming a standard deviation in confidence for each question of 1.0 for each group (n= 120 and n= 75). In regard to assessing differences in the proportion of correctly answered CME knowledge questions, we had 80% power to detect a difference of 18%, assuming that approximately 80% of subjects correctly answered each question between 2 groups of size n = 27 and n = 58 (pocket guidelines groups) and 82% power for 2 groups of size n = 40 and n = 45 (educational session groups) for those who completed the CME knowledge questions. All analyses were performed using IBM SPSS ver. 22.0 (IBM Corp., Armonk, NY, USA).

Ethical approval
This study received full ethics approval from the Southern Adelaide Clinical Human Research Ethics Committee, Australia (approval 123.12).

Component 1: self-reported confidence survey
A total of 195 completed surveys were received over 2012-2014 (120 from the early group and 75 from the late group). Raw data are available in Supplement 4. The work position and experience of junior doctors is presented in Table 1. Self-reported confidence in prescribing vancomycin improved across the 8 domains between the early and late groups (P< 0.001) ( Table 2).
The association of attending an educational session with self-reported confidence in prescribing vancomycin was evaluated in respondents who responded late in their year of training (n= 75). Those who had attended an educational session had a higher degree of con-  Values are presented as mean ± standard deviation (95% confidence interval). Likert score 1 = strongly agree, 2 = agree, 3 = not sure, 4 = disagree, 5 = strongly disagree. VAN Table 3). The association of possessing pocket guidelines with self-reported confidence in prescribing vancomycin was also evaluated in respondents from the late group (n= 75). Those who possessed pocket guidelines had a higher degree of confidence in terms of treating patients with vancomycin, choosing an initial dose, choosing a maintenance dose, and knowing how often to take vancomycin blood levels (Table 4).

Component 2: continuing medical education knowledge assessment
Preparedness to prescribe, monitor, and treat patients was determined by knowledge scores obtained after completion of an online CME module on vancomycin. Eighty-five respondents completed the CME questions. Demographic factors and experience prescribing vancomycin are presented in Table 5. The mean and standard deviation for the total knowledge score was 8.55± 1.55 from a maximum achievable score of 10. Scores were not influenced by prescrib- Attendance versus non-attendance, using the unpaired t-test. Values are presented as mean ± standard deviation (95% confidence interval). Likert score 1 = strongly agree, 2 = agree, 3 = not sure, 4 = disagree, 5 = strongly disagree. VAN ing experience, attending an educational session, or possession of the pocket vancomycin guidelines (Table 6). In multivariate linear regression, there were no significant effects for experience prescribing vancomycin (β= 0.09 ± 0.36, P = 0.82), attending an educational session (β = −0.56 ± 0.35, P = 0.12) or possessing pocket guidelines (β = 0.62 ± 0.38, P = 0.11). The range of correctly answered individual knowledge questions is presented in Table 6, with no differences observed for those who attended a prior educational session, were in possession of pocket guidelines, or had more experience prescribing vancomycin. However, in multivariate logistic regression, the odds of correctly answering the question about the loading dose were unexpectedly lower than in those with more experience prescribing vancomycin than in those with less experience (odds ratio [OR], 0.17; 95% CI, 0.04 to 0.81; P = 0.02), and was higher for those with a pocket guide than for those without (OR, 6.3; 95% CI, 1.11 to 36.2; P= 0.04). The odds of correctly answering the question related to managing initially elevated vancomycin levels were lower for those who attended the educational session than for those who did not (OR, 0.20; 95% CI, 0.05 to 0.83; P= 0.03). Experience prescribing vancomycin, prior attendance of an educational session, and possessing pocket guidelines were  Values are presented as number (%). Likert score: 1 = strongly agree, 2 = agree, 3 = not sure, 4 = disagree, 5 = strongly disagree. VAN not predictors of a correct response for any of the other questions.

Discussion
This study examined a pertinent topic with a pragmatic design employing educational interventions in the challenging environment of an authentic clinical context. During this study, we observed that junior doctors' self-reported confidence was higher for all questions when asked later in the hospital teaching year. While it could be argued that increased confidence occurs simply with increasing experience over the year spent working as a doctor, some two-thirds of doctors reported very limited experience, having prescribed vancomycin as little as 10 or fewer times. In the current study, those doctors who had attended an educational session were more confident in the domain of therapeutic drug monitoring of vancomycin; specifically, knowing when to measure blood levels and how frequently to monitor them once the patient is in the target range, and borderline significance was found for confidence in the more complex task of interpreting vancomycin blood results to amend dosing. These are important findings, as measuring blood levels at the wrong time and frequency can result in misinterpretation of the results and lead to incorrect dosage adjustment [11]. As our educational intervention was multifaceted, we are unable to determine the effect of individual components; thus, we cannot say if future resources should be directed to face-to-face sessions, online CME, or provision of pocket guidelines. The CME with knowledge questions was developed with considerable input from pharmacy, infectious diseases, and clinical pharmacology to ensure that the CME content was contemporary. The time required to prepare the CME with questions was significantly in excess of the time required for preparation and delivery of the face-to-face sessions, yet the CME, once prepared, can be disseminated to a large audience if required. Getting junior doctors to take time out of their busy schedule for an educational session is challenging, but during an internal evaluation of these sessions, junior doctors overwhelmingly agreed or strongly agreed that attending the sessions was useful to their clinical practice.
The junior doctors who received pocket guidelines were significantly more confident on 6 of the 8 questions about dosing and monitoring vancomycin, suggesting strongly that provision of the pocket guidelines improved their confidence. These finding are meaningful, as a systematic review found that low levels of confidence had a negative impact on the preparedness of junior doctors, as did deficiencies in areas such as prescribing [12]. Furthermore, perceived confidence has been reported to have a significant effect on the clinical behaviour of medical graduates [13].
The proportions of correct responses to the online CME knowledge questions on vancomycin were generally very high, for all but 1 question regarding the management of red man syndrome. Interestingly, pharmacists have also scored low on formal CME questions about the management of red man syndrome [14]. Experience in prescribing vancomycin, attendance at a prior educational session, or possessing pocket guidelines did not increase the total knowledge scores. The multivariate analysis of answers to individual CME questions provided some unexpected findings. Those with more experience prescribing vancomycin had a lower likelihood of answering the question about the loading dose correctly than those with less experience. Similarly, the question about managing elevated vancomycin levels produced a surprising result. Those who attended an educational session were paradoxically less likely to answer this question correctly than those who did not. This counter-intuitive performance may potentially be explained by the possibility that those with more experience were less inclined to consult the guidelines for advice before prescribing. Alternatively, the educational content for these areas may have been unclear or potentially confusing, or it may be that doctors who chose to attend the educational sessions did so because they felt less informed than those who did not attend. Nevertheless, these findings emphasise the importance of careful review of the content, format, and delivery of educational interventions, as well as the need for those with educational expertise and content knowledge to evaluate them rigorously.
There are some limitations to this study. As it was cross-sectional in design, we cannot infer causality. In particular, the differences observed for those attending or not attending the educational session and those with and without pocket guidelines may be due to reverse causality. This study did not assess actual vancomycin prescriptions written by these doctors, so we do not know if their clinical behaviour changed after the educational intervention. Further studies assessing educational support to improve junior doctors' preparedness to clinically use vancomycin should involve the rigorous evaluation of such interventions, using more programmatic approaches implemented across multiple sites with numerous sources of stakeholder input to avoid the constraints of self-reported data, as has recently been proposed by others [15]. Larger sample sizes are required to detect some of the smaller but non-significant improvements in knowledge that were observed in the study. Noteworthy, however, is that our study contained more subjects than many of the randomised and non-randomised studies included in a systematic review on educational interventions to improve prescribing for junior doctors [10].
In conclusion, to our knowledge, this is the first evaluation of junior doctors' preparedness to prescribe the antibiotic vancomycin. Possession of pocket guidelines was associated with significantly higher self-reported confidence to use vancomycin, while attending an educational session was associated with higher self-reported confidence to perform therapeutic drug monitoring. Generally high knowledge scores were obtained by those completing an online CME assessment on vancomycin. However, no apparent effect on knowledge scores was associated with attending an educational session, possessing pocket guidelines, or having increased experience prescribing vancomycin. Based on our findings, future initiatives to improve the preparedness of junior doctors to prescribe vancomycin could include http://jeehp.org J Educ Eval Health Prof 2017; 14: 13 • https://doi.org/10.3352/jeehp.2017.14.13 education and the provision of pocket guidelines; however, careful design and close evaluation of educational content, usability, and format require the utmost consideration.