Self-Assessment of Addiction Medicine Core Competencies in Four Year Groups of Psychiatrists in Training: Efficacy of the Addiction Medicine Training Needs Assessment Scale in a Local Training Context

Background: In addiction medicine training, self-assessment is increasingly used to support self-regulation learning by identifying standards of excellence, competence gaps, and training needs. To ensure psychiatrists in Lithuania also develop specific addiction competencies, the Lithuanian Health Sciences University faculty in Kaunas developed an addiction psychiatry curriculum. Objectives: The aim of this research is to explore the efficacy of the AM-TNA scale to measure individual and group differences in proficiency in the core competencies of addiction medicine. A cross-sectional study and a convenience sample were used. Method: We studied the differences in performance in addiction medicine competencies between 4 successive year groups and analysed the variance to determine the statistical differences between the means of 4 year groups with biases, resulting from repeated measurement statistically corrected-for. Results: Of the psychiatrists in training, 41% or 59% completed the scale. The assessment of competencies suggested that all but 2 competencies differ significantly (p < 0.05) between the 4 groups. The post hoc analyses indicated that mean scores for 24 of the 30 core competencies differed significantly between the year groups (p < 0.05) and showed a gradual increase in scores of self-assessed competencies over the 4 year groups. We found adequate scale variance and a gradual increase in self-assessed competencies between the 4 year groups, suggesting a positive association between the results of incremental professional training and improved self-assessed substance use disorders (SUD) competency scores. Conclusions: This study illustrates the efficacy of the AM-TNA scale as an assessment instrument in a local training context. Future research should aim to have larger sample sizes, be longitudinal in design, assess individual progress, and focus on comparing and combining self-reported competencies with validated objective external assessment and feedback.

resulting from repeated measurement statistically corrected-for. Results: Of the psychiatrists in training, 41% or 59% completed the scale. The assessment of competencies suggested that all but 2 competencies differ significantly (p < 0.05) between the 4 groups. The post hoc analyses indicated that mean scores for 24 of the 30 core competencies differed significantly between the year groups (p < 0.05) and showed a gradual increase in scores of self-assessed competencies over the 4 year groups. We found adequate scale variance and a gradual increase in self-assessed competencies between the 4 year groups, suggesting a positive association between the results of incremental professional training and improved self-assessed substance use disorders (SUD) competency scores. Conclusions: This study illustrates the efficacy of the AM-TNA scale as an assessment instrument in a local training context. Future research should aim to have larger sample sizes, be longitudinal in design, assess individual progress, and focus on comparing and combining self-reported competencies with validated objective external assessment and feedback.

Introduction
Lifelong Self-Regulated Learning and Self-Assessment Substance use disorders (SUD) are a major contributor to the global burden of disease, ranking fifth in the top-10 causes of disability and death and a major cause of health care expenditure [1,2], requiring not only improvement of effectiveness and organization of addiction medicine training across the globe [3], but also adequate postgraduate and continuous medical education for addiction medicine specialists.
In order to stay abreast of the latest developments and to strengthen professional expertise and competence, addiction specialists are required to be successful lifelong self-regulated learners: being able to identify their own strengths and weaknesses and having effective selfassessment and improvement strategies and tools [4]. Self-assessment can be used as an explicit way to strengthen formative assessment and is, in established addiction medicine curricula, increasingly used to support selfregulation learning [5]. However, stand-alone self-assessment has, especially when used as summative evaluation for medical students, accuracy limitations [6]. In order to encourage the use of valid self-assessment tools by addiction professionals, we chose to study self-assessment from the perspective of learning experience and as a formative assessment tool. We also stress the importance of combining self-assessment with other forms of assessment, such as feedback by peers, senior colleagues, preferably using multiple assessment tools [7][8][9].

Addiction, Training, and Assessment in Lithuania
In Lithuania, alcohol use contributes to 25% and smoking to 14% of total deaths [10]. The Lithuanian SUD-services are the responsibility of psychiatrists, who study SUD as a common comorbidity of other psychiatric conditions during their 4-year general psychiatry residential training. In order to ensure these trainees also develop specific SUD competencies, the Kaunas Lithuanian University of Health Sciences (LUHS) developed a specific addiction psychiatry curriculum, including a 3-month supervised addiction medicine placement in the final 2 years. Progress on SUD competencies is assessed using the Lithuanian version of the Addiction Medicine Training Needs Assessment scale (AM-TNA) [11]. This self-assessment instrument is useful for connecting addiction medicine training needs to the content of the training courses, through supporting self-regulated learning by both generalists and specialists. As a result, the aim of this study is to explore the AM-TNA's self-assessment efficacy to identify both individual and group differences in strengths and weaknesses in core competencies among 4 successive year groups (YG) of psychiatrists in training enrolled in the aforementioned addiction psychiatry course.

Objectives
• Exploring the efficacy of the AM-TNA scale in assessing individual differences in proficiency in addiction competencies. • Exploring the association (if any) between the AM-TNA self-assessment scores and the level of postgraduate training groups (years 1-4) of psychiatrists in training.

Materials and Methods
Design In this preliminary evaluation of self-assessed addiction medicine core competencies and training, a cross-sectional comparison was chosen to measure differences in perceived proficiency of 30 core addiction competencies, in 4 successive YG of psychiatrists in training at Kaunas LUHS.

Sample and Participants
To measures differences in training outcomes, we used a convenience sample of all psychiatrists in training at the Kaunas LUHS psychiatry clinic in May 2019. The Lithuanian training capacity for psychiatrists is around 100: being the responsibility of the medical schools of the Universities of Vilnius and Kaunas.

Ethical Considerations
The study complies with the Declaration of Helsinki: written informed consent from the participating psychiatrists in training was established before the completion of the anonymous questionnaire.

The Instrument
The AM-TNA was developed to measure progress and gaps on addiction medicine competencies in Indonesia. The instrument consists of a 30-item questionnaire containing self-reported proficiency on internationally recognized addiction medicine specific core competencies, using a 5-point Likert-scale ranging from not-at-all proficient-fully proficient. The instrument assesses 21 clinical and 9 non-clinical competencies in 3 professional domains: assessment and diagnosis, starting treatment, and maintaining treatment [12]. The Indonesian AM-TNA scale was translated into Dutch, Lithuanian, and English. The scale has besides a proven reliability and internal validity a clear-cut 2-dimensional factor structure [11].

Analysis
To demonstrate psychiatrists in training can self-assess their proficiency in addiction competencies using the AM-TNA scale we expect, besides a high response rate, very few missing values, an adequate variance in results, and a significant difference in means between the 4 YG. To assess the relationship between these groups and self-assessed competencies, an a priori analysis of variance (ANOVA) is planned. To test the mean differences of 30 competencies between the 4 YG, a between-subject test will be used. To explore the association between the AM-TNA self-assessment scores and the level of postgraduate training (YG 1-4), we expect that mean scores will gradually increase over the years. In order to correct for the risk of false positives resulting from repeated measurement and to limit the instance of false positive results (Type-I errors), we will execute a Bonferroni-adjusted post hoc analysis: a series of t tests on each pair of classes using a corrected p value.
All competencies except # 6 had F values well above 1, indicating that in-between group variance was much more important than with-in group variance. The Bonferroniadjusted post hoc analysis of 6 pairs and 4 YG is presented in Table 1. Bonferroni correction was α/n = 0.05/6 = p < 0.008 indicates that the mean scores of 19 of the 30 competencies differ significantly between one or more training years (Table 1, column 2). 11 clinical competencies (1, 2, 3, 6, 7, 10, 12, 15, 16, 18, 20) do not show significant statistical differences between the training groups, whilst all competencies but 6 and 10 showed a tendency in the predicted direction. Comparing the total of the clinical and non-clinical competencies resulted in significant differences (p < 0.001) between the 4 YG (resp. 1 < 4, 2 < 4 and 1 < 4, 2 < 4, 3 < 4). The gradual increase in marginal means of the combined clinical and non-clinical competencies over the 4 classes in training is visualized in Figure 1.

Conclusion
The cross-sectional study design fitted the preliminary evaluation need and the sample represented around 50% the psychiatrists in training in Lithuania. We found a significant difference in mean scores between the 4 YG of psychiatrists in training. This indicates that the AM-TNA scale can measure systematic group differences in perceived proficiency in the SUD core competencies.
Furthermore, an excellent response rate (79%) and no missing values supported the efficacy of the AM-TNA scale (Objective 1). The Bonferroni plots illustrated a gradual increase in marginal means of the combined clinical and non-clinical competencies in the 4 training groups. This suggests an association between the AM-TNA scale self-assessed learning (vertical axis) and cumulative training years (horizontal axis) (Objective 2). This explorative, cross-sectional study has limitations. The results are based on a small convenience sample, and recruiting respondents from a single training location (LUHS) may have introduced some selection bias into our study. The study design choice also means we cannot determine causality.
This study clearly confirms the relationship, between AM-TNA scale self-assessment scores and the number of years spent in psychiatry training. We may have missed some individual variability and some individual self-assessment of proficiency in SUD competencies may have plateaued or even declined. Finally, the AM-TNA is a selfreported scale, reflecting the respondents' perceived competency and not their objectively measured skills. This limitation would be concerning if self-ratings in the YG 1 were unusually high as it may indicate socially desirable responses. However, this was not detected in our study.
Competency-based assessment is gold standard in medical training. It is also worthwhile to note that selfassessment of medical competencies, despite its accuracy limitations, is an important component of lifelong professional learning, like continuous medical education and an established tool to promote both academic achievement and self-regulated medical learning [13].
Validated self-assessment tools are particularly useful in encouraging self-assessment of addiction caregivers' perceived SUD competencies, and as such stimulate the acquisition of addiction skills by specialized and non-specialized professionals. Despite the study's limitations, the AM-TNA self-assessment scale measured individual proficiency and produced significant group differences in perceived SUD competencies in a small local training context. Accuracy and external validity issues of self-assessment scales like the AM-TNA can be overcome when used as formative evaluation tools in combination and in support of external and objective evaluations and observations, critical appraisal, reflection reports. This study illustrates that the self-assessment AM-TNA scale can measure individual proficiency and significant incremental differences between mean scores of perceived proficiencies in addiction medicine core competencies in a 4 YG psychiatry training setting. We suggest it could also function well in larger-scale training settings. The AM-TNA scale might also support individual self-regulated learning and could be helpful to improve individual competencies. Future research should aim to investigate differences in larger sample sizes over time, assess individual progress, and focus on comparing and combining selfreported competencies with validated objective external assessments of learning, attitudes, and clinical practice.