Development of a national point-of-care ultrasound training course for physicians in Japan: A 3-year evaluation

Purpose: Point-of-care ultrasound (POCUS) allows bedside clinicians to acquire, interpret, and integrate ultrasound images into patient care. Although the availability of POCUS training courses has increased, the educational effectiveness of these courses is unclear. Methods: From 2017 to 2019, we investigated the educational effectiveness of a standardized 2-day hands-on POCUS training course and changes in pre- and post-course exam scores in relationship to participants’ (n = 571) clinical rank, years of POCUS experience, and frequency of POCUS use in clinical practice. Results: The mean pre- and post-course examination scores were 67.2 (standard deviation [SD] 12.3) and 79.7 (SD 9.7), respectively. Higher pre-course examination scores were associated with higher clinical rank, more years of POCUS experience, and more frequent POCUS use (p < 0.05). All participants showed significant changes in pre- to post-course exam scores. Though pre-course scores differed by clinical rank, POCUS experience, and frequency of POCUS use, differences in post-course scores according to participant baseline differences were non-significant. Conclusion: A standardized hands-on POCUS training course is effective for improving POCUS knowledge regardless of baseline differences in clinical rank, POCUS experience, or frequency of POCUS use. Future studies shall evaluate changes in POCUS use in clinical practice after POCUS training.


Introduction
Point-of-care ultrasound (POCUS) is a focused examination where ultrasound images are acquired, interpreted, and integrated into patient care by a clinician at the bedside.POCUS emerged in the 1990s as a useful bedside tool to guide clinical management.Since then, its use has expanded across nearly all specialties and includes a broad range of applications.In recent years, access to portable ultrasound machines has improved substantially as handheld, pocket-sized devices with high performance and lower cost have become available (Soni et al., 2020).
Despite the growing body of evidence supporting POCUS use and increasing availability of POCUS devices, lack of training continues to be the primary barrier to POCUS use among physicians in-practice (Nathanson et al., 2023;Wong et al., 2020).Societies including the Society of Hospital Medicine, American Collage of Chest Physicians, and American College of Physicians have published guidelines and developed POCUS training programs for physicians in-practice (American College of Physicians, 2023;Greenstein et al., 2017;Society of Hospital Medicine, 2023;Soni et al., 2019).Hands-on POCUS training courses have also gradually begun to spread in Asia (Yamada et al., 2018;Yamada, 2020).POCUS training standards have been developed for some specialties, but there is neither a broadly accepted standardized curriculum nor a national post-graduate certification accepted across all specialties (Singh et al., 2022;Soni et al., 2020).
Most POCUS training courses are attended by clinicians of different specialties and with varying backgrounds with respect to clinical rank and prior POCUS experience (Yamada et al., 2018).There are few POCUS training courses that have objectively evaluated the effectiveness of the course (Singh et al., 2022), and it is not clear whether these courses are effective in improving POCUS skills of participants from different backgrounds.The purpose of this study was to examine the effectiveness of a standardized POCUS training course on POCUS skills of medical students, residents, and faculty physicians with differences in baseline clinical rank, years of POCUS experience, and frequency of POCUS use.

Design
We held eleven 2-day, hands-on POCUS training courses in Japan from 2017 to 2019.The course was held three times in 2017 (in Chiba, Gifu, and Aichi prefectures) and four times in 2018and 2019(in Chiba, Aichi, Gifu, Kanagawa in 2018and in Fukuoka, Aichi, Saitama, Tokyo in 2019).The course was modeled after the Society of Hospital Medicine and American College of Chest Physicians POCUS courses and was accredited by the Japanese Society of Hospital General Medicine and the Japanese Association for Acute Medicine (Yamada et al., 2018).
All participants completed internet-based modules on general principles of ultrasound and operation of an ultrasound machine before attending the hands-on course.The course started with a pre-course exam, followed by live lectures and hands-on sessions covering focused cardiac ultrasound (FOCUS); lung, vascular, and abdominal ultrasound; shock assessment, and multiple case studies, followed by a post-course exam (Figure 1).The hands-on sessions involved 3 participants and 1 instructor (3:1 ratio) per skill station with a live model.All instructors passed a qualifying instructor examination, and an instructor handout was used to standardize instructional content and terminology.During the hands-on sessions, a course director circulated between stations to ensure consistency and quality of instructional content.The content of the hands-on sessions is shown in Table 1.
All course participants completed a pre-and post-course written examination that was administered live at the start and finish of each course.The examinations included 30 multiple-choice questions on fundamentals of ultrasound, FOCUS, lung, lower extremity vascular, and abdominal ultrasound.Each question had an ultrasound video.The first 20 questions tested simple image interpretation (3 points per question), and the remaining 10 questions were case based to test clinical integration (4 points per question).The pre-and post-course examinations were different, but the question objectives were the same.The change in pre-and post-course examination scores was used to evaluate educational effectiveness of the training course.Pre-and post-course examination scores were evaluated by clinical rank, years of POCUS experience, and frequency of POCUS use.

Participants
Course participants included faculty physicians, residents, and medical students in 5th year and above.We posted emails inviting participants to attend the course on the mailing lists of various academic groups, including the Society of Hospital General Medicine, the Japan Primary Care Association, and freestanding academic groups for hospitalists, primary care physicians, intensivists, emergency physicians, and others.Individuals were invited to participate on a first-come, first-serve basis.No restrictions were placed on years in practice, clinical rank (junior resident, senior resident, or faculty), or departmental affiliation.In Japan, national training regulations require junior residents to complete a 2-year multi-specialty internship with rotations in internal medicine, surgery, emergency medicine, and other specialties after graduation from medical school.During this stage, they are known as junior residents.Afterwards, each resident receives 3 years of training in their specialty of choice, during which they are known as senior residents.After they complete their senior resident training and pass their medical specialty examination, they become an attending physician or specialist.Physicians who had completed their senior resident training were defined as faculty in this study.Medical school is 6 years in Japan.After passing computer-based knowledge testing and an Objective Structured Clinical Examination (OSCE) at the end of the 4th year, medical students participate in clinical clerkships during the 5th and 6th years.Our POCUS training course was open to 5th-and 6th-year medical students.

Data collection
All course participants completed a pre-course questionnaire about their department, years in practice, clinical rank, years of POCUS experience, and frequency of POCUS use.The pre-course exam was administered at the beginning of each course, and answer sheets were collected immediately.The exam was scored by administrative staff, but the scores and correct answers were not given to the participants.The post-course exam was administered at the end of each course, and afterwards, answer sheets were collected, and correct answers and explanations were provided for each question.
Participants were not allowed to take a copy of the examination home.

Ethical approval and consent
The  2.

Pre-and post-course examination scores
The overall mean pre-and post-course examination scores were 67.2 (standard deviation [SD] 12.3) and 79.7 (SD 9.7), respectively.A score breakdown by clinical rank, years of POCUS experience, and frequency of POCUS use is shown in Table 3.The post-course scores were significantly higher than the pre-course scores in all groups (p < 0.01) (Figure 3).

Pre-course examination score analysis
Pre-course examination scores were compared by clinical rank, years of POCUS experience, and frequency of use (Figure 4).There were significant differences between students     and senior residents (p = 0.03), students and faculty (p < 0.01), junior residents and senior residents (p = 0.03), and junior residents and faculty (p < 0.01).There were also significant differences between all three groups by years of POCUS experience and between two groups by frequency of POCUS use (p < 0.01).
Next, we compared pre-course scores by years of POCUS experience and frequency of POCUS use, stratified by clinical rank (Figure 5).In the group with no ultrasound experience (n = 115), the pre-course exam scores were 62.0 (SD 13.3) for medical students, 62.5 (SD 10.9) for junior residents, 67.4 (SD 14.2) for senior residents and 65.5 (SD 10.1) for faculty.There were no significant differences between any of the groups.When comparing groups by frequency of POCUS use, there were significant differences in the group using ultrasound less than once per week (n = 371) between medical students and faculty (p = 0.04) and junior residents and faculty (p < 0.01).In the group using ultrasound at least once per week (n = 200), there was a statistically significant difference between junior residents and faculty (p < 0.01).
Analysis of differences in pre-and post-course examination scores The mean overall difference in pre-and post-course examination scores was 12.5 (SD 11.6) (Figure 6).Although there was a significant increase from pre-to post-course exam scores in all groups (Figure 3), there were no statistically  significant differences in the change of pre-and post-course exam scores between any of the groups by clinical rank or years of POCUS experience (Figure 6).The change in exam scores for those using ultrasound less than once a week versus more than once a week were 13.3 (SD 12.3) and 11.0 (SD 10.1) which was statistically significant (p = 0.03).
Additional analyses were conducted comparing change in preand post-course exam scores by years of POCUS experience and frequency of POCUS use, stratified by clinical rank.These analyses did not find any statistically significant differences between any of the groups (Figure 7).

Discussion
We investigated the educational effectiveness of a standard-   POCUS findings are integrated with other clinical findings to make clinical decisions, and therefore, both clinical experience and ultrasound skills are important.Generally, it is well accepted that higher clinical rank signifies greater clinical experience.However, ultrasound skills cannot be presumed to be better with higher clinical rank, and on the contrary, trainees are more likely to have been exposed to ultrasound during medical school or residency training and be better skilled in POCUS.In our study, faculty with more years of POCUS experience and who used POCUS frequently had higher pre-course scores.However, there was no difference in pre-course exam scores by clinical rank for participants with similar levels of ultrasound experience.These data suggest that years of POCUS experience may have more influence on POCUS skills than clinical rank.
This study has limitations.Competency in POCUS requires mastery of basic ultrasound knowledge and skills in image acquisition, image interpretation, and clinical integration.Our pre-/post-course examination assessed basic ultrasound knowledge, image interpretation, and clinical integration skills, but did not assess image acquisition skills.Additionally, we were not able to assess knowledge or skills retention post-course.A strength of our study is the large number of participants (n = 571) from whom we collected data which is greater than other similar studies.
Future studies shall track course participants longitudinally to assess POCUS skills and knowledge retention several months post-course, as well as evaluate incorporation of POCUS use into clinical practice.Elucidating barriers and facilitators to POCUS use in clinical practice will improve our understanding of how to systematically implement and standardize POCUS use across healthcare systems.Finally, a basic POCUS skill certification process is not currently in place in Japan, and developing such a certification will help maintain standards and promote safe POCUS use in different clinical settings.Responses to each pre-and post-course exam question cannot be shared publicly because the results were recorded on paper media and transcribing them into digital format in a sharable format at this point is not feasible.The "POCUS raw data revised 2.xls" file contains individual level scored data.

Conclusion
All the raw data (before undergoing scoring) can be made available by the authors to researchers/reviewers, upon reasonable request.Please contact the corresponding author Taro Minami to request access.
The additional data that cannot be shared includes the study participants' personal identifiers.As required by ethical guidelines and privacy laws, these data have been securely stored and protected to uphold the participants' confidentiality rights.Sharing this data type would violate our commitment to participants and potentially infringe on legal obligations.

Extended data
The exam questions cannot be disclosed because the course is still being held as an educational course, and the questions should be protected.The questions will be made available by the authors to researchers/reviewers, upon reasonable request.Please contact the corresponding author Taro Minami to request access.

John Stephens
University of Sunderland, Sunderland, UK Thank you for the invitation to review the article.Overall the work is contemporary; an enjoyable and informative article that has great potential to direct further research of the subject area for a wider audience as well as the research team.This is articulated by the team at page 9, paragraph 5.As well as the proposed longitudinal work, research involving qualitative methodologies could, for example, explore the learning processes and application to practice for different professions, and differing levels of experience.
The abstract accurately conveys the nature of the research providing a useful summary for anyone sifting evidence as part of literature search of the subject area.The introduction and literature although very succinct does provide a clear rationale for the work undertaken.
Methods are reported in a transparent fashion, with a wide range of participants which I feel adds to interest for the research.Statistical analysis is succinctly and accurately reported.This section would have possibly been enhanced further through inclusion of which members of the team had carried out the analysis.
Results are fairly comprehensive and clearly presented with very good use of tabulation and charts.Although a quite 'busy' section the accompanying text provides a valuable report to inform a balanced discussion and an appropriate conclusion.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound?Yes

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?Yes

Have any limitations of the research been acknowledged? Yes
Are all the source data underlying the results available to ensure full reproducibility?Yes

Are the conclusions drawn adequately supported by the results? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: education, healthcare professional education, curriculum design, cardiorespiratory physiotherapy.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.Importantly, though the study is well conducted, a crucial element not evaluated is the ability of learners to acquire images themselves, which is a key feature of POCUS.As this cannot be reported in the current study, future studies should evaluate similarly the baseline skill and image acquisition ability of various learners though a national standardized training course.
What would be interesting to report is the type of question missed / gotten wrong on the assessments by level of training and/or rank and how they changed post-training.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound?Yes

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?Yes

Have any limitations of the research been acknowledged? Yes
Are all the source data underlying the results available to ensure full reproducibility?Yes

Are the conclusions drawn adequately supported by the results? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Point of care ultrasound, data science I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Michael Janjigian
New York University Grossman School of Medicine, New York, USA Overall this is an excellent manuscript which contributes to our understanding of POCUS training programs.I have a few minor points of clarification and to correct what seems like an error in Table 3.
The introduction is clearly written with a clearly defined study objective.
Consider adding a cost analysis of the course given your recommendation to expand this course nationally to resource limited areas.
Given that the results of this study revolve heavily around self-reported years of POCUS experience (Novice, <2 year, >2 year) and frequency of use, I would want more background on how these questions were framed in the survey or perhaps an explanation of why participants of an introductory course would have prior experience or use POCUS routinely.
For frequency of POCUS use, is it possible participants reported ultrasound guided venipuncture or IV access or other common procedures?Do students or residents receive any formal or informal POCUS training apart from this course?
The bar graph in Figure 3 shows the pre-test scores for frequency of use reversed compared to Table 3, there appears to be an error.Figure 4 matches Figure 3 so I suspect the error is in Table 3.
The discussion includes medical students as physicians which would not be defined this way to my understanding.Consider changing the language accordingly.
"Second, pre-course exam scores differed significantly by clinical rank, ultrasound experience, and frequency of use, but the improvement in post-course exam scores did not differ significantly, signifying the training course was beneficial to give all learners a similar level of POCUS knowledge post-course, regardless of their baseline POCUS knowledge."Did you test the difference in post-test outcomes across groups or just the change from preto post-(absolute or the change)?If the latter, which is how I read your report, then this statement should be reworded to focus on the change in knowledge, not the final level.Or report the statistics on the final test score across groups which visually appears to be similar.

○
As mentioned above regarding resources required to run the program, paragraph 2 of the discussion comments that resources are required to run a 2-day course but does not explain how expanding this course broadly (students, residents, faculty, across the country) would address the resource problem apart from providing a standardized curriculum.
An additional limitation is that you are using a test as a surrogate for POCUS competency rather than a hands-on test or OSCE.Clinical integration is incompletely assessed through a test question.Though I agree that a test is a reasonable surrogate, it would be worth commenting on limitations in assessments.Another limitation is that much of the quantitative data is from selfreported experience and frequency of use, which is not well characterized in the manuscript and fundamentally qualitative in nature.
In the conclusion, you state that "all participants" improved their knowledge.You have shown that the cohort as a whole improved knowledge rather than each individual.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound?Yes

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?Yes

Have any limitations of the research been acknowledged? Yes
Are all the source data underlying the results available to ensure full reproducibility?Yes

Are the conclusions drawn adequately supported by the results? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Point-of-care ultrasound programmatic development and assessment I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Figure 1 .
Figure 1.Point-of-care ultrasound training course schedule.
study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of the Tokyo Bay Urayasu Ichikawa Medical Center (protocol code 265, date of approval: July 20th, 2017) and Tokyo Medical and Dental University (protocol code M2019-085, date of approval: August 27th, 2019).Written informed consent was obtained from all participants involved in the study.Data from participants who did not consent to participate in the study or did not complete the training course, course questionnaire, or pre-and post-course exams were excluded.Statistical analysis A paired t-test was used to compare the pre-and post-course examination scores.Student's t-test was used to analyze the pre-and post-course examination scores and the change in scores by frequency of POCUS use.One-way analysis of variance and Bonferroni correction were used to analyze the scores and changes in scores by years of POCUS experience and clinical rank.Comparisons were also made by years of experience and frequency of use, with groups stratified by clinical rank.Data analysis was performed using STATA 17.0.Results A total of 630 learners participated in 11 POCUS training courses from 2017 to 2019.No participants had previously attended a POCUS training course.Of the 630 participants, 59 were excluded for not completing both the pre-and post-course examinations (n = 49) or the pre-course questionnaire (n = 10).Data were analyzed from 571 participants in total from across Japan (Figure 2).Participant characteristics are shown in Table

Figure 2 .
Figure 2. Distribution of course participants by prefecture across Japan.

Figure 3 .
Figure 3. Mean pre-and post-course exam scores by clinical rank, year of POCUS experience, and frequency of POCUS use.Abbreviation: POCUS, point-of-care ultrasound.

Figure 4 .
Figure 4. Comparison of pre-course exam scores by clinical rank, years of POCUS experience, and frequency of POCUS use.Abbreviation: POCUS, point-of-care ultrasound.

Figure 5 .
Figure 5.Comparison of pre-course exam scores by years of POCUS experience and frequency of POCUS use, stratified by clinical rank.Abbreviation: POCUS, point-of-care ultrasound.

Figure 6 .
Figure 6.Comparison of change in pre-and post-course exam scores by clinical rank, years of POCUS experience, and frequency of POCUS use.Abbreviation: POCUS, point-of-care ultrasound.
ized 2-day hands-on POCUS training course developed for physicians across Japan and the change in pre-and postcourse exam scores in relationship to participants' clinical rank, years of POCUS experience, and frequency of POCUS use in clinical practice.Data were collected from a total of 571 physicians across 11 courses from 2017 to 2019, the largest collection of data from a POCUS training course to date.There were three key findings of our study.First, the course was effective in educating participants regardless of baseline differences, including clinical rank, years of

Figure 7 .
Figure 7.Comparison of change in pre-and post-course exam scores by years of POCUS experience and frequency of POCUS use, stratified by clinical rank.Abbreviation: POCUS, point-of-care ultrasound.

A
standardized, hands-on POCUS training course significantly improved POCUS knowledge among all participants.Although pre-course exam scores differed by clinical rank, years of POCUS experience, and frequency of POCUS use, these factors did not impact the course's educational effectiveness based on post-course exam scores.Upon completion of the POCUS training course, all participants had a similar level of POCUS knowledge despite baseline differences.Future studies shall evaluate POCUS skills and knowledge retention and incorporation of POCUS use into clinical practice several months post-course.This project contains the following underlying data: -POCUS raw data revised 2.xls -Codebook 0518.xlsx(coding keys and pre-course questionnaire items) Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Reviewer Report 31
January 2024 https://doi.org/10.21956/mep.21080.r35359© 2024 Bhasin A. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Ajay BhasinNorthwestern University Feinberg School of Medicine, IL, USA Well written article describing the utility and effectiveness of a national point of care ultrasound training course for physicians in Japan.It describes nicely the methods used to train learners, differences between baseline capabilities and, interestingly, convergence of post-training capabilities to nearly the same level of POCUS knowledge.

Table 3 . Mean pre-and post-course examination scores.
***Senior resident: physicians in postgraduate years 3 to 5 pursuing specialty training after completing junior residency ****Attending physicians after completion of senior residency Abbreviations: POCUS, point-of-care ultrasound; SD, standard deviation.