Perception of the Use of the Direct Observation of Procedural Skills in Occupational Therapy Postgraduate Year Training in Taiwan: Survey of the Perspectives of Trainees and Supervisors

Introduction Assessment of clinical competence is a significant part of the training for young occupational therapists (OTs). Objective and systematic assessment allows both supervisors and trainees to be aware of the training objectives and monitor the progress. The direct observation of procedural skills (DOPS) is a work-based assessment to evaluate professional knowledge, skills, and attitude in clinical training. This study investigated the perspectives of OT educators and trainees on using DOPS and their discrepancy for OT postgraduate year (PGY) training. Methods This study used a quantitative online survey. Eighty-six supervisors and 41 trainees of OT PGY training programs from 95 hospitals returned the questionnaire (a 90.5% return rate), and 64 supervisors and 30 trainees who used DOPS were analyzed. Outcomes included the practicality in using the DOPS in clinical settings, the ease of rating the DOPS, and advantages and the disadvantages of the DOPS. Results Most respondents reported that completing one DOPS required at least 11 minutes for direct observation (11-40 minutes: teacher 92.2%; trainee 80.6%). Most respondents (teacher 96.9%, trainee 96.8%) had feedback after direct observation of DOPS, and about half of the feedback assessments took 5 to 10 minutes (teacher 53.1%, trainee 48.4%). Most OT educators and trainees agreed that clinical resources were sufficient and that DOPS matched with OT training goals, benefited OT competence training, and had a fair, objective, and consistent scoring system. Significantly higher percentages of OT trainees felt stressed in and satisfied with the DOPS assessment than trainers. Differences between teachers and trainees regarding easiness of rating DOPS items were not significant. Conclusion Most OT educators and trainees agreed that DOPS was a practical and appropriate assessment for OT PGY training.


Literature Review
Clinical education is the most important part of health professional education [1] and also one of the strategies for developing continuing competence in occupational therapy (OT) [2]. The postgraduate year (PGY) training program for general medicine was implemented by the Taiwanese government in 2003 [3] and for other health care professions in 2007, including OT [4], pharmacy [5], and nursing [6], to address a need for improved professional training and the quality of health care services. The programs are implemented by the Joint Commission of Taiwan under the supervision and instruction of the relevant professional associations to equip PGY trainees with (1) professional knowledge, the ability to provide quality care, evidencebased skills, and resource management ability; (2) patientcentered, comprehensive, and holistic perspectives and skills; (3) professional ethical reasoning and communication skills; and (4) the ability to work in a team [7,8].
Assessment of clinical competence is a significant part of the training for young OTs. Objective and systematic assessment allows both supervisors and trainees to be aware of the training objectives and monitor the progress. The assessment of practical performance in clinical conditions (does) is ranked as the highest level in Miller's hierarchical model for the assessment of clinical competence [9]. Direct observation of procedural skills (DOPS) is a workplace-based assessment tool specifically designed to assess practical skills. It requires the assessor to directly observe a trainee conducting a procedure, document the evaluation in a checklist, and give feedback to the trainee based on objective findings [10].
Several studies have reported the experiences of supervisors and trainees with the DOPS in health professional training, such as medical, nursing, and veterinary students, and showed that examiners and examinees were both very satisfied with the DOPS [10,11]. However, the feasibility and user experience of the DOPS for OT PGY training has not been examined. We investigated the perception of OT supervisors and trainees on the DOPS in Taiwan for PGY training. Therefore, the aims of our study were to investigate (1) the experience of using the DOPS in clinical settings, (2) the ease of rating clinical performance with the DOPS, and (3) the advantages and disadvantages of the DOPS for OT PGY training.

2.1.
Procedures. This is a cross-sectional survey study. Data were collected from January 2017 to December 2018. We invited the teaching directors of each hospital to participate in this project from January 2017 to June 2017. A link to the online questionnaire was sent from July 2017 to December 2017. As of January 2018, the response rate was not good. The second recruitment and first reminder were carried out from January 2018 to July 2018. Follow-up reminders were sent one and two months later. Information in the database of the Ministry of Health and Welfare, Taiwan, was used to invite OT chiefs in charge of OT PGY training programs in 2017 to participate in this study. The online questionnaire was sent by email along with regular postal mail to participants. Participants signed an informed consent to be included in this study, and their data coding and analysis were anonymized. In our first part of the electronic questionnaire, there was the background of the questionnaire, the purpose of the research, the use of data, and the informed consent form. After downloading the filled forms, any identified information was removed, and then the deidentified data were used for data coding and analysis. When the online electronic questionnaire URL was given, each respondent was asked to report a unique user number to be used while filing the survey, which was convenient for distinguishing the source and avoiding repeated collection of operations.

Instrument.
Prior to using DOPS for assessment, training included a 2 hours of general introduction to DOPS and a 2-hour scoring consistency workshop. This questionnaire was developed by two senior OTs (Y. C. Liu and P. C. Huang) based on the research questions and literature of medical education with work-based assessment in 2015 [11][12][13][14]. Five experts in OT PGY training were invited to assess and discuss the questionnaire in a focus group, and revised this questionnaire accordingly. A second group of five external experts in OT fieldwork education and research were invited to revise and finalize the questionnaire.
The final questionnaire had four sections. The first section included 11 items of basic sociodemographic information of the participants, including job position, age, sex, highest educational level, teaching/working years, specialty area, status of hospital, and service location. The second section used a 5-point Likert scale to rate 21 items related to experiences with using DOPS (i.e., practicality, subjective opinion, cover domain, and scoring system). The third section assessed trainer's and trainee's perception of ease of using the DOPS in rating the clinical performance of PGY OT trainees with a 5-point Likert scale of 11 items. The fourth part included 16 items of the advantages and disadvantages of the DOPS for the evaluation of OT PGY trainees.

Statistical Analyses.
Demographic characteristics and distribution of participants' responses in perception of DOPS are presented with descriptive statistics (i.e., frequency and percentage and mean and standard deviation). Comparisons between supervisors and trainees in experience of using DOPS, ease of rating the DOPS, and advantages and disadvantages of DOPS were assessed with the χ 2 and t tests. Data were analyzed with SPSS 18.0 software (SPSS, Chicago, IL, USA).

Results
Completed questionnaires were returned by 86 supervisors and 41 trainees of the 95 hospitals (individuals) who received the invitational letter and survey by emails and regular mail for a 90.5% return rate. The response rate was the number of questionnaires returned divided by the number of that have applied for PGY teaching hospitals times 100%. Data from 64 supervisors and 30 trainees who had used the DOPS and completed surveys were included in the analysis. 37.2% of the participants were located in the northwestern Taiwan, 12.8% of the participants were located in the middle western Taiwan, 43.7% of the participants were located in the southwestern Taiwan, and 6.4% of the participants were located in the eastern Taiwan. The distribution of location of participants represents the population of OT in Taiwan. The demographic characteristics of the participants are presented in Table 1. Significantly more women, being younger, and lower levels of education were noted in the trainee group than in the supervisor group. There were no differences in specialty areas and classification of hospital in the two groups.
Regarding experience with DOPS, more than 70% of trainees (range: 72.4% to 93.3%) and supervisors (range: 71.3% to 85.9%) responded "strongly agree" or "agree" with most items except for teacher's rating of sufficient teachers (67.2%), assessing clinical reasoning ability (62.6%), consistency in scoring standard (61.0%), and both teachers' and trainees' rating of stressfulness in assessment (17.2% and 41.4%, respectively) and needing further training (54.7% and 55.1%, respectively). In addition, less than 10% of trainees (range: 0 to 6.8%) and supervisors (range: 0 to 9.4%) reported "strongly disagree" or "disagree" with most items except for teachers' rating of need further training (20.6%) and both trainees' and teachers' rating of stressfulness in assessment (10.3% and 23.5%, respectively). The use experience of DOPS of the trainees and supervisors and their comparisons are presented in Table 2.
Results of the χ 2 tests indicated that there were significant differences between trainees and teachers on most items of experiences of using the DOPS for OT PGY training except for items of low cost, better than traditional teaching assessment, fairness, objectiveness, and consistency in scoring standard. Significantly higher percentages of "strongly agree" were noted in trainee's perception (range: 20.7% to 41.4%) than in supervisor's perception (range: 1.6% to 12.5%).
Most respondents reported "simple" (trainee range: 10.7% to 34.5%; teacher range: 23.0% to 35.0%) or "neutral" (trainee range: 17.2% to 53.6%; teacher range: 27.9% to 47.5%) in the easiness of rating most of the DOPS in clinical settings. Results of χ 2 tests indicated there were no significant differences between trainees and teachers on all items of easiness of rating DOPS for OT PGY training program. The easiness of rating DOPS as reported by the trainees and supervisors and their comparisons are presented in Table 3.  The results of the χ 2 tests revealed no significant differences on perceptions in advantages and the disadvantages of DOPS between trainees and supervisors. Table 4 presents the comparison of advantages and disadvantages between trainees and supervisors.

Discussion
To the best of our knowledge, this is one of the few studies to examine the feasibility of the DOPS and explore the use experience, advantages, and disadvantages of the OT PGY training programs by including the perspectives of both trainees and supervisors. The results found that the DOPS tool was a practical measurement tool to assess the competence of OT PGY trainees. Time for implementing DOPS and providing feedback is a significant factor for feasibility and acceptability [11]. Some studies [15,16] found a lack of time or being seen as an additional workload in administrating the DOPS and giving feedback in physician training. This may reflect busy the clinical hours of the physicians and time constraints for using the DOPS in the clinical training of medical doctors. In contrast to findings of the above studies [11,15,16], approximately 90% of the respondents in this study spent at least 11 minutes for rating DOPS, and half of the respondents reported to have at least 6 to 10 minutes for feedback. Most of the trainees (86.7% agree or strongly agree) and supervisors (75% agree or strongly agree) agreed time for implementation of DOPS was sufficient for OT PCY training. In addition, more than 75% of trainees and supervisors agree that immediate feedback was one of the advantages of using DOPS in OT PGY training.
Training of rating DOPS is another significant issue for consideration in the assessment for OT PGY training. Half of the trainees (55.1%) and supervisors (54.7%) claimed that they need further training in using DOPS. Proper implementation is one of the significant factors related to the educational impacts of administrating DOPS [17] and relies on sufficient training in using assessment tools such as DOPS.
Approximately 40% of supervisors (35.9% neutral, 1.6% disagree, and 1.6% strongly disagree) did not support consistency in the scoring standard for rating DOPS in this study. Intercase variation may affect the reliability [11]. Training is needed not only on the procedure of administrating DOPS but also on scoring training on different patients or conditions. In addition, a study suggested rating of trainees' performance by at least three supervisors observing at least two procedures each to achieve good reliability [11].
Consistent with previous studies [10,11], some trainees (41.4% agree or strongly agree) felt that taking DOPS is a stressful experience. However, the trainee's satisfaction with DOPS is high and even higher than the supervisor's. It is 5 Occupational Therapy International possible that attending a work-based examination such as the DOPS is stressful, but the trainees appreciate this experience as learning and positive although stressful experience.

Implications for Occupational Therapy Education.
The DOPS is a practical assessment tool for OT PGY training to monitor the trainee's knowledge, skill, and attitude. Sufficient time for examination and feedback with DOPS is an essential component for successful implementation. Training for using DOPS is needed not only for the implementation procedure but also for scoring criteria across patients and conditions.

Strengths and Limitations.
The strength of this survey included a nationally representative sample and a high response rate (more than 90%). Participants included OTs from different areas of Taiwan, clinical specialty areas of OT, and hospital settings.
The limitations included the following. This study recruited one trainer who was the major supervisor of the trainee during the period of study from each hospital. Other therapists who had once been supervisors before the study period only were excluded. Selection bias might exist. The experience of using the DOPS might be subject to recall bias because this survey was not filled out immediately after any specific DOPS assessment.

Future Study.
In addition to trainees and supervisors in the OT PGY training programs, the DOPS is also used for the fieldwork education of senior OT students and their clinical supervisors. Future studies may investigate the user experiences of the senior OT students and their clinical supervisors in fieldwork education.

Conclusion
Most OT educators and trainees agreed that the DOPS is a practical and appropriate assessment for the OT PGY training. Sufficient time for rating the DOPS and feedback about performance is essential to learning effects. Training in using the DOPS is necessary and should include rating criteria with different conditions and patients with the same procedures.

Data Availability
The data sets generated and analyzed during the present study are not publicly available due to participant confidentiality but are available from the corresponding author on reasonable request.

Ethical Approval
The study protocol was reviewed and approved by the Ethical Committees for Human Research of the National Cheng Kung University Hospital in Taiwan. The study procedures, risks, and benefits were explained to all participants.

Consent
The participants signed an informed consent before enrolling in this study.

Disclosure
A preprint has previously been published [18].

Conflicts of Interest
The authors declare that they have no competing interests.