Implementing work place based assessment: The modified direct observation of procedural skills (DOPS) across medical specialties — An experience from a developing country

Objective: To assess the skill level of residents regarding central venous catheterisation insertion, and to assess the reliability of scores in a simulated situation. Methods: The quasi-experimental study with pre-and post-test design was conducted from February to June 2013 at the Aga Khan University, Karachi, and comprised four workshops attended by residents. The workshops were video-recorded for feedback and self-assessment. At the end of the workshops, knowledge and procedural skills were assessed using a self-generated 38-item, task-specific instrument after ensuring its content validity. Data was analysed using SPSS 19. Results: There were 40 residents in the sample. The self-generated instrument was reliable with Cronbach's alpha value 0.83 and inter-rater coefficient 0.79. There was significant improvement in the skills level post-intervention compared to the baselines mean values (p=0.001). The subjects were satisfied with the workshops, as indicated by a mean score of 8.83±1.367. Conclusions: The workshops appeared to improve the central venous catheterisation insertion skills of the residents.


Introduction
Workplace-based assessment (WPA) has been used to increase the clinical confidence of medical residents by increasing the level of realism of the assessment and providing a mechanism for formative feedback. 1rect observation of procedural skills (DOPS) of residents by the faculty remains a vital component of assessment across specialties.Assessment through observation provides ongoing data on resident performance with actual patients. 1Similarly, maintenance of professional competence is a lifelong process, and is motivated by a number of factors, including curiosity, self-identified gaps in knowledge, and the desire to provide the very best care to patients. 2PS focusses on observing and evaluating the procedural skills of postgraduate residents in a workplace setting.Central venous catheter (CVC) insertion is one of the commonly performed bedside procedures 3 by Internal Medicine (IM), Nephrology, Critical Care, Pulmonology, Cardiology, Anaesthesia and Emergency Medicine residents who are part of the Aga Khan University (AKU) postgraduate medical education residency programmes.Literature reports a 15% rate of complications associated with the procedure that include arterial puncture, blood steam infection and pneumothorax. 3However, attention towards the insertion technique and appropriate training with increased procedural experience have shown to reduce errors. 3e importance of appropriate CVC insertion training is recognised by professional societies, accreditation bodies and hospitals with residency training programmes across the globe with competence requirements and standards in place.The Accreditation Council for Graduate Medical Education (ACGME) has also stipulated that IM residents must show basic proficiency in CVC insertion. 4The American Board of Internal Medicine stipulates that residents should demonstrate competence and safe performance of CVC insertion by means of assessments performed during residency training. 5ntinuous, comprehensive and elaborative assessment and feedback are integral components of competencybased residency education with emphasis on WPA. 6 However, there are many challenges to overcome for successful, sustainable assessment and feedback strategies.This system of education ultimately requires a demonstration that the resident is truly competent to progress in training or at least to the next phase of a professional career. 7The competency demonstration and assessment require development of a valid and reliable assessment tool using direct observation coupled with feedback.Self-assessment by residents and intensive faculty development are also essential. 7Simulation-based mastery learning (SBML) is used to teach clinical skills.However, in Anaesthesia, the utmost value of such practice is expected to be with the most intricate and invasive procedures, like airway management, epidural puncture for spinal anaesthesia, CVC and lumbar epidural. 8e use of innovative assessment tools to assess procedural competence of residents in the clinical environment is challenging, time-consuming and labourintensive.Studies report successful teaching of CVC insertion on partial-task simulator designed mannequins.This also simplifies gathering validity-based evidence, which otherwise would require training of multiple experts. 9However, there is a dearth of studies to report assessment of the procedure through direct observation of a resident's performance on a mannequin and its effectiveness.
Residents at AKU Department of Medicine are inducted in six Medicine and Allied specialties, including IM, Gastroenterology, Neurology, Cardiology, Pulmonology and Nephrology.Currently there are no structured formative or summative system assessment methods being used to gauge the procedural skills of residents in any specialty.There is also an absence of set standards of performance over which to assess the residents' procedural skills progression in training.Each subspecialty has its own system where residents are evaluated either on a monthly or quarterly basis on a generic rating form.
Sufficient evidence is thought to be essential in making a judgment about the proficiency of the learner in any procedural skill.Competence requires more than just technical skills, like humanistic and team competencies.Therefore, it was thought to introduce a modified version of DOPS in order to make a judgment of the humanistic, technical and team competencies. 9Despite the importance of safe practice and assessments during the training of residents, there is a dearth of literature on specific measurement tools to assess the procedure.There is a need for a locally developed tool that could test procedural skills in an objective manner. 10,11The current study was planned to do that among residents performing CVC insertion in a simulated situation.

Subjects and Methods
The quasi-experimental study with pre-and post-test design was conducted from February to June 2013 at the AKU Department of Medicine, Karachi.Kirkpatrick's training evaluation model 12 was used as the theoretical framework.The reaction and learning levels of Kirkpatrick's model were evaluated related to the teaching and assessment of safe and accurate insertion of CVC as a pilot project in order to establish DOPS methodology.Ethical approval was obtained from the institutional ethics review committee.A project team of six content experts through an iterative process worked to develop a 38-item, task-specific DOPS instrument relevant to local and contextual needs based on literature and already in-use checklist for central line insertion at AKU. Focus group discussions (FGDs) were held with content experts from Pulmonary Care, IM, Nephrology and Critical Care who reviewed and modified the form based on evidence-based best practices with an amalgam of the principles of objective structured technical skills (OSATS) based on their extensive experience of performing and supervising the procedure.The instrument included components of technical skills, communication skills and feedback.The instrument was piloted on a group of residents for any possible revisions.The modified DOPS (mDOPS) instrument was further revised on the basis of the findings of the pilot study.
A faculty refresher workshop was conducted to build the capacity of the faculty involved in performing the procedure to assess the residents' performances on a mannequin through mDOPS.
Four workshops were conducted to assess the competence of residents in CVC insertion on a mannequin.The sample size estimation was based on expert judgment of the faculty members with the specific purpose of correct use of CVC insertion technique.Residents from IM, Nephrology, Pulmonology, Cardiology, Critical Care and Emergency Medicine were included using non-probability purposive sampling technique.Each workshop had 10 residents from level 2 to 5 after furnishing informed consent.
The residents went through a training process consisting of video demonstration, didactic lectures, faculty demonstrations on a mannequin followed by hands-on training process on mannequins and direct observation and assessment by expert faculty on mannequin with a questionnaire-based post-test.Knowledge and procedural skills were assessed while being videorecorded during the workshop for feedback and selfassessment.Residents' acceptability and feasibility of the process was evaluated via a questionnaire.In case a resident failed to perform adequately on a mannequin, s/he was required to go through the training again on the mannequin.Since this was a formative process and meant for learners' improvement in performance based on feedback, there was no effect on residents' evaluations.Data was analysed using SPSS 19.Descriptive analysis was done for mDOPS data.Mean ± standard deviation was used for quantitative variables, while frequencies and percentages were used for qualitative variables.The mean change scores from baseline to postintervention was compared using paired t-test.P<0.05 was considered significant.
Descriptive data analysis was done regarding the feasibility and satisfaction questionnaire.The generalisability theory was used to model the reliability of scores with different number of assessors.

Results
There were 40 residents in the sample.The mDOPS instrument was reliable with Cronbach's alpha value 0.83 and inter-rater coefficient 0.79.There was significant improvement in the skills level post-intervention compared to the baselines mean values; from 10.1±4.2 to 12.6±3.1 (p=0.001).
The subjects were satisfied with the workshops, as indicated by a mean score of 8.83±1.367out of a maximum possible 10 (Figure-1, Table ).
Residents agreed to commit over the next three years to improving and enhancing their CVC insertion skills through WPA (Figure -2).

Discussion
Professional practice requires effective clinical decision-making and patient safety measures.However, assessing these elements is difficult. 13Literature reports that a structured tool in assessing the observed residents' skills improves accuracy of observation. 14DOPS is now being used in a number of programmes across the globe to  Table : Participants' satisfaction survey results.

Items N Mean±SD
assess the junior residents' performance. 15In some training programmes it has been reported to replace the existing logbooks for the assessment of procedural skills. 16,17r findings showed that mDOPS improved the performance of the residents in a simulated situation.The residents were satisfied with the assessment of the procedure through direct observation and reported it to be effective in meeting patient safety, and in enhancing their confidence, and showed willingness to participate in the next phase of real-life setting and its implementation as an assessment tool.Furthermore, the generalisability coefficient 18 shows that the differences detected between residents; knowledge component and skills demonstration component on the assessment tool were real differences.We were able to report the first two levels of Kirkpatrick's model of resident reaction and learning to Literature reports conceptually relatively modest relationship between Kirkpatrick's learning criteria and behavioural criteria 19,20 which may be due to less opportunities provided for the learnt skills to be demonstrated in real-life context or ineffective observation.Therefore, enough opportunities need to be provided to the residents in order to collect and interpret behavioural data related to direct observation in order to report the behaviour and results criteria of Kirkpatrick's model of training evaluation.
The current study has limitations as residents from specialties, like Anaesthesia and General Surgery, where CVC insertion is carried out were not included.
After having assessed and certified residents' core knowledge and procedural skill in a simulated environment, competency on CVC Insertion will be assessed in the next phase of this ongoing study.There is a need to implement the simulation-based training for CVC insertion across specialties and include other speciality residents for credentialing the residents for safe practice in CVC insertion.Furthermore, the practice may be used for other residency programmes within the country to reduce complications related to CVC insertion.

Conclusion
Acquisition of skills requires both day-to-day learning opportunities and supervision of opportunities.Development of a reliable and valid instrument for assessing workplace-based performance of procedural skills is crucial.Adopting strengths of DOPS and OSATS was beneficial in assessing CVC insertion.The evidencebased mDOPS provided a standard for assessing the procedural skill of central line insertion in simulated and real-life situation which can be used across specialties.Further, mDOPS instruments could be developed for other procedures performed by IM residents.
Limitation: Submission was after 7 years due to unavoidable circumstances.

Disclaimer:
The text has been presented as a project on Research Day at the AKU Department of Medicine.
provide the educational effectiveness of the training programme which is the internal criteria and most commonly assessed.Learning criteria are measures of the learning outcomes, assessed by 'pre' and 'post' knowledge tests, and skill demonstration in the training context.However, the results and behaviour criteria, which are external and focus on changes that occur after the training programme12  will be assessed later by immediate post-training measures of performance in reallife patient encounters (Figure-3).

Figure- 3 :
Figure-3: Flow-chart and plans for the future.