Implementing Entrustable Professional Activities at Yong Loo Lin School of Medicine Singapore Undergraduate Medical Education Program

Entrustable professional activities (EPAs) have emerged as a useful and feasible way to put outcomebased competency frameworks into practice. It has given curriculum planners a tool to signpost the expected outcomes of a trainee as well as for the trainers to effectively evaluate if the student/resident is attaining the relevant competencies at the right time. Work on EPAs is well underway in graduate medical education and some allied health professions. However, currently, there is a paucity of published work in this area in undergraduate medical education. We describe a transferrable approach to feasibly document EPAs in an undergraduate medical education curriculum in Singapore and discuss limitations and challenges faced when developing EPAs in this setting.


Introduction
Over the past decade, health professional training institutions across the globe have been involved in transforming both undergraduate and postgraduate programs from process-oriented training to competencyoriented training built on outcome-focused curricula. This shift aims to enhance the quality of care given by health professionals through providing clearly defined, demonstrable sets of cognitive, psychomotor and affective skills (Epstein & Hundert, 2002;Frank & Danoff, 2007;Miller, 1990). However, the shift towards competency-based education has not been without problems. One example is the clear gap between what is known and what is practiced.
In addition, defined outcomes and competencies are, at times, confined to the curriculum maps and booklets instead of being demonstrated by the graduating class (Brooks, 2009;Frank et al., 2010;Grant, 1999;Talbot, 2004;ten Cate & Scheele, 2007). To address challenges and limitations such as these, several methods of organizing curricula have emerged.
These include the use of entrustable professional activities (EPAs), competencies, milestones, and others. International standards describing and defining the various methods have recently been published (Group, 2012).
Amongst these methods, the use of entrustable professional activities (EPAs) has emerged as a leading strategy, especially in postgraduate medical education, to ensure that trainees actually do show the relevant and necessary set of skills during the course and at the completion of their training (Bowen, 2006;Ten Cate, 2005). EPAs are defined as professional activities or specific professional work that a student/trainee could be entrusted to perform without supervision or at different levels of supervision (Boyce et al., 2011;Carraccio & Burke, 2010;Mulder et al., 2010). Ten Cate defines clearly the characteristics of these "professional activities". They are important parts of the qualified health professionals" scope of work that require specific training in the relevant knowledge, skill and attitude domains. They are activities that These descriptions of EPAs apply to the efforts of specialist bodies to define the progression of a trainee from a novice to an "entrustable expert" in that specialty. Most start with the new medical school graduate and, with the end in mind, have addressed the question, "What activities should a trainee be allowed or expected to do independently at the end of training?" They also give expected stages of development towards mastery. All EPAs relevant to a particular professional program when assembled together demonstrates the competencies necessary for that profession.
Such EPA based descriptions of training and curricula, to date, involve mainly postgraduate and allied health settings (Jones et al., 2011;Mulder et al., 2010) The schemes described in the literature do not address the "ground state" expected of an entering medical trainee. In other words, none have explored the use of EPAs in helping to map the undergraduate medical education curriculum. An approach starting with undergraduate medical education will help focus the multiple clinical specialties involved in the undergraduate curriculum to train the "pluripotential graduate", rather than an all-too-common approach of training an "embryonic specialist" whilst the student is still an undergraduate.

Implementing EPAs in an undergraduate curriculum -solutions and success factors
Yong Loo Lin School of Medicine (YLLSoM) established in 1905 has undergone many curricular transformations. The curriculum change in 2006 shifted the design of the learning system towards an integrated systems-based structure with the focus on learning outcomes. The EPAs were developed to define the graduates" outcome capabilities (Samarasekera et al., 2015). Similar to many schools that have adopted a competency framework, the faculty members at YLLSoM of the National University of Singapore had difficulty mapping the holistic competencies of the newly adopted Accreditation Council for Graduate Medical Education -International (ACGME-I) to the practicality of daily workbased teaching, learning, and assessment. Aligning the clinical training outcomes with the teaching-learning and assessment activities with the ACGME-I outcome framework in a practical meaningful way was a considerable challenge for the medical school. Teachers, who were spread across multiple clinical teaching sites across the island of Singapore and involved in training students at different years of clinical training (i.e years 3-5), struggled to determine where to pitch their teaching. Students, likewise, were unsure of the level of mastery they should achieve at different stages in their training.
The use of EPAs was seen as a great way to address these concerns. At the start of implementation and documentation of the curriculum in terms of EPAs, it quickly became apparent that a simple, standardized approach was needed across the various specialties and for domain-independent skills.

Defining clinical problems and conditions that were relevant to training
At the start, a standardized vocabulary consisting of 106 patient-oriented problems was derived through use of a wiki, email discussions, and face-to-face discussions of the Undergraduate Curriculum Committee. Next, the undergraduate education director of each clinical department led the effort to select from the list problems that were felt to be core to the teaching of their discipline at undergraduate level. This involved input from department faculty members, clinical teachers from the various teaching hospitals as well as graduate medical educators (i.e.; residency faculty). For each problem selected, they would match several representative core differential diagnoses or conditions that they expected a student to be able to address. This list of problems and conditions represented the core but was not meant to be exhaustive or to limit student assessment.

Reference tables
The reference tables for problems and conditions were also centrally defined and standardized. These are generic reference tables that can be applied to all problems and all conditions. They contain competencies and the expected times line of achieving those competencies. There are 3 reference tables for problems (Table PR1, Table PR2, Table  PR3) and 3 reference tables for conditions (Table C1, Table C2, Table C3).
Each reference table for problems consists of several rows (Tables 1-3). Each row lists an expected component ability students are expected to be able to achieve during the course of their training. The columns in the table indicate the minimum level of skill acquisition expected by the end of a particular 7 year of training. Each subsequent column represents the next level of training, and the number within each cell represents the level of independence expected. The numbers in the cells are based on the Dreyfus model of skill acquisition (Dreyfus, 1980) with the number "1" assigned for "unable to perform", and "5" for "able to teach/supervise". Hence, there is guidance on the expected level of skill a student is to achieve for each component ability, as well as the time line in which to accomplish it. The only difference among the 3 reference tables for problems is in the expected level of skill acquisition and the time line in which to achieve it.  (Table PR1) Outcomes of approach to a problem

Minimum standard expected by the end of this
year's training Knows the pathophysiology of the problem 3 4 4 Able to recognise the problem based on symptoms/history 3 4 4 Able to recognise the problem based on physical examination findings 3 4 4 Able to reach appropriate diagnosis and differential diagnosis based on history (see curriculum document for list of diagnosis and differential diagnosis) 3 4 4 Able to reach appropriate diagnosis and differential diagnosis based on physical examination (see curriculum document for list of diagnosis and differential diagnosis) Able to order and correctly interpret the relevant basic/first line investigations 3 4 4 Able to order and correctly interpret complex investigations 2 3 3 Able to assess for and identify the etiology/complication/severity 2 3 4 Able to seek help/advice from appropriate senior and/or referral to the appropriate health service provider 2 3 4 Minimum Standard: 1= unable to achieve outcome; 2= requires a lot of guidance to achieve outcome; 3 = requires moderate amount of guidance to achieve outcome; 4 = able to achieve outcome with little or no guidance (entrustment); 5 = has the ability to guide/teach others. The same is true for the 3 reference tables for conditions (see Tables 4-6). In general, a problem that is classified as a PR1 (see Table  1) indicates the need for mastery of the various component activities earlier in the course of training, while a problem that is classified as PR3 (see Table 3) indicates many of the component activities would still be emergent by the end of medical school. This is similar for conditions and their classification as C1, C2 or C3.   (Table PR2) Outcomes of approach to a problem Minimum standard expected by the end of this year's training Knows the pathophysiology of the problem 2 3 4

PI/PII
Able to recognise the problem based on symptoms/history 2 3 4 Able to recognise the problem based on physical examination findings 2 3 4 Able to reach appropriate diagnosis and differential diagnosis based on history (see curriculum document for list of diagnosis and differential diagnosis) Able to reach appropriate diagnosis and differential diagnosis based on physical examination (see curriculum document for list of diagnosis and differential diagnosis) Able to order and correctly interpret the relevant basic/first line investigations 2 3 4 Able to order and correctly interpret complex investigations 2 2 3 Able to assess for and identify the etiology/complication/severity 2 3 4 Able to seek help/advice from appropriate senior and/or referral to the appropriate health service provider 2 3 4 Minimum Standard: 1= unable to achieve outcome; 2= requires a lot of guidance to achieve outcome; 3 = requires moderate amount of guidance to achieve outcome; 4 = able to achieve outcome with little or no guidance (entrustment); 5 = has the ability to guide/teach others.  (Table PR3) Outcomes of approach to a problem Minimum standard expected by the end of this year's training Knows the pathophysiology of the problem 2 2 3 Able to recognise the problem based on symptoms/history 2 2 3 Able to recognise the problem based on physical examination findings 2 2 3 Able to reach appropriate diagnosis and differential diagnosis based on history (see curriculum document for list of diagnosis and differential diagnosis) 2 2 3 Able to reach appropriate diagnosis and differential diagnosis based on physical examination (see curriculum document for list of diagnosis and differential diagnosis) 2 2 3 Able to order and correctly interpret the relevant basic/first line investigations 2 2 3 Able to order and correctly interpret complex investigations 2 2 3 Able to assess for and identify the etiology/complication/severity 2 2 3 Able to seek help/advice from appropriate senior and/or referral to the appropriate health service provider 2 3 4 Minimum Standard: 1= unable to achieve outcome; 2= requires a lot of guidance to achieve outcome; 3 = requires moderate amount of guidance to achieve outcome; 4 = able to achieve outcome with little or no guidance (entrustment); 5 = has the ability to guide/teach others. Minimum Standard: 1= unable to achieve outcome; 2= requires a lot of guidance to achieve outcome; 3 = requires moderate amount of guidance to achieve outcome; 4 = able to achieve outcome with little or no guidance (entrustment); 5 = has the ability to guide/teach others.  Minimum Standard: 1= unable to achieve outcome; 2= requires a lot of guidance to achieve outcome; 3 = requires moderate amount of guidance to achieve outcome; 4 = able to achieve outcome with little or no guidance (entrustment); 5 = has the ability to guide/teach others.  (Table C3) Outcomes of managing the condition Minimum standard expected by the end of this year's training Standard: 1= unable to achieve outcome; 2= requires a lot of guidance to achieve outcome; 3 = requires moderate amount of guidance to achieve outcome; 4 = able to achieve outcome with little or no guidance (entrustment); 5 = has the ability to guide/teach others.

Mapping problems and conditions to reference tables
Once members of the clinical department listed the relevant problems, they determined which level of the problem table was appropriate for the undergraduate medical student. Likewise with the associated conditions, each condition was paired with a level of the condition table (Table 7). The endproduct was a single tab on a spreadsheet that mapped the medical knowledge and patient care domains for that specialty in the undergraduate program.
Additionally, tables were created for domainindependent skills including professionalism, systems-based practice, practice-based learning and improvement, communication and teamwork, and healthcare maintenance and wellness promotion. These tables standardized domain-independent skills expectations across all departments and specialties.

Applications
The applications of the EPA documents as have been created at the YLLSoM were manifold. With 300 students per cohort, 7 teaching hospitals, and over 2000 clinical teaching faculty, standard-setting was challenging. This is likely true at most medical schools. These documents helped guide the clinical teachers on standard setting, what to teach and where to pitch for different levels of trainees. Furthermore, the documents served as an excellent benchmark to help guide students in self-study. Additionally, basic science teachers had an easier way of determining how to present their materials to maximize relevance for subsequent clinical encounters. In this way, the EPA documents became part of the curriculum map and when entered into a curriculum database, became part of the data set that is queried during curriculum planning and review.

South East Asian Journal of Medical Education
Vol. 9 no. 1, 2015 For our medical school, entrustable professional activities are currently not the basis for summative assessment, but could certainly help focus assessment areas. Additionally, these documents could be extended across the continuum of training to include graduate medical education (GME).

Challenges
The development and implementation of EPAs in the undergraduate setting had not been without its challenges. The term "entrustable" in medical schools was felt to pose a legal issue (Dijksterhuis et al., 2009;Kashner et al., 2010;Kennedy et al., 2008). Students, as nonlicensed professionals, cannot render care independently. As such, although they may reach a level of entrustability during their schooling for certain activities, they could not be entrusted to carry out these activities without adequate supervision. Hence, we had to modify the definitions of the Dreyfus model of skill acquisition to reflect entrustability rather than entrustment.
Another challenge with using EPAs in undergraduate education settings that used traditional norm-based methods to grade students was the tendency to extend the criterion-based approach to EPAs. For example, if a pass mark of 50% was set for a student clinical exam, which 50% of the EPA table could be safely ignored? Clearly this was not a tenable approach to assessment within the EPA framework. There remains much work to be done in this area to align the two approaches.
Finally, EPAs were not the end-all be-all for curriculum mapping. There was still a need to document specified learning outcomes for structured learning activities (e.g.; lectures, tutorials, etc.) and link them to EPAs. This represented a significant amount of work that needed to be done by all who anchored teaching and learning events in the curriculum.

Conclusion
Extending the current use of EPAs "backwards" from the graduate medical education (GME) realm to include the undergraduate medical education sphere is essential in ensuring that the "product" of medical school is aligned with the subsequent expectations of baseline knowledge skills and attitudes required by the graduate medical education of the entry level trainee.
The method of documenting EPAs in the undergraduate curriculum described here can easily be extended "forward" to document graduate medical education by adding the

South East Asian Journal of Medical Education
Vol. 9 no.1, 2015 13 additional problems and conditions one would expect to be covered in GME. Additional rows could also be added to the reference tables to describe activities expected of an advanced trainee, and columns could be extended to indicate the point at which an advanced trainee would be expected to attain entrustability.
In addition to extending the proposed EPA documentation approach into GME, we can also consider extending further along the training continuum to continuing medical education. When a person achieves Dreyfus level 4 or 5, this doesn"t mean they will always be at that level of expertise. In recognition of this, the American Board of Medical Specialties have instituted Maintenance of Certification requirements for most specialties, a mechanism to ensure currency on EPA subskills.
Apart from extending it "forward", for domain independent skills, we can even consider extending EPAs "backward" to the admissions exercise (at undergraduate or graduate medical education entry point), possibly getting these domain independent skills incorporated into admissions skills assessment exercises such as the multiple mini interview or situational judgment test (Dev, 2012;Eva et al., 2004). This would truly capture the continuum of training and perhaps enhance our ability to select more appropriate candidates.
EPAs are not a substitute for competency assessment. Mastery in one activity in the problem or condition table does not indicate proficiency in the EPA; similarly competence in one EPA does not indicate proficiency or competence to practice. While EPAs can be used to guide assessment, it must be done so with the spirit of blueprinting and purposeful sampling to have a more holistic picture of competence This is in keeping with current literature on transferability of skills, including clinical reasoning skills, which demonstrates that assessment in multiple areas is required (Bowen, 2006;Norman, 2005)