Where do we go from here? Moving from systems-based practice process measures to true competency via developmental milestones

For many educators it has been challenging to meet the Accreditation Council for Graduate Medical Education's requirements for teaching systems-based practice (SBP). An additional layer of complexity for educators is evaluating competency in SBP, despite milestones and entrustable professional activities (EPAs). In order to address this challenge, the authors present the results of a literature review for how SBP is currently being taught and a series of recommendations on how to achieve competency in SBP for graduate medical trainees with the use of milestones. The literature review included 29 articles and demonstrated that only 28% of the articles taught more than one of the six core principles of SBP in a meaningful way. Only 7% of the articles received the highest grade of A. The authors summarize four guiding principles for creating a competency-based curriculum that is in alignment with the Next Accreditation System (NAS): 1) the curriculum needs to include all of the core principles in that competency, 2) the objectives of the curriculum should be driven by clinical outcomes, 3) the teaching modalities need to be interactive and clinically relevant, and 4) the evaluation process should be able to measure competency and be directly reflective of pertinent milestones and/or EPAs. This literature review and the provided guiding principles can guide other residency educators in their development of competency-based curricula that meets the standards of the NAS.

recently, to assist educators in reaching the Outcomes Project's goal, the concepts of developmental milestones (3) and entrustable professional activities (EPAs) (4) have been established.
The educational outcomes are grounded in the six core competencies: 1) patient care; 2) medical knowledge; 3) practice-based learning and improvement (PBLI); 4) SBP; 5) professionalism; and 6) interpersonal skills and communication. Of these competencies, PBLI and SBP are the newest additions. SBP has six core principles and is defined as an 'awareness of and responsiveness to larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value' (1). Specifically, all residents need to demonstrate the ability to: 1) work effectively in various health care delivery settings and systems relevant to their clinical specialty; 2) coordinate care within the health care system relevant to their clinical specialty; 3  cost awareness and risk benefit analysis in patient and/or population-based care, as appropriate; 4) advocate for quality patient care and optimal patient care systems; 5) work in interprofessional teams to enhance patient safety and improve patient care quality; and 6) participate in identifying system errors and implementing potential solutions.
The goals of the ACGME, as evidenced by Phase II of the Outcomes Project, are for residency programs to use dependable measures to assess residents' competencies and, subsequently, to use those individual assessments to evaluate the educational effectiveness of the residency as a whole. As is clearly defined, these abilities are expected to be specific to each specialty (2) Á and each competency should be sufficiently unique to merit its own category. Yet, data have shown that residencies are having difficulties measuring the individual competencies as independent constructs, and assessment tools often measure 'behaviors that consistently map onto three or more of the general competencies' (5). As a result, educators continue to struggle with the creation, definition, and categorization of six separate competencies. Furthermore, uncertainty continues to persist as to which assessment tools are most valid. Because true competency extends far beyond simple knowledge, educators are looking for measures that can assess 'authentic' behaviors of real-life clinical context. To date, only a 'few models offer clear and direct routes' for competency assessments without having measurement pitfalls (1).
Despite often being educators' lowest priorities, this challenge especially exists for SBP and PBLI. Indeed, a survey of family medicine program directors rated SBP as the lowest importance of all the competencies, with 70% reporting no method to evaluate SBP (6). A major challenge is that SBP is truly a skill-based activity that cannot be validly captured with simple knowledge-based assessments. In response to these and other similar educational challenges, the ACGME developed 142 discrete curricular milestones that map to the six specific core competencies (3).
Even with the development of various educational constructs (e.g., competencies, milestones, EPAs), residency programs continue to struggle with the use of milestones and/or EPAs in their curricular goals, objectives, and observational evaluative tools (4) Á and little related information has been reported specific to SBP. Hence, we explored how SBP is being integrated into residency education and whether or SBP measures were in alignment with the Next Accreditation System (NAS). We began by reviewing our own internal medicine residency program, and found that our SBP curriculum and its assessments measured associated knowledge and attitudes Á but probably not true competency (7). It was obvious, then, that we needed to evaluate in far more greater detail the six components of SBP as they apply to internal medicine residency programs. To move from process measures to hard outcomes like SBP-related developmental milestones or EPAs, we conducted an extensive review of the literature on SBP in internal medicine residency programs. The results of that literature review are presented here, along with recommendations on how to achieve competency and best use NAS evaluative measures in SBP for graduate medical trainees.

Search strategy
We reviewed literature published from 1999 to April, 2012 present using PubMed, Google Scholar, and ERIC ( Fig. 1) Á using the following key search terms: 1) core competencies; 2) system-based practice; and 3) graduate medical education. For this search 1,158 titles were generated. Of the original 1,158 titles reviewed, 179 titles remained after duplicates, books and titles that named a specialty other than internal medicine were eliminated. The 179 respective abstracts were reviewed and 37 full articles were chosen after applying our inclusion criteria. To these 37 articles, 6 more articles, gleaned from a bibliographic review, were added. We also supplemented our pool with seven articles from the ACGME's Outcomes Project web-based reference guide. After carefully reviewing these 50 articles, we excluded 21 which, despite their titles, did not align with our predetermined inclusion criteria.

Inclusion and exclusion criteria
All abstracts and articles were included if they contained either a description of the curriculum and/or the associated evaluation methods. The described curriculum had to be focused on the core competency of SBP within US internal medicine residencies. Articles were excluded if they: 1) were in foreign medical programs; 2) were primarily intended for medical students/fellows; 3) focused on more than two core competencies; or 4) involved learners in other (non-internal medicine) specialties. Articles discussing SBP and one other core competency (patient care, medical knowledge, PBLI, professionalism, or interpersonal skills and communication) were included.  However, opinion pieces, review articles, and consensus conferences were excluded Á as were papers from nonpeer-reviewed journals.

Articles
As stated, the initial search resulted in 1,158 references ( Fig. 1). After the above search strategies, removal of duplicates, and a thorough screening using inclusion and exclusion criteria, 29 articles were included in the systematic review (8Á36). Given the exploratory nature of this study, we had little basis for organizing these studies (5). A subsequent review of the literature helped define SBP and its principles and implied abilities Á from which six categories were developed based on Graham's taxonomy of SBP (1) ( Table 1). The primary aim of the reviewers, then, was to categorize each article into one of the six groups; however, several articles contained no single primary theme, but instead meaningfully addressed more than one of the six SBP principles and were subsequently grouped under more than one category. When this occurred, all three authors agreed to do so.
A quality index, based on definitions from the ACGME's requirements for achieving competency, was established to 'grade' articles. This index ranged from A to D (with A being the highest) and was applied to descriptions of the curriculum and evaluation methods. Because our literature review went back to 1999, the grading system did not include the criteria for milestones or EPAs Á which emerged much later (3,4). The definition of each grade is listed below Á and centers primarily on the measured focus of the curriculum and evaluation: 1. Patient outcome(s); 2. Behavioral change by the provider (mostly closely linked to milestones or EPAs); 3. Perceptions, knowledge, or attitudes of providers and/or patients; and 4. Description of content only Á no evaluation was completed.
Initial classification of category and quality was done independently by two of the authors (JM and EP). A third reviewer (CH) was used to reconcile disagreements. All reviewers are faculty members involved in residency education. One (JM) is a core faculty member who codirects two residency educational units, whereas the other (EP) leads the primary care research block. The third reviewer (CH) is the primary care program director.

Results
After systematically culling the larger sample, the systematic review included 29 articles (Fig. 1). From these, the two reviewers exhibited 66 (19/29) and 90% (26/29) agreement on their assignment of category and grade, respectively.
Only one internal medicine graduate curriculum, represented in 2 of the 29 articles (7%), incorporated all the six principles of SBP into their described curriculum; eight (28%) articles included curricula covering more than one aspect of SBP in a meaningful way. The majority of articles (45%, or 13/29) were classified as 'system evaluator' Á concentrating on the principle of 'identifying system errors and implementing potential solutions', or what is most readily labeled as quality improvement (Table 2). Few articles formally addressed coordination of care or patient advocacy.
Only 7% (2/29) were graded A Á the highest quality. These articles represented the ACGME's 'gold standard' of assessing mastery of a competency via patient outcomes. These two articles reflect examples of what most educators set out to achieve when developing curricula Á and go beyond the typical assessment of learners' knowledge of content. Interestingly, none of the articles, even those published after 2009, stated specifically how milestones and/or EPAs were addressed in their curricular design and/or evaluation process, if they occurred. However, both of the two 'A' articles were among those more recently written Á suggesting that educators may be refining curricula to mirror more closely the goals of the Outcomes Project and NAS (Table 3).

Study and search limitations
Our search included only published, peer-reviewed articles Á each of which contained its own set of limitations and possible biases. Also, the articles did not use uniform definitions or terms, and exclusively focused on internal medicine residency programs. Lastly, because existing rating criteria had not been previously applied to medical education literature, our development and use of the present coding scheme is somewhat novel and, as such, of unknown validity.

Discussion
This comprehensive literature review suggests that efforts are underway to incorporate SBP into internal medicine residency education Á with isolated examples geared toward evaluating competency in this arena. Yet, the data also indicate that residency programs are lagging behind the Outcomes Project's goal of full integration by 2011 Á the largest barrier seemingly to be the design of curricula and operational measures that can independently assess each of the six competencies, even with the inclusion of milestones. This review shows that despite much curricular development, there appears to be little  The % of articles that covered that category.  agreement and data regarding best practices in assessing learner competence Á especially with regards to SBP and its six core principles. Furthermore, areas that require behavior or performance-based assessment (e.g., milestones, EPAs) are, in contrast to knowledge and written assessments (37), still among the least developed areas.
Before moving ahead to Phase IV (2011 and beyond) of the Outcomes Project, educators must verify that the first three phases have been successfully completed. This review of the literature has shown that in an attempt by the ACGME to make definitions widely applicable to all programs and institutions, there exists 'lack of enough specificity to move our understanding towards what exactly it is residents need to demonstrate in order to indicate proficiency in SBP' (38). Each article described different methods and approaches, and the data show that most curricula concentrate efforts on only one of the six principle areas of SBP. As a result, there is much progress to make before SBP can be comprehensively taught and assessed as a core competency.

Lessons learned and guiding principles
Despite the abovementioned challenges, educators should focus on the practical value of proposed educational changes. In the next steps, as an educational community, we must agree upon a basic model of teaching and assessing each competency that also provides room for divergence and innovation. Keeping in mind the 'gold standard' of improved patient care, we must move towards hard outcome measures of competency Á for example, documenting milestones and completion of EPAs (39,40).
We offer four guiding principles that might inform educators when creating or restructuring SBP curricula. The principles were applied to our own SBP program efforts (5), but can just as easily be used in any curriculum that will address any of the six core competencies of patient care, medical knowledge, PBLI, SBP, professionalism, and interpersonal skills and communication.
1. The curriculum must include all core principles of the specified competency.
In a hypothetical example of reformatting a SBP curriculum, the knowledge content should first be The examples would then be listed for each category of SBP and its corresponding milestone (Table 4). 3. The teaching modalities should be interactive and clinically relevant.
The concrete examples outlined above should be incorporated into a curriculum via a teaching strategy that is interactive and clinically based. Some didactic sessions may be warranted, but other pedagogies should be considered. Of course, the degree of innovation will depend on institutional resources, but educational technology such as podcasts (if available) can often overcome scheduling or time barriers. Other examples include case-series, peerassisted learning, simulations, and direct observations. The teaching and learning modality should also inform the evaluation process. 4. The assessment of competency should utilize milestones and/or EPAs.
Attention to the course evaluation should be turned from content-oriented to assessment-based, and evaluative tools should focus on patient outcomes or physician behaviors Á such as chart reviews, patient satisfaction, or Plan, Do, Study, Act (PDSA) cycles. For example, in addressing the 'resource manager' component, one could conduct chart reviews to see how often residents are asking about patients' abilities to manage the costs of medications and, when a problem is identified, how the resident addressed the issue. For the 'team evaluator' aspect, a 360 degree evaluation of a resident involving ancillary staff, social workers, and nurses could be conducted to measure specific physician behaviors toward members of the interprofessional team.
Other assessment tools that could be considered are direct observation of SBP skills Á such as observing, scoring, and documenting residentÁpatient encounters around the use of interpreters. There are many other examples; emphasis should remain on curricular objectives driving by clinical outcomes and tied to a milestone (Table 4) or EPA.

Conclusion
Many of the lessons learned in reformatting our own SBP curriculum centered on best delivery of teaching methods and assessment tools Á both of which should align with current changes in health care and health professions education.
Despite not yet being able to share successful experiences from our reformatted curriculum (7), sharing lessons learned may assist other programs in enhancing or creating their own SBP curriculum as mandated by the ACGME. We believe our experience, this literature review, and the guiding principles provided can assist in the development of competency-based curricula.