Feedback in Medical Education: An Evidence-based Guide to Best Practices from the Council of Residency Directors in Emergency Medicine

Within medical education, feedback is an invaluable tool to facilitate learning and growth throughout a physician’s training and beyond. Despite the importance of feedback, variations in practice indicate the need for evidence-based guidelines to inform best practices. Additionally, time constraints, variable acuity, and workflow in the emergency department (ED) pose unique challenges to providing effective feedback. This paper outlines expert guidelines for feedback in the ED setting from members of the Council of Residency Directors in Emergency Medicine Best Practices Subcommittee, based on the best evidence available through a critical review of the literature. We provide guidance on the use of feedback in medical education, with a focus on instructor strategies for giving feedback and learner strategies for receiving feedback, and we offer suggestions for fostering a culture of feedback.


Level of evidence
Case series/Qualitative studies or lowquality cohort or case-control study*** 5 Expert/consensus opinion *Defined as <80% follow up; **includes survey studies and crosssectional studies; ***defined as studies without clearly defined study groups. RCT, randomized controlled trial.  Level 5 evidence or troublingly inconsistent or inconclusive studies of any level *Extrapolation refers to data used in a situation that has potentially clinically important differences than the original study situation.

Natesan et al.
Feedback in Medical Education: Evidence-based Guide to Best Practices from CORD While some learners may value constructive feedback over reinforcing or encouraging feedback, both have been shown to be valuable. 10,18,29,30,75-78 It may not be necessary or helpful to include both constructive and reinforcing feedback during the same conversation. 79 Giving constructive feedback may be particularly challenging due to fear of retaliation (especially in systems where learner feedback is tied to summative evaluations, linked to author, pay or promotion). 80,81 However, limited literature suggests that the fear of retaliation may be unwarranted. 82,83 Other potential concerns surrounding constructive feedback include damaging rapport with learners or triggering an emotional response from trainees; however, these can be reduced by ensuring the feedback is clear, focused on actions or behaviors (as opposed to the individual), and supported by specific examples. 2,84 As a step toward improved performance, incorporating co-creation of goals 58, [85][86][87] and the formation of learning or action plans into feedback can increase the benefit to trainees. 1,6,12,17,[21][22][23]60,78,87,88 Additionally, encouraging collaborative discussion and learner reflection during the feedback conversation may be beneficial. 7,12,48,60,74,89 Faculty should be attentive and dedicated to providing feedback, as faculty effort and engagement have been shown to improve feedback. 7,90 The setting in which feedback is given is also important. 5,26 Feedback should be given in a non-threatening and supportive environment. 15,17,25,29,48,74,91 It may be prudent to use different types of settings for different feedback activities. 4 For example, constructive feedback may best be given in a one-on-one setting after a clinical encounter, whereas positive feedback on physical exams, procedural skills, or clinical decision-making may be more effective if given during or immediately after the patient encounter. 4,92 The optimal method used to provide feedback is unknown, and strengths and weaknesses of various forms of feedback have been highlighted. 19,90 Verbal feedback may be more helpful for engaging in collaborative discussion, but written feedback is more easily recognized and can serve as a reference for future reflection. 19, 90 It is important to note that inconsistencies exist between verbal and written feedback. An example is when learners receive positive laudatory feedback verbally, only to find disparaging or negative critiques in the written review. This inconsistency can lead to distrust and frustration among the learner and should be avoided. 47

Sources of Feedback and Personnel Involved
The source of feedback and the individuals involved can also impact feedback quality. 88,93 It is important that feedback be from a credible source. 94 Learners consider feedback more valuable and credible when given by those they consider experts in that specific domain. 8,21,94,95 However, the reliability of assessment may vary with assessor groups for different skills assessed; so, it can be valuable to deliberately align assessment and feedback areas with rater domains of expertise when possible. 71 The relationship between the individuals involved in the feedback discussion is also important. Having a good relationship based on mutual respect and trust can enhance the quality and accuracy of feedback. 3,12,57,62,77,87,90,[96][97][98][99][100] Training individuals on how to give feedback can also improve the quality and specificity of feedback delivered. 32,48,56,91,[101][102][103][104][105][106][107] Training can lead to improved comfort with providing feedback and increase the likelihood of the learner incorporating reflection and goal-setting into feedback discussions. 108,109 This is important, as lack of training in those providing feedback has been highlighted as a barrier to giving meaningful feedback. 20 Feedback may come from multiple sources, and prior literature has demonstrated that both learners and supervisors value multisource feedback (MSF) as described in Table  1. 49,52,63,[110][111][112][113][114][115][116][117][118][119] Limited literature supports that MSF may be more helpful for identifying strengths and weaknesses compared to standard assessment methods and may be more likely to result in behavior change. 52,63 Multisource feedback may also be effective in distinguishing between high, intermediate, and low performance in learners. 113 Additionally, data on the correlations of assessments between assessor groups is mixed, and different assessor groups may provide distinct feedback. 63,71,110,[120][121][122][123][124][125][126][127][128] Variations in assessments between assessor groups could suggest that assessments may be different but not necessarily less valuable, lending support to the importance of having multiple perspectives in feedback systems to provide learners with more comprehensive data about their skills. 63,120,122 Moreover, learners may value feedback from various groups differently 51 ; so, care should be taken to align assessor qualifications with the assessments they will be performing.
Barriers to MSF do exist and include lack of training in those providing feedback, time and resources required to gather MSF, and the ability of learners to incorporate this type of feedback. 20,112,124,129 Multisource feedback can be gathered synchronously or asynchronously, 121 but regardless of route, it should be timely and ideally incorporate multiple settings. 20,63 The incorporation of learner self-assessment into feedback can also have a positive impact. 17,22,23,128,130,131

Techniques and Tools for Providing Feedback
Currently, there is no consensus regarding the best methods for feedback and no formal endorsement by educational bodies of a single strategy. 18 When providing feedback, it is important to use a variety of techniques and tools tailored to the individual learner and situation. We summarize several feedback techniques including direct observation, real-time feedback, self-assessment, multiple sources, and other specialized techniques in Table 3.
Each of these techniques has strengths and weaknesses. Direct observation has been shown to be highly valued and can increase clinical knowledge, skills, and attitudes; however, there is limited data to suggest a behavioral change. 132 Strengths of direct observation include the emphasis on timely, learner-centered feedback. 107 Challenges to direct Feedback in Medical Education: Evidence-based Guide to Best Practices from CORD Natesan et al.  These barriers may be overcome by creating a structured, longitudinal direct observation and feedback program. 136 Realtime feedback is highly learner-centered, has been shown to improve the quantity of feedback given, and is generally well liked by users. 137 However, it doesn't necessarily improve feedback quality; studies have shown that less than 20% of the feedback given in real time is specific or corrective, often only focusing on positive and encouraging aspects of care. 27, 138 While learner self-assessment may not correlate well with external assessments, 122,127,128 it can contribute positively to feedback discussions by encouraging reflection and establishing a shared understanding and mental model for feedback. 17,22,23,74,85,128,130,131,139 Combining self-assessment with feedback can positively impact improvement behaviors. 130,131 Importantly, while evaluative models for feedback have been shown to improve the number of feedback evaluations, they may not improve the quality of corrective feedback. 140,141 Multisource feedback tools are generally well liked and have good efficacy for competencies such as inter-professional communication and professionalism; however, they may be limited in their ability to identify struggling learners. 142 Overall these techniques are quick and efficient and can work for a wide variety of learners to provide formative feedback. 143

Tools for Giving Feedback
Much like the variety of techniques for giving feedback, many tools have been developed to assist in providing feedback. Feedback tools have been demonstrated to increase the number of feedback encounters and improve learner satisfaction with feedback. 7,32,59,155-157 However, it is important to note that feedback tools are not a replacement for verbal feedback or preceptor experience. 7,19 We provide a summary of physical and electronic feedback tools including feedback cards, minicards, field note tool, MSF tools, web-based platforms, apps, crowdsourcing, and video recording in Table 4.
Like the techniques described above, each tool has its own strengths and weaknesses. Feedback encounter cards have repeatedly been shown to increase the perceived number of feedback encounters and, typically, improve learner satisfaction of quality, amount, and timeliness of feedback. 2,32,155,156,158 However, some studies have reported that feedback may not be specific enough. 156,159 This challenge can be mitigated by pairing encounter cards with a curriculum for educators and learners regarding giving and receiving feedback. 2,32 Mini-cards and the Mini-Clinical Examination Exercise can identify the struggling learner and provide formative assessments to support their growth. 88,148,[160][161][162] Both tools can be integrated into routine clinical work while providing reliable assessments if at least 6-8 such encounters are used. 162 A limitation noted for these card-based observation tools is that they may be perceived as a one-way evaluation and less likely to result in a learner-driven action plan. 148,161 As MSF has become more incorporated into feedback approaches, several tools have been developed and studied as listed in Table 3. 112,121,122,124,129,142 With the increased availability of smartphones and portable devices, an array of new electronic-based feedback tools have been created and implemented with the hope of making the administration of feedback more convenient, accessible, and timely for educators and learners. 18 Studies have shown that Feedback in Medical Education: Evidence-based Guide to Best Practices from CORD Natesan et al. Designed to take a minute to complete in order to facilitate same-day, timely responses in brief narrative comments, these systems were felt to be easy to institute and feasible approach to assessing students, particularly regarding professionalism behavior. These online survey platforms can increase the amount and timeliness of feedback. However, there is a need to emphasize data consolidation and distribution with these tools to ensure that feedback is distributed in a timely manner.
Facebook Dashboard, QuickNotes, TIPreport, One Minute Mentor/ Minute Feedback System, and online surveys such as Google Forms and SurveyMonkey App-based 18,70, [168][169][170] This is a feedback tool accessed through a mobile application to allow ease of use. These apps were shown to help collect useful data and provide an increased amount of quality feedback. They also were found to have benefits of accessibility, low cost, and ability to trend resident progression.

Natesan et al.
Feedback in Medical Education: Evidence-based Guide to Best Practices from CORD these web-based tools can be beneficial for improving faculty engagement in and frequency of their feedback. 11, 163,164 The timely nature of this feedback also leads to increased satisfaction from learners. 164 However, these platforms can be limited by faculty comfort with, and knowledge of, technology. 165 Additionally, specific and corrective feedback may be challenging. 27,138 It is important to consider data consolidation and distribution with these tools to ensure that feedback is distributed in a timely manner. 137,166 To improve the accessibility of online feedback tools, several platforms have used quick response (QR) codes. 163,167 The use of QR codes to access online feedback forms was found to be user-friendly and resulted in faster completion than paper and online web-based tools not associated with a QR code. 167 Various apps have been created, which have led to an increase in the quality of feedback. 18,70,168-170 Additional strengths include accessibility, low cost, and ability to trend resident progression. 18 However, much like web-based platforms, app-based platforms can be limited by faculty and resident engagement. 18,170 When instituting any app-based evaluation tool, it is important to pair it with training on the app and changes to feedback culture, such as regular encouragement, incentivization, physician champions, or regular reminders. 18, 168,170 Using online social media platforms (eg, Twitter messaging) is another tool to increase the volume and timeliness of feedback; however, effectiveness may be limited. 18,171 Videoassisted feedback can be a valuable tool for feedback similar to direct observation. 133,172 However, much like other forms of direct observation, video recording may not represent true, real-world encounters as learners may act differently due to the Hawthorne effect. Additionally, video recording can cause anxiety in trainees. 133 Inviting Co-workers to Evaluate Physicians Tool

Receiving Feedback
Traditional approaches place learners in the role of passively receiving feedback, 79,82,176,177 which have been criticized for being too centered on the actions of the instructor. More modern models shift to include the learner as an active participant in soliciting and responding to feedback. 4,13,153

Soliciting Feedback
A crucial initial step to engaging in effective feedback is the act of soliciting feedback that opens the individual to the critiquing process. 55, 178 The ability to engage in feedback-seeking behaviors is dependent on four factors: the purpose and quality of the feedback; the learner's emotional response to feedback; the learner-evaluator relationship; and the workplace culture. 4,13,50,176,179,180 While the environment is outside our control, appropriately prepping learners to take contextual factors into account and shifting the focus to environmentally appropriate feedback models may be particularly helpful. 181,182 One common example is the implementation of end-of-shift feedback evaluations. While these have not been identified by faculty as providing a higher quality of feedback, their systematic and reliable delivery results in higher resident satisfaction with the feedback. 32

Accepting Feedback
Despite the best intentions of the feedback giver, feedback receptivity is never assured. Literature demonstrates that faculty and learners even disagree on their perceptions of how much feedback is being given. 1,16,55 Nevertheless, learner perception significantly impacts feedback acceptance and integration. 130,180,183 Different experts have categorized such factors in different ways. 1,50,57,184 One of the more usable classifications includes categorization of personal (ie, resilience, humility), relational (ie. the strength of supervisory relationship, power differentials), and contextual (ie, culture) factors. 57

Personal Factors
Much of feedback receptivity depends on the learner's frame of reference. Possessing a growth mindset and employing routine self-reflection is key. 62, 89,96,100,182,183,[185][186][187] Learners often approach feedback situations as a performance, probing the situation to see what is expected of them and then acting in a way to better shape their reputation and evaluations. 1,13,16,184,188 Those who have blind spots regarding their weaknesses may be resistant to feedback that challenges their existing self-perception. 130,[185][186][187][188][189][190][191] Failure to internalize feedback happens when a mismatch in external and internally generated assessment occurs. For instance, EM residents consistently assign themselves higher milestone competency ratings than their evaluating attendings. 189 When feedback is perceived as an attack on personal identity, feedback internalization is effectively Feedback in Medical Education: Evidence-based Guide to Best Practices from CORD Natesan et al.
hindered. Thus, learners should perceive feedback as opportunities for improvement, rather than statements on character. 1,134,192 Evidence suggests that learners educated on feedback have shown comfort in giving and receiving feedback. 105 Melding self-generated learning goals with faculty-provided observations closes the feedback loop and produces more improved, usable, and well-received feedback aimed at mastering current skills and setting goals for future accomplishments. 12,22,130 To bridge the gap between reception of the feedback to internalizing it, multiple experts have outlined various practical tips for learners to use feedback for performance improvement. 190,191,[193][194][195][196][197][198][199][200] We distilled the consistent themes among our recommendations below.

Relational Factors
Feedback receptivity is significantly impacted by relational factors such as the strength of the supervisory relationship and power differentials. Regardless of the experience level of the assessor, learners consistently recognize feedback as valid when coming from someone they trust and respect, find credible, 1,182,192 and have sought out rather than been assigned, 181,197 such as from role models. 198,199 Mutual respect, establishing shared priorities, and the strength of the educational alliance (defined as the learner's belief of shared goals, activities, and bonds) 200 facilitated better feedback receptivity. 57 Interpersonal skills also affect the relationship and receptivity. Power dynamics and fear of the effect of corrective feedback are barriers to feedback integration. 57 Learners value feedback when given in a caring, nonjudgmental manner 31,62,99 from educators who are friendly and approachable. 201

Contextual Factors
Environmental and cultural considerations affect the receptivity of feedback. The tension between assessment and feedback, specifically the fear of consequences, can lead to learner development of a fixed mindset, limiting growth opportunities. 57, 96 For professionalism issues, feedback should be given one on one. 1,20 In busy learning environments, learner-centered approaches grounded in self-directed learning theories (eg, Learner-Centered Approach to Raise Efficiency [L-CARE)] in Clinical Teaching) have been proposed to facilitate more efficient learning. 202 Ultimately, various studies demonstrated benefit and/or learner preference for standardized, 139,203 structured, 150,203 multisource, 150 and longitudinal 1,105,204,205 feedback processes.

FEEDBACK CULTURE
Feedback culture is defined as written or verbal comments regarding medical knowledge, performance, technique, or patient care within the pedagogical approaches that are routine within a profession. 206,207 The learning culture and

Implementation
Institutions should provide and encourage educational opportunities to all individuals involved in feedback interactions including learners and educators. This will allow a culture of growth emphasizing a bidirectional feedback approach 1,62,100 with a shift from performanceoriented assessments to learner-oriented feedback. 56 One method is to emphasize the concept of lifelong learning and normalize the need to identify strengths and weaknesses as a way to grow. Training on giving feedback upward and receiving feedback as an educator can help provide the framework for effective bidirectional feedback. 1,16,99,181,194,200 Learners need an environment where vulnerability is acceptable and assessment focuses on a set of shared goals. 14,47 Other strategies include establishing expectations for both educators and learners, promoting specific tasks for all involved, and providing professional development sessions. 57,210 For establishing longitudinal relationships, providing protected faculty time for observational assessments and using standardized feedback tools are beneficial. 97,136,207 Furthermore, institutions should encourage a culture of growth. Learners develop a fixed mindset when they perceive performance is linked to assessment, rather than a growth mindset when the relationship is not tied to assessments. 96 An interdisciplinary, multimodal approach to feedback through MSF can provide additional insight regarding

Barriers to Successful Implementation
Successful implementation of an optimal feedback culture requires a firm understanding of the potential barriers. Grade inflation, discomfort in providing negative feedback, concern with preserving healthy working relationships, 2,32,84 time constraints, and personal deficiencies in feedback delivery each present unique challenges. 67, 84 Administrative support and the encouragement of the importance of feedback are also important. 67 While feedback tools may pose a barrier, choosing a user-friendly tool that is of appropriate length and provides sufficient detail with required narrative comments is key. 8,9,24 Although limited literature suggests this may be unwarranted, 82 educators often avoid corrective feedback due to fear of retaliation (especially in systems where learner evaluations are linked to pay or promotion). 80,81 Transparency and focus on the importance of corrective feedback as a learning tool 12 can prevent reluctance to provide negative feedback. 12,80 Finally, a culture of "niceness" can make the learning environment overtly positive, which can hinder the delivery of honest feedback and the creation of a culture of constructive feedback. 12,99,100 Being "nice" can be construed as focusing on the positive with a priority on minimizing any negative feelings in the other person, while being "kind" can be construed as focusing on what is best for the learner overall -even if it means creating negative feelings.

Special Considerations
Implicit bias, which is the unconscious attitudes we have toward people or associated stereotypes, impacts both feedback provided to learners and the perception or receptibility of feedback from faculty. 47 To minimize this potential bias, assessments should be performed by multiple assessors in multiple different settings. 47,63,210 Furthermore, institutions should implement training to identify areas where biases exist, while working to alleviate these biases with full transparency. 47 Gender bias may lead to different distributions of the frequency and type of feedback. One study found female preceptors completed more feedback forms and provided more corrective feedback to male learners, whereas male preceptors used more communal language and less agentic language with female learners. 212 Additionally, female learners had more discordant feedback, especially regarding the balance of autonomy and feedback receptivity, than their male counterparts. 213 Finally, in a study by Stroud, female faculty were found to be perceived as less credible when delivering feedback. 95 Like racial, cultural, and gender bias, generational gaps can also affect meaningful feedback. Different generations have different patterns of learning. For example, the millennial generation is more engaged in technology and collaborative learning, while preferring clear objectives and timely feedback. 28,61 Additionally, feedback should be provided to all learners, not just low performers. High performers may exhibit the "halo effect," which can result in them receiving less constructive feedback. 1 Learner shame responses can be triggered by repeated humiliation experienced in receiving feedback from facilitators. Providing feedback that is focused on behaviors, providing support that normalizes errors in the learning process, and guiding learners through reflection can decrease these learner responses.

LIMITATIONS
Although we performed a comprehensive search guided by a medical librarian in conjunction with a bibliographic review and expert consultation to augment content when needed, we used a single search engine (MEDLINE), and it is possible that we may have missed some pertinent articles. In instances where evidence in the form of high-quality data was limited or lacking, we relied upon expert opinion and group consensus for the best practice recommendations. The literature specific to feedback for the field of EM and within graduate medical education is limited. To supplement, we included relevant articles from other medical specialties and health-related professions as we believe that EM, as a specialty, can learn from other colleagues across many disciplines. Finally, in areas where evidence was not available, we used the consensus from the expertise of our authorship group. While our author group possesses experience in research and scholarship in both feedback and medical education, there is a potential for bias to be introduced during this process. Therefore, we also sought peer review from the