The impact of multisource feedback (MSF) including patient feedback on psychiatric trainees

Introduction: MSF is mandatory for trainee doctors therefore it would be tempting to assume its educational benefits. UK medical trainees do not have to obtain patient feedback as part of MSF. There is evidence examining various aspects of patient feedback and MSF, but little specifically on the performance effects of MSF with patient feedback. By systematically reviewing the literature I aim to explore these effects. Method: Multiple databases were systematically searched for eligible literature which was critically reviewed, evaluated and summarised Results: Quality of studies was variable; most had small, uncontrolled samples, assessing outcomes at lower Kirkpatrick levels (Barr et al., 2000). Although one RCT and a prospective cohort provided stronger evidence. They were generally positive about the educational benefits of MSF with patient feedback, providing evidence to be interpreted with caution. They also revealed insight into particular factors important to include in MSF to maximise benefit. Conclusions: There is evidence that MSF with patient feedback can lead to performance improvement although more robust evidence is needed. Consideration could be given to improve the MSF tools trainees in the UK currently use.


Introduction
Postgraduate medical training has undergone major reform in recent years. Modernising medical careers has led to competency based assessment of doctors replacing the traditional apprenticeship model ( Van der Vleuten, 1996). Workplace-based assessments (WPBA) were introduced in 2005 and are now mandatory for all trainees (Brittlebank Murton C MedEdPublish https://doi.org/10.15694/mep.2016.000023 Page | 2 et al., 2013). They aim to assess what doctors do in practice, at the top of Miller's pyramid (Miller, 1990).They arguably provide more objective and valid judgement of performance (Norcini & Burch, 2007) than the observations of an educational supervisor and have formative potential rather than simply assessments. Feedback is integral to WPBA and there is convincing evidence that well delivered feedback can help doctors change clinical practice over time (Norcini & Burch, 2007;Saedon et al., 2012;Veloski et al., 2006).
Expert clinical judgement is a combination of scientific understanding with humanistic qualities; particularly important for psychiatrists where inter-professional team working and working relationships with patients are essential (Lelliott et al., 2008).The Chief Medical Officer for England stated "whilst patients want their doctors to have good clinical knowledge and technical skills, they rate interpersonal aspects of care as equally, if not more, important" (Baker, 2005). There have been questions about the validity of measuring humanistic qualities (Baker, 2005;Evans, Elwyn & Edwards, 2004), but multisource feedback (MSF) appears the most suitable WPBA to assess these and clinical skills.
Because of the emphasis placed on WPBA in medical training it would be tempting to assume they lead to performance improvement, especially considering evidence that feedback can help change practice (Saedon et al., 2012;Veloski et al., 2006). There is a large body of evidence examining various aspects of WPBA, especially psychometric properties. However less examining the educational impact of WPBA including MSF. There is a large body of evidence looking at various properties of patient feedback, but less looking into the effects of patient feedback on clinical performance. There is some assessing its educational effects, including a systematic review (Reinders M et al., 2011) and other studies (Greco, Brownlea & McGovern, 2001;Patel et al., 2011), finding beneficial effects of patient feedback, although usually at lower Kirkpatrick levels (Barr et al., 2000). The strongest evidence for WPBA fuelled improvement in medical performance comes from systematic reviews by Miller (Miller & Archer, 2010), and later Ferguson (Ferguson, Wakeling & Bowie, 2014). Miller studies several WPBA and states that MSF seems the most likely to fuel practice improvement, however acknowledges problems with methodological rigour, conflicting results and confounding factors of studies. Ferguson examined MSF and concluded that there was limited evidence suggesting MSF leads to practice improvement. These reviews do not look specifically at MSF including patient feedback or psychiatric trainees.
By systematically reviewing the literature I seek to answer the questions: Does MSF with patient feedback improve performance of psychiatric trainees? What are the effects and influencing factors of MSF with patient feedback on psychiatric trainees? Following initial literature review this question was expanded to 'all qualified doctors' due to lack of more specific evidence.

Search strategy:
The databases MEDLINE, PSYCHINFO, EMBASE, CINAHL, ERIC, British Education index, Australian education index, Cochrane database were searched individually using the same search terms and strategy. Educational databases were accessed via Plymouth University website and NHS databases via NHS Athens.
Search was broadened to include the following terms from bookmarks of relevant papers: 6. "medical education" OR medicine OR "medical training" OR "postgraduate training" OR "clinical competence" Searches were combined and duplicates removed. Terms were kept broad to maximise chances of finding relevant articles. Subject headings from thesaurus and * to include multiple word endings were used. Potentially relevant references; titles and abstracts were screened, with relevant articles read in full. Manual searching of reference lists of relevant papers was conducted.
Eligibility criteriainitially studies were searched for which attempted to evaluate the educational impact, impact on performance or explore the effects of MSF including patient feedback on psychiatric trainees, however search was broadened to include studies attempting to: a) evaluate the educational impact or impact on performance of MSF including patient feedback on postgraduate doctors b) explore the effects of MSF including patient feedback on performance of postgraduate doctors Empirical studies of any design were included; i.e. RCT, controlled trials, qualitative, observational, cross sectional.

Exclusion criteria:
-Articles on undergraduates or non-medical professionals.
-Review articles, letters or commentaries -Studies using simulations rather than real patients.

Limits:
Search was conducted December 2015/January 2016. Only English language studies were reviewed. No time limits were placed.
Data extraction: data from included articles was compiled into a table to simplify comparison (Appendix 1).
Evidence was critically appraised and the outcome evaluated using the Barr adaption of Kirkpatrick evaluation model applied to the results (Barr et al., 2000), modified by Ferguson and Overeem (Ferguson, Wakeling & Bowie, 2014;Overeem et al., 2010)- Figure 1.  This relates to wider changes in the organisation/delivery of care, attributable to an education programme Covers any improvements in the health and wellbeing of patients/clients as a direct result of an education programme.

Quality Indicator Detail
Research question Is the research question(s) or hypothesis clearly stated?

Study subjects
Is the subject group appropriate for the study being carried out (number, characteristics, selection, and homogeneity)? Were results supported by data from more than one source?

Results
Medline was used for primary search and revealed 837 potentially relevant papers. Titles and abstracts were screened. Full text articles of 21 papers were reviewed and 6 articles were included.
Search was re-run on Psychinfo which revealed 2 papers which were included.
The Cochrane database revealed 1 potentially relevant article, full text was reviewed but it did not fit eligibility criteria. Manual review of reference list revealed 7 relevant articles which were included.
The other databases did not reveal any articles not already included.

General findings
Appendix 1 summarises general findings. There were 10 studies from Canada, 2 from USA and 3 from The Netherlands. There was 1 RCT, 1 prospective longitudinal cohort, 1 cross sectional survey study, 5 descriptive or observational survey studies using questionnaires, 5 qualitative and 2 mixed methods studies. 1 RCT measured behaviour change externally (3b) Alongside the MSF report, the intervention group completed a self-assessment form and a tailored coaching session. Physicians who engaged in MSF demonstrated a significant improvement in a number of professional behaviours including communicating patients and family, timeliness and responsibility and accountability, compared to a control group who did not take part in MSF. However it is not clear whether improvement was due to the MSF feedback or tailored coaching session. There are also wide confidence intervals indicating that the results should be interpreted with caution (Brinkman et al., 2007). 1 other study attempted to externally measure changes in performance by collecting MSF ratings on the same individuals 5 years apart in a prospective longitudinal cohort (Violato, Lockyer & Fidler, 2008). Performance ratings from colleagues and co-workers increased significantly, patient performance rating increased but not significantly. Overall it concluded that there was evidence for construct validity and there were small to moderate improvement in scores over time (3b). However it was difficult to conclude whether the changes were solely as a result of the MSF, with likely confounders within the 5 year time period.

Educational impact and other outcomes of MSF with patient feedback on performance of doctors including
Many of the studies used questionnaires or qualitative methods to determine practitioners self-reported changes or contemplation of changes (3a or 2a). Fidler  reported 83% had contemplated a change, most frequently for aspects of practice associated with clinical skills and resource use (2a). 66% of doctors had initiated changes following MSF most commonly in areas of communication and support of patients (3a). Hall (Hall et al., 1999) asked physicians via questionnaires and focus groups about their reactions to the MSF feedback. Approximately 2/3 of doctors were considering or had implemented changes in practice, most commonly in areas of communication with patients (2a or 3a). 308 physicians volunteered for the study which also looked at psychometric properties, however it is not clear how many of these responded to the follow up questionnaire about impact on their practice or were involved in focus groups.
Overeem (Overeem et al., 2009b) used qualitative interviews for people with outlying MSF results 1 year after MSF to explore educational impact. They found 11 out of the 23 consultants reported making changes (3a) the other 12 displayed an awareness of the need for improvements (2a). In a later study (Overeem et al., 2012) Overeem aimed to identify educational impact and the factors affecting this via questionnaire. There was a fairly low response rate (52%), but 55% of responders reported making changes to practice following the MSF intervention-(MSF, reflective portfolio and facilitative interview) (3a), however it is not clear if the MSF was solely responsible for these changes.
In an initial pilot study Sargeant (Sargeant et al., 2003) included 142 family physician with an 80% response rate. It used a descriptive survey design with a questionnaire and aimed to explore the responses of family physicians to MSF; the physician achievement review (PAR). 61% indicated they planned to make practice change (2a), or had actually done so (3a). Of the changes identified, 75% were either written or verbal communication, mostly communication with patients. Sargeant went on to further explore this with qualitative studies sampling from the original pilot study population (Sargeant, 2006;Sargeant, Mann & Ferrier, 2005;Sargeant et al., 2007;Sargeant et al., 2009). Sargeant (Sargeant et al., 2007) used open ended interviews to increase understanding of consequential validity (intended and unintended outcomes) of MSF. 15/28 participants received high or average scores which were perceived as positive feedback and subsequently did not make any changes. 13/28 receiving average to low scores interpreted feedback as negative, or inconsistent with self-opinion. 7 of these low scoring group reported making changes in response to feedback, often after a prolonged reflective period (3a). Several reported prolonged psychological stress (unintended outcome). Some did not accept the feedback and did not change, or accepted it but felt unable to change. Violato  examined psychometric properties and feasibility of MSF to assess surgeons, using a survey to identify changes to practice. On the follow up survey completed 3 months after the MSF intervention, 71.6% surgeons contemplated or had initiated changes (level 2a and 3a). Data are not displayed, however it appears the same population as Lockyer  were used, however different results and conclusions are obtained.
Other studies examined intention to change (or not), or modifications of attitude (2a). Lipner (Lipner et al., 2002) evaluated the American Board of Internal Medicine's continuous professional development (CPD) system patient and peer assessment component, including its educational value. Doctors get MSF feedback, complete selfassessments and a quality improvement plan (QUIP). 42% stated intention to change communication with patients, 25% stated intention to change communication with peers (2a). Problems included that scores were highly positively skewed, however it was felt that there was variability allowing for distinction between levels of performance. This was possibly due to the self-selected/volunteer cohort. The presence of factors other than MSF (QUIP) could confound results.
Lockyer ) used a descriptive survey design with a questionnaire, to determine and explain the likelihood that surgeons would make changes following receipt of MSF performance data. It concluded surgeons were unlikely to make practice changes in response to feedback data even if their MSF indicated a need to consider change (2a).
Overeem (Overeem et al., 2010) used a cross sectional survey comparing 3 different sources of MSF to identify their feasibility and impact on performance, 109 specialist doctors took part. The PAR contains colleague, non-medical co-worker and patient feedback. The AAI is purely qualitative with colleagues and co-workers asked to mention strengths and suggestions on how to improve. The ABIM contains feedback from colleagues and co-workers only. All groups were asked to complete a reflective portfolio, undergo a facilitative interview and complete a personal development plan. Outcomes on performance were evaluated via questionnaire. A majority of consultants indicated intention to change (2a) but this varied with the tools; PAR 61%, AAI 66%, ABIM 25%. It is not clear how much of the impact on performance was due to MSF, or the other components of this system. However, the different MSF tools were the only variables.
Studies which commented on educational usefulness or satisfaction with the intervention (level 1) included Lipner (Lipner et al., 2002) where 82% found it a positive learning experience indicating they would use it again. Overeem (Overeem et al., 2010) where 53% considered the PAR to be useful to future practice, compared to 89% for the AAI and 75% for ABIM. Sargeant (Sargeant et al., 2003) where 89% believed the report to be useful.

Factors influencing educational impact
Many of the included studies, also explored factors which influence the use of MSF feedback. Doctors' response to feedback and whether it was perceived as positive or negative was explored by several studies. Sargeant (Sargeant, 2006;Sargeant, Mann & Ferrier, 2005;Sargeant et al., 2008;Sargeant et al., 2009) found physicians who agreed with their feedback appeared more accepting of its accuracy and were more likely to consider practice improvements based on that feedback. However some with positive feedback reflected little and made few changes as they felt there was no need. Where there were inconsistencies or average feedback, longer reflection occurred and generally it was used for change. Negative feedback was met with varied responses' 1/3 of the group of 16 with negative feedback (Sargeant, 2006) accepted and used it, however 2/3 of this group where feedback did not confirm self-perception expressed strong negative emotions and extended reflection. Some eventually changed, seeming to process their emotions and move on to use their feedback. Others did not accept their feedback and did not change; their initial negative emotions seeming amplified by concerns about fairness of the process. Reactions were influenced by perceptions of accuracy, credibility and usefulness of feedback. Concerns were raised that if perceived as negative, MSF feedback will have no or negative effects.
In contrast to this, Fidler  found that physicians who considered or initiated changes had lower mean scores than those who felt no changes were necessary, indicating they used their feedback to guide changes.
Overeem (Overeem et al., 2012) found specialists reporting lower scores from colleagues reported making more changes, indicating they were responding to the results positively. Lockyer  proposes possible lack of value of the feedback as a reason for lack of change in surgeons.
Source and format of the feedback was also cited as an influential factor in performance changes. Sargeant (Sargeant, Mann & Ferrier, 2005;Sargeant et al., 2003;Sargeant et al., 2007) stated physicians generally agreed with their patients' feedback feeling patients were the most appropriate to assess their practice. However, responses to medical colleague and co-worker feedback was varied. Most frequently used feedback was specific, addressed communication and was from patients, the least used feedback addressed clinical competence and was from medical colleagues. Sargeant (Sargeant, 2006;Sargeant et al., 2009) also stated that many participants believed that facilitation of the feedback and reflective process could enable assimilation, acceptance and use of feedback for practice improvement.
Fidler and Hall Hall et al., 1999) found changes were most likely to be considered or implemented in response to their patient feedback.
Lockyer  proposed lack of change in surgeons was possibly because of the way scores were presented meaning they may not have perceived ratings as requiring attention, or that information was not based on surgical outcomes but focussed more on qualities such as communication.
Overeem (Overeem et al., 2009b) concluded that MSF feedback could be a positive force for practice improvement if: skilled facilitators were available to encourage reflection, concrete goals are set and follow up interviews carried out. Personal reflections being shared with colleagues was stated as helpful. A lack of openness and constructive feedback were identified as detrimental. Overeem (Overeem et al., 2010) found that variation in perceptions of usefulness and intentions to change as a result to MSF was because free text in some tools was valued, with participants feeling that mainly quantitative feedback was not specific enough to guide development. Multiple perspectives were also felt to be more likely to fuel change. In contrast to other studies, consultants said they valued co-workers, not patient perspectives. Overeem (Overeem et al., 2012) did regression analysis which found two variables had most effect on reported change; perceived quality of mentoring and negative scores from colleagues. The strongest evidence for positive effects on performance comes from studies externally assessing improvement, including an RCT (Brinkman et al., 2007) and a prospective longitudinal cohort (Violato, Lockyer & Fidler, 2008). However there were methodological issues and confounders identified in both and although Brinkman showed a statistically significant improvement following MSF there were wide confidence intervals indicating caution is needed with interpretation.

Discussion
Many of the other studies assessed self-reported changes or intention to change rather than measured change.
Intention to change cannot be proven to result in actual change (Webb & Sheeran, 2006) and it could be argued that healthcare systems are more concerned with behaviour change resulting in better patient care than intention to improve or self-reported improvements. However measuring impact on patient care in this area is difficult and health psychology theories such as theory of planned behaviour (Ajzen & Madden, 1986) and protection motivation theory (Rogers, 1985) assume intentions cause behaviour (Ferguson, Wakeling & Bowie, 2014).
Larger percentages of participants found the tools educationally useful than actually contemplated or initiated changes (Lipner et al., 2002;Overeem et al., 2010;Sargeant et al., 2003). This could be because scores can be positively skewed (Lipner et al., 2002), possibly due to self-selected cohorts, and doctors perceiving feedback as positive may not deem change necessary (Sargeant et al., 2007).
Of the studies that looked at intention to change or self-reported changes, most found that a majority of participants (over 50%) responded in a positive way to MSF and were either considering or had initiated changes following feedback Hall et al., 1999;Lipner et al., 2002;Overeem et al., 2010;Overeem et al., 2009a;Overeem et al., 2009b;Overeem et al., 2012;Sargeant et al., 2003;. However, some had small effect sizes with few participants responding positively; Sargeant found a minority reported changes, because they either deemed feedback positive therefore not requiring attention, or negative and either inaccurate or not useful (Sargeant, 2006;Sargeant et al., 2007). Another exception was a group of surgeons showing little interest in changing even if feedback indicated the need to .
Methodological quality varied extensively. The quality of the review is a subject of the quality of the studies. Most studies were conducted on small, uncontrolled, volunteer based samples with self-reported changes, which affects the strength of the findings. Some questionnaire based studies had low response rates; e.g. Overeem: 55% (over 60% is deemed satisfactory) (Overeem et al., 2012). Some studies used mixed methods and did not display data on educational impact  or were vague about precise numbers (Hall et al., 1999). Lockyer and Violato may have used similar populations and come to different conclusions ). Sargeant's samples (Sargeant, 2006;Sargeant, Mann & Ferrier, 2005;Sargeant et al., 2003;Sargeant et al., 2007;Sargeant et al., 2009) all relate to the same population from the pilot study in 2003, meaning findings could be over reported.
Methodological problems with my review include; literature was extensively searched however only English language articles were read meaning publication bias could not be excluded. Some unpublished studies may have been missed. There was only 1 reviewer, therefore a greater chance of having missed literature due to human error. I feel that despite methodological difficulties with the studies, this systematic review shows there can be an educational benefit from MSF with patient feedback. This is in fitting with literature from other fields such as psychology which have shown MSF can lead to small performance improvements over time (Smither, London & R, 2005). The size of the effect is not completely clear, due to the methodological issues, meaning results, even though they are generally positive must be interpreted with caution. The studies have also given insight into the important factors to consider to maximise benefit of MSF and minimise harm. For example, facilitation of feedback in order to promote effective reflection (Overeem et al., 2009b).The reflection literature proposes that reflection is a key step in integrating new information and learning (Moon, 2004). It also appears important to consider MSF tools carefully in order to gain maximum benefit; including having enough participants (Lelliott et al., 2008), multiple perspectives (Ferguson, Wakeling & Bowie, 2014;Overeem et al., 2010), patients Sargeant, Mann & Ferrier, 2005;Sargeant et al., 2007) and free text (Ferguson, Wakeling & Bowie, 2014;Overeem et al., 2010). It would also be pertinent to educate those asked to complete MSF feedback (Canavan et al., 2010;Saedon, Saedon & Aggarwal, 2010), as feedback is generally shown to be a positive force for learning over time , but needs to be specific, constructive and useful (Ferguson, Wakeling & Bowie, 2014;Overeem et al., 2009b;Sargeant, Mann & Ferrier, 2005;Sargeant et al., 2007).
This review also highlights the need for further research in this area, especially using external measures of change and comparative methods, able to more definitively demonstrate links between MSF, performance improvement and even improved patient care. It also highlights the need for consideration to be given to improve the MSF tool used by UK trainees currently.

Take Home Messages
Despite methodological difficulties with the studies, this systematic review shows there can be an educational benefit from MSF with patient feedback.
The studies have also given insight into the important factors to consider to maximise benefit of MSF and minimise harm. For example, facilitation of feedback in order to promote effective reflection MSF tools must be carefully designed to gain maximum benefit; including having enough participants, multiple perspectives, patients and free text .
There may be need to educate those asked to complete MSF feedback as feedback is generally shown to be a positive force for learning over time, but needs to be specific, constructive and useful This review also highlights the need for further research in this area and for consideration to be given to improve the MSF tool used by UK trainees currently.