How psychiatrists’ attitudes towards multi-source feedback including patient feedback influenced the educational value: a qualitative study [version 1; peer review: awaiting peer review]

Background:  Multi-source feedback (MSF) is well-established in psychiatric training. However, evidence on the educational impact is not definitive and there is scanty evidence exploring its value for professional development of psychiatry trainees in the United Kingdom (UK). Evidence suggests the MSF tool currently used is not suitable for specialist trainees. This qualitative research project explored psychiatric doctors’ attitudes towards MSF with patient feedback, to determine how this influenced the feedback’s educational usefulness. Methods: A qualitative study using a phenomenological approach based on a constructivist approach. Purposive sampling identified trainee psychiatrists who completed a more extensive MSF, including patient feedback, than they currently use. They discussed their results in supervised sessions to plan how to use the feedback. Semistructured interviews were conducted separately with trainees and their supervisors following completion of MSF. The data was analysed thematically. The study was completed in 2020. Results: Seven trainees and five supervisors participated. Four themes were identified. Most had positive opinions about the educational usefulness of MSF, including patient feedback, and made changes to their behaviour following the feedback. Interviewees valued patient feedback and identified it as important in psychiatry. Most valued their patient feedback over their colleague feedback. The complexities of patient feedback in psychiatry and how this may influence the educational usefulness of the feedback were discussed in detail. Conclusions: Findings suggest a need to review the current system of MSF in psychiatry in order to maximise educational benefits. In Open Peer Review Reviewer Status AWAITING PEER REVIEW Any reports and responses or comments on the article can be found at the end of the article. MedEdPublish Page 1 of 8 MedEdPublish 2022, 12:5 Last updated: 03 FEB 2022


Introduction
Multi-source feedback (MSF) is established as a key learning facilitator in medical training in the united Kingdom (UK) 1,2 . UK post-graduate psychiatry trainees currently use MSF without patient feedback. This qualitative study explored the effect of MSF with patient feedback on the educational impact of the MSF process.
MSF gives doctors an opportunity to 'measure' humanistic qualities which, combined with scientific understanding, make up expert clinical judgement 3 . The focus on communication and interpersonal relationships is particularly pertinent for psychiatrists for whom multi-disciplinary team working and professional relationships with patients are so important 4,5 .
There are a variety of tools used to gather MSF. Most seek feedback from peers and colleagues, but not all ask patients for feedback. The mini-Peer Assessment Tool (mini-PAT) does not include patient feedback yet is the most commonly used MSF tool by UK psychiatric and some other speciality trainees 6 . The evidence suggests that it may not be the best way of assessing specialist trainees 7 . Consultant Psychiatrists are recommended to use the ACP 360 to gather MSF; this does include patient feedback 8 . Patient feedback is important, both for patient safety and for maximising learning potential. Following the Mid-Staffordshire crisis, the Francis and Berwick reports emphasised the importance of listening to the patient voice, for example through patient feedback 9, 10 . Feedback, particularly from patients, has been identified as one of the strongest influences on learning and educational development 11,12 . Doctors themselves identify it as educationally useful and often feel patients are the people best-equipped to assess their performance [13][14][15][16][17][18] .
Research has been undertaken to guide educators on how to maximise the educational benefits of MSF. The doctors' pre-existing perceptions of MSF influence its educational impact. In particular views on the usefulness, value and accuracy of feedback influence the likelihood of a doctor implementing change 14,19 . Valence of feedback also influences impact on behaviour, with positive feedback being more likely to prompt change 14 .
There is currently no available evidence relating to the use of MSF with patient feedback by UK psychiatry trainees' and how their attitudes towards feedback influence its educational usefulness; the present study investigated this issue.

Methods
We conducted a qualitative study using a phenomenological approach based on a constructivist paradigm, to understand and describe the lived experience of trainees and supervisors in relation to receiving and reflecting on MSF including patient feedback 20,21 . Semi-structured interviews are frequently used in phenomenological qualitative research in order to glean in-depth information on how a person thinks about an experience.

Setting and participants
The study was derived from data collected for a broader Clinical Education MSc and was undertaken October 2018 to July 2020. We interviewed UK Psychiatric trainee doctors and their supervisors; doctors from Cornwall were included first for convenience purposes.
Trainee psychiatrists were purposively nominated by the lead author to provide maximum variance in the sample 20 . Initial contact with potentially eligible doctors was made by the medical staffing department via email; if they gave permission, they were individually contacted via their work e-mail to ascertain eligibility. Trainees' supervisors were recruited to triangulate the findings and provide another opinion on the level of impact on learning from MSF 22 . Written and verbal information about the study was given to potential participants, and all participants provided written informed consent before study initiation.
Trainees obtained MSF using the ACP 360 tool and were asked to reflect on the results with their supervisor to generate a plan for implementation of change.

Data collection
The interview guide (Extended data) 23 was developed following a comprehensive literature review and was iteratively revised by the lead author. Semi-structured, audio-recorded, interviews between 30-and 60-minute long were carried out in private, at a place of the participants choosing, by the female lead author (CM), at least a month after the trainees had completed the ACP 360 and made their plans in supervision. CM is a Consultant in Liaison Psychiatry as at time of writing; she completed this qualitative research as part of a MSc in Clinical Education and was provided expert supervision by LS. The participants worked in the same trust as CM and were assured in writing that their interview responses would in no way affect their training or work and that only anonymised quotes would be published. Trainees and supervisors were interviewed separately. Participants were asked to share accounts of their experiences of MSF with patient feedback and how they felt it had influenced their or their trainees' practice.
Data collection and analysis occurred concurrently, allowing for iterative revisions of the interview guide and continued until data saturation was reached 24 . Based on background review the sample size was initially speculated to be approximately twenty (ten trainees and ten supervisors) 25 .

Data analysis
Interviews were transcribed verbatim, although not returned to participants for review, after which the data was anonymised and the recordings deleted. Two investigators (CM, MA) independently performed line-by-line coding on the initial transcripts. In order to triangulate the coding and minimise researcher bias, the research assistant (MA) and lead author (CM) separately coded the transcripts, CM used NVivo software 26 and MA coded manually. Coding began when the first interview was complete and a 'zigzag approach' was used whereby data analysis contributed to data collection 27,28 . Initial codes were recorded highlighting recurring concepts, concepts identified as relevant from the literature review, and concepts specifically labelled as important by the participants. Through an inductive process, the identification of codes facilitated the development of taxonomies 29 enabling the development of themes as part of the thematic analysis. Codes were compared and discussed and a coding hierarchy or 'tree' was initiated. This was applied to all data and codes were continually checked and rephrased, and new hierarchies were developed until a consensus on the correct codes generated for each transcript was reached. The researchers discussed and decided on the grouping of codes to form the themes which emerged from the coding hierarchy. They continually checked for agreement and discrepancies, which were resolved through conversation and reflection 20 . A summary of the results was shared with participants for comment and review. This report has been written with reference to the COREQ Reporting Guidelines 30 .

Ethical considerations
This study was performed in accordance with the Declaration of Helsinki and was approved by the Higher Research Authority and Plymouth University Ethics Committee.
Throughout the study design, implementation and analysis process, the study team reflected on how our professional positions, equality and relationships could influence the data and its analysis 20,31,32 . The interviewer (CM) was a Consultant Psychiatrist at Cornwall Partnership NHS Foundation Trust where the trainees and supervisors worked. In an attempt to reduce any biases in reporting, participants were interviewed separately and assured their answers would not affect their training, and that only anonymised quotes would be published. To minimise individual bias, we regularly engaged in discussion to compare, challenge and confirm our interpretations.
Higher Research Authority (HRA) approval was sought via the Integrated Research Application System (IRAS; IRAS number 242554) 33 . Plymouth University Ethics committee also provided University ethical approval and capability and capacity assessment was agreed with Cornwall Partnership NHS Foundation Trust (CFT).

Results
Seven trainees (T) and five supervisors (S) participated in the ACP 360 and interviews. Each trainee and supervisor pair were assigned a corresponding number -e.g. T4 and S4. One supervisor had two trainees, they were assigned the S1 and T1a and T1b respectively. Two trainees dropped out of the study; the reasons voluntarily given were time constraints and sickness. Four themes were identified from the data analysis: these are described below.

Demographic information
Demographic information for the trainee participants is included in Table 1.
All supervisors were Consultant Psychiatrists qualified as Clinical or Educational Supervisors and experienced in supervising trainees. Many had completed the ACP 360 themselves in the past. Four of the supervisors were male and one female. Four of the participants' ethnicity was white British and one participant's ethnicity was Indian.
Theme 1: Attitudes towards MSF including patient feedback The majority of participants had positive attitudes towards MSF including patient feedback. Doctors valued the more comprehensive MSF, particularly given the importance of communication and the doctor-patient interaction in psychiatry.
T3: "It was really good to see assessment of things such as your emotional intelligence… I personally valued feedback about the more personable, more human aspects of the job…I think it's much more relevant for Psychiatry trainees to be filling in the ACP 360…I think that's quite an invaluable tool for being a Psychiatrist" It was identified as being more useful than the current MSF tool used for UK psychiatric trainees (mini-PAT).
S2: "I'm trying to think why I liked it better than the mini-PAT... I found the free text comments just a bit more focussed…. it just seems a better framework to draw out things to work on" Although not all doctors were so enthusiastic: S4: "I think people are often a bit jaded and cynical about multi-source feedback as a whole" There were a variety of reasons offered for some participants' cynicism about MSF. These included concerns about summative use of feedback, as MSF in psychiatry is currently used in trainees' annual review of competence. There were also significant challenges getting enough patient feedback for it to be useful. The ACP 360 has the potential to collect 25 patient feedback forms, but trainees struggled to obtain enough within the time frame of the research. This was identified by some as reducing their confidence in the MSF due to doubts about its validity.
T1b:" you can get away with, is it five pieces of feedback from each? Which isn't a lot of people really…I don't know how accurate or useful it is for that reason" Potential reasons for this offered by participants included patient fears over loss of anonymity affecting their care, or people simply becoming frustrated with being asked for feedback. However, since patients were not interviewed, exact reasons for this are unknown.
There were also concerns about self-selection of assessors and the general anonymity of the MSF affecting the usefulness. The changes were made at a variety of Kirkpatrick levels 23 , but observed changes reached level 3b (measured or observed changes in behaviour).
One trainee reported they had not found the ACP 360 particularly useful, but had taken action from it anyway and this was observed by their supervisor. This action was identified as not particularly important. This trainee reported their feedback to have been less positive. Some of the changes were more difficult to define, i.e., a change in mind-set.
Trainees also reported emotional effects from the feedback. Most reported a generally positive emotional experience and this was linked to how positively they viewed their feedback.
T3: "I was chuffed to read some of the positive comments… I think I will always magnify negative comments that much more than some other people might, but I don't find myself too overtly destroyed and damaged" Some participants reported negative emotional effects and these doctors were less likely to attribute educational usefulness to the process

T1a: "I suppose that did have an emotional impact; you're feeling criticised for something that's largely out of your control… the aspect of it that had the emotional impact was this kind of sense of the feedback being a mecha-nism… for one patient in particular, venting frustrations… I did feel a bit deflated about it and it took some time to unpick well what's to do with me and what's to do with others? "
Theme 3: Opinions about the usefulness of patient feedback compared to colleague feedback in psychiatry All participants had opinions about patient feedback in MSF, despite the fact some trainees had never previously collected patient feedback. Most were generally positive about patient feedback, even if they felt their own patient feedback had not been particularly helpful. They described this as being due to the nature of psychiatric work, due to the importance of communication and interpersonal interactions.

T4:" it's very important, the relationship you have with your patients, more so than other specialties… in psychiatry you'd really struggle if you weren't able to establish rapport and don't have the kind of emotional intelligence"
The majority of trainees felt their patient feedback had been more helpful, as patients were the best placed to assess certain skills, being the end-receivers of these skills.

S1: "I mean really we all know what gives good feed-back… giving people a diagnosis which is socially acceptable, not challenging their difficult behaviour and telling them
there is a medication that will take away their problems… but that's not the best way to practice" There were concerns that feedback may be used to complain or express frustration about factors external to the doctor, such as service or funding deficits.

T1a: "one was a lengthy rant, more about service provision than me….it went into a very long complaint about how this person felt let down and how services weren't there for them"
However, most trainees received very positive patient feedback. Although this meant they were often more enthusiastic about it, there was a lack of suggestions on how to make changes. Potential reasons proposed by the participants for overly positive feedback were linked to patient concerns over loss of anonymity and how that would affect their care S4: "… scores were exceptional…He's a very good trainee… but it becomes difficult to know what that really means…and it was difficult to get any subtlety out of the scoring"

Discussion
This research was the first in the UK to explore the attitudes of psychiatric trainees and their supervisors towards an extensive MSF with patient feedback (the ACP 360). The process was generally viewed as positive and led to observable changes in practice; however, the complexity of patient feedback in psychiatry was highlighted as a key challenge.
This study highlights how the individual doctor's perception about the importance, validity and usefulness of MSF forms the foundation for educational benefit. As explored in the theme 'Attitudes towards MSF including patient feedback' most participants perceived the process as positive and responded to the feedback in a positive way, consistently with previous research 13, [34][35][36][37][38][39][40][41] . Specifically, participants valued the exploration of more human aspects of their practice such as emotional intelligence; this was felt to be particularly, if not uniquely, relevant to psychiatry. The participants' view of patient feedback as intrinsically valuable within psychiatry may account for our results contradicting previous studies, which found many doctors perceive feedback as positive and therefore not requiring attention, or negative and consequently inaccurate or unhelpful 15,42 . This was reinforced in the third theme exploring how patient feedback compared to that of colleagues was viewed.
Most trainees valued patient feedback over their colleague feedback, consistently with previous studies that found that the source of doctors' feedback is influential [13][14][15]38 . Specifically, the view that patients are the 'end user' of a doctor's medical skills means a higher value is placed on patient feedback and therefore it is more likely to prompt reflection and change [13][14][15] . There were however some studies of consultants suggesting that co-workers' views, rather than patients', are regarded more highly 38 .
The first theme also explored that participants reflections on the motivation and perceived validity of the feedback process influenced usefulness and impact. Concerns were raised over the summative use of MSF, as echoed in the literature 2 .
Confidence in the validity of the MSF results was influenced by challenges in obtaining enough patient feedback and subsequently impacted participants' attitudes to the process. Participants' feelings towards this were in line with current evidence suggesting feedback from eight medical colleagues, eight co-workers and 25 patients is required to achieve adequate reliability and generalisability 43 . Concerns were also raised about the anonymity of the process and self-selection of assessors, which is echoed in the literature 6,44 .
A key concern for educators is how any educational process results in change and the second theme identified in this study explored the effects of the feedback on participants. This has been widely studied, and although some studies have methodological limitations, the results have been broadly positive [13][14][15]19,[34][35][36][37][38][39][40][45][46][47][48] . In this study, some effects such as changes in communication style were observable by the trainees' supervisors. Other subtle, less measurable changes were identified, such as altered confidence, the impact of which can be profound and longlasting 14,15,49,50 . This was a particular issue when participants perceived their feedback as less positive and consequent emotional impacts were identified. These participants put less emphasis on the importance of the changes they made and were less likely to make changes. This supports previous research suggesting that feedback that is discordant with a doctor's self-perceptions is less readily accepted and results in psychological distress 14,15 . Of note, receiving feedback which had a negative emotional impact has been shown, in some cases, to lead to performance deterioration 50 .
The fourth theme explored the complexity of patient feedback in psychiatry. This has not been identified in the research currently available and is a novel contribution of this study. Both positive and negative feedback prompted concerns that issues such as the complex nature of the doctor-patient interaction, the patients' mental state and service difficulties in psychiatry could influence feedback and therefore educational usefulness. Overly positive feedback was also problematic with concerns raised about the educational usefulness, as well as the validity of the feedback.
The main limitations of this study include the sample size and participants being from one training area. However, the sample size was flexible and data collection continued until saturation was reached, and therefore findings are still relevant. The ethical approval process was followed to recruit doctors from another site, however due to data saturation being reached, this was not necessary. There is no reason to suggest problems with generalisability, and results broadly correlated with those found in the literature, meaning generalisability is likely to be good. Although qualitative research is less able to prove causality i.e., whether MSF directly led to improved performance, this research was able to give much richer detail about the complexity of this topic. This study was limited to UK psychiatric trainees. However, as the mini-PAT is used for many UK speciality trainees and there is limited evidence of its educational usefulness 2,7,44 , findings from this research could be interesting to a variety of doctors in training across specialities.
Overall, the dependability and rigour of the study was high, due to peer debriefing, member checking, triangulation and careful documentation and consideration to the rationale behind all aspects of the study, among other reasons.
A larger national study, exploring trainees completing extensive MSF containing patient feedback would be essential, particularly if rolling this out to trainees was considered. Mixedmethodology research could quantitatively measure changes to doctors' performance and gather more qualitative data. Including patient participants, in order to explore potential reasons for not giving feedback, along with opinions on improving feedback, would also be necessary.

Conclusion
MSF with patient feedback was viewed as a positive and educationally useful experience by participants and led to observable changes in practice. The process also led to emotional effects for trainees which influenced opinions about usefulness. The complexity of patient feedback in psychiatry was discussed in detail. This has not been extensively explored in previous literature, and several issues were raised which influenced the doctors' opinions about the validity of feedback and could inform future discussion on how to maximise the benefit from patient feedback.

Data availability
Underlying data The anonymised transcripts from the semi-structured interviews contain detailed reflections on the feedback and educational process which would make the data potentially identifiable.
The consent forms (and as such IRAS approval) allowed for publication of anonymised quotes from the coded data only.
The coded data is available upon written request from the corresponding author (merryn.anderson@nhs.net). Read-only access to anonymised coded data will be granted to researchers wishing to review data in order to contribute to further research.