Feedback preferences of patients, professionals and health insurers in integrated head and neck cancer care

Abstract Background Audit and feedback on professional practice and health care outcomes are the most often used interventions to change behaviour of professionals and improve quality of health care. However, limited information is available regarding preferred feedback for patients, professionals and health insurers. Objective Investigate the (differences in) preferences of receiving feedback between stakeholders, using the Dutch Head and Neck Audit as an example. Methods A total of 37 patients, medical specialists, allied health professionals and health insurers were interviewed using semi‐structured interviews. Questions focussed on: “Why,” “On what aspects” and “How” do you prefer to receive feedback on professional practice and health care outcomes? Results All stakeholders mentioned that feedback can improve health care by creating awareness, enabling self‐reflection and reflection on peers or colleagues, and by benchmarking to others. Patients prefer feedback on the actual professional practice that matches the health care received, whereas medical specialists and health insurers are interested mainly in health care outcomes. All stakeholders largely prefer a bar graph. Patients prefer a pie chart for patient‐reported outcomes and experiences, while Kaplan‐Meier survival curves are preferred by medical specialists. Feedback should be simple with firstly an overview, and 1‐4 times a year sent by e‐mail. Finally, patients and health professionals are cautious with regard to transparency of audit data. Conclusions This exploratory study shows how feedback preferences differ between stakeholders. Therefore, tailored reports are recommended. Using this information, effects of audit and feedback can be improved by adapting the feedback format and contents to the preferences of stakeholders.


| INTRODUCTION
Much effort has been devoted to improve professional practice and outcomes in health care during the past decades, unfortunately with varying effects. A widely used strategy to improve health care is "audit and feedback", 1,2 defined as any summary of clinical performance of health care over a specified period of time, given in a written, electronic or verbal format, offering professionals performance information and motivation to improve. 3 One of the methods to derive the information for audit and feedback is using quality indicators. 4,5 Quality indicators are aimed at detecting suboptimal care either in structure or process (eg, the percentage of patients discussed in multidisciplinary team meetings), or outcomes (eg, patient-reported outcomes [PROs] and experiences [PREs]). They can be used as a tool to guide the process of quality improvement in health care. 6 Although positive effects of audit and feedback have been reported, namely decreased duration of hospital stay 7 and decreased mortality rates, 8 this improvement strategy has not been found to be consistently effective. 2,[9][10][11][12] So far, research has focussed on increasing the effectiveness of feedback, for example by including a worksheet in the feedback to facilitate goal setting 13 and timing of audit and feedback. 3,14,15 Audit and feedback researchers have recommended a shift towards comparative effectiveness studies, evaluating how and when audit and feedback components will work, rather than its overall effectiveness. 16 The format of feedback may significantly affect the interpretation of data. [17][18][19] However, there is only limited information available regarding formats of feedback, for example on how to summarize and display results of outcome measures in the best way. [20][21][22] Furthermore, implementation of audit and feedback is likely to be more effective when feedback messages can influence barriers to change behaviour.
These barriers appear to differ across individuals. 23 In addition, most audit and feedback interventions use written or graphical feedback in one uniform format for all recipients. 7 This will surely not meet the preferences of all recipients, and effects will be low if recipients do not understand the feedback. In developing feedback formats, it is therefore necessary to involve all stakeholders receiving feedback, | 1277 van VEVELD Vet vEl. so as to guarantee that the presentation of feedback meets their preferences. 20,24 In health care systems worldwide, various stakeholders use feedback on quality indicators for different purposes, such as: (i) patients, who are the recipients of health care and for whom feedback on PROs and PREs can be used to improve and monitor their own or others' health and health care pathways; (ii) medical specialists, who deliver health care and for whom the feedback on their own delivered care may improve health care; (iii) allied health professionals, including nurses, who have a similar role as medical specialists, although restricted to allied health care; and (iv) health insurers, who search for quality information suitable to create differences in quality of care levels as a basis for their contracting. We hypothesize that by adapting feedback to the preferences of these different stakeholders, they will better respond to the information delivered, and more improvement in effects of audit and feedback could be possible.
In this exploratory study, we aim to investigate the preferences of various stakeholders on receiving feedback, with the Dutch Head and Neck Audit (DHNA) as an example. Head and neck cancers (HNCs) are heterogeneous both biologically as well as in clinical behaviour, and they grow relatively fast in an anatomically and functionally complex area. 25,26 Patients often have problems with speech, swallowing and physical disfiguration due to treatment, 27,28 requiring the collaboration of both medical specialists and allied health professionals. Therefore, high-quality integrated care for patients with this type of tumour is needed. 29,30 The DHNA uses quality indicators to measure the quality of integrated care for patients with HNC within 14 Dutch hospitals. 31 By investigating the preferences on feedback of all four stakeholders in the DHNA (medical specialists, allied health care workers, patients and health insurers), including "Why," "On what aspects" and "How" do you prefer to receive feedback on professional practice and health care outcomes, this study can provide useful tools to potentially improve quality of care by adapting the feedback format and contents to stakeholders' preferences. This can serve as an example for other integrated oncologic care pathways where audit and feedback will be used or, unfortunately, is still less effective.

| Study design
In this exploratory, qualitative study the first author conducted semistructured interviews with four stakeholders to investigate preferences on feedback using the "consolidated criteria for reporting qualitative research" checklist (COREQ). 32 Interviews were transcribed verbatim and qualitatively analysed by the first and third author.

| Setting
Approximately, 3000 patients are diagnosed yearly with HNC in the Netherlands. 33 HNC care is centralized in 14 hospitals: eight Head and Neck Oncology Centres (HNOCs) and six affiliated centres. The affiliated centres have committed themselves to using the same treatment protocols as the related HNOC. The various medical specialists and allied health care professionals involved in HNC care are united in two national foundations: one for medical specialists (NWHHT) and one for allied health professionals (PWHHT). Previously, there were two Dutch patient associations: "Stichting Klankbord" and "NSVG". The former represented all patients with HNC, the latter only laryngectomized patients. Currently, they collaborate in one Dutch patient association called "Patiëntenvereniging Hoofd-Hals". In the Netherlands, there are four major health insurers as well as several smaller companies. In 2014, a quality registration was set up to measure the quality of integrated HNC care, using quality indicators selected by the four stakeholders. 31

| Participants
Four different groups of stakeholders were interviewed about their preferences. Research shows that 13-15 interviewees are usually sufficient to reach data saturation (the point at which no new information is mentioned in interviews) 34 . Therefore, at least 13 persons were invited for each stakeholder group. However, only the four major health insurers were invited.
A patient panel (including the chairmen of both patient associations) that participated in a previous study was asked by e-mail to participate again (van Overveld, 2016, unpublished). A letter with additional information about the research methods and an informed consent form were handed over to the patients at a meeting prior to the interview. The location for the personal appointment was either at their home, their work or at the hospital. Medical specialists and allied health professionals and nurses, belonging to the national foundations, were invited to participate in an interview, either by telephone or in person. We aimed to interview at least one professional of each profession (radiation oncologist, medical oncologist, oral and maxillofacial surgeon, otorhinolaryngologist, speech therapist, physiotherapist, dietician, oral hygienist and nursing consultant) involved in HNC care. We contacted the four major health insurers by e-mail, to ask whether they would be willing to participate in an interview, either by telephone or in person. Persons approached were specialized in health care purchasing policy, innovation and advice or innovation and quality. Prior to an interview by telephone or a meeting, the professionals, patients and health insurers received a document with examples of the type of graphs to be discussed (see Result section Table 6, first column). In this article, the term "professionals" will be used when referring to medical specialists together with allied health professionals, and "allied health professionals" refers to both allied health professionals and nurses.

| Data collection
Each interview took approximately 20-30 minutes and was audio- "On what aspects would you prefer to receive feedback regarding professional practice and health care outcomes?", for example interest in specific indicators; (iii) "How do you prefer to receive feedback on professional practice and health care outcomes?", for example frequency, timing, report form, type of graph preferred and transparency, for example whether patients prefer to receive national average scores on PROs and PREs and whether results of quality of care in hospitals can become public. In addition, the interviews with patients were focussed particularly on the PROs and PREs with regard to questioning health care outcomes, because patients had a better understanding of the feedback on these questions compared with feedback on, for example, survival. Questions for the health insurers focussed merely on the goal of feedback, because they will use feedback in a different way compared with patients and health professionals. Different graph types were selected from feedback reports used in other research or found on the Internet, for example a bar graph, pie chart, line graph, point graph, area graph, box plot, Kaplan-Meier graph or a funnel plot. Moreover, a distinction was made between graphs for outcome indicators such as survival and PROs and PREs, because, in general, different graphs are used for both types of data.

| Analysis of interviews
Interviews were transcribed verbatim and qualitatively analysed using ATLAS.ti (version 7). 35 The first two interviews of each stakeholder group were coded independently by the first and third author (LO and TV) (female, MSc, first author; male MSc, third author; both working in the same research institute). All identified items were compared and discussed until consensus was reached. The remaining interviews were coded by the first author and checked by the third author to enhance the reliability and validity of the results. The same two researchers then categorized all identified items into the interview topics.
Subcategories of all codes dealing with the same subject were made by the two researchers within each category, resulting in a code tree.
For example, a division into three subcategories was made within the category "Why do you prefer to receive feedback?": individual level, hospital level and national level. Or, in the category "How do you prefer to receive feedback", all codes regarding distribution of the report were compiled, thereby forming a subcategory. Disagreement was discussed between the two researchers and if necessary with the last author (RH) (female, PhD, last author) until consensus was reached.

| Study population
For the patients as stakeholders, a response rate of 76% was reached, because three patients did not participate due to time constraints or did not respond to the e-mail or reminder. A total of eight patients and the chairmen of both patient associations participated in the semistructured interviews, all in person ( Table 1).
The medical specialists and allied health professionals had a response rate of 94% and 69%, respectively. Reasons for not participating were time constraints, the person did not belong to the board of the national foundation for allied health professionals anymore or the person did not respond to the e-mail or the reminder. A total of 15 medical specialists (n=15) and nine allied health professionals participated in an interview (n=9), either by telephone (n=18) or in person (n=6) ( Table 2).
The professions of these members included three radiation oncologists, two medical oncologists, five oral and maxillofacial head and neck surgeons, five otorhinolaryngologist head and neck surgeons, one speech therapist, two physiotherapists, two dieticians, two oral hygienists and two nursing consultants. Furthermore, the health insurers had a response rate of 75%, because one health insurer was not willing to participate. In total, three health insurers participated in an interview, either by telephone (n=1) or in person (n=2).

| Preferences
Tables 3, 4, 5 and 6 present an overview of the preferences of patients, professionals and health insurers regarding the three topics. In the following paragraphs, the preferences have been summarized. In addition, Figure 1 presents quotes from different stakeholders on the main research questions.

| Feedback on professional practice & health care outcomes
In general, all stakeholders prefer feedback on professional practice and health care outcomes ( Health insurers specifically stated that it is not their aim to judge hospitals for the good work they deliver, but to apply feedback as a discussion tool in their interactions with care providers. Health insurers consider feedback to be a necessary tool to improve care for the patient (eg, by informing the patient and representing their interest based on the feedback). In comparison, professionals consider feedback to be a method to improve care together with the patient. • Find a balance between giving feedback and giving too much information • Give an overview of the results first, followed by the details • Present it in such a way that one can easily understand without explanation Use of average scores: • Give feedback on own scores compared with the average score, the best hospital and the worst hospital when data will be presented anonymously • Give the scores of all hospitals including national average scores, the best and the worst performing hospital Distribution of feedback: • Feedback by e-mail • First, the hospitals can try to work it out on their own, then they can ask for more background information or explanation of the investigator • Organize a committee to monitor the content and format of the feedback report • Take case mix into account • Give feedback on the quality of data • Use specific themes each year when data will be compared on a national level • National feedback in the form of a conference is a useful idea; however, feedback in your own organization will be useful as well

| Feedback on PROs and PREs
The main reason for patients to want to receive feedback on PROs and PREs is to be able to engage in the discussions with peers or profes-

| On what aspects do you prefer to receive feedback?
Patients would prefer to receive feedback on the professional practice that matches their health care pathway; for example, the patient does not want to receive feedback on the professional practice of the physiotherapist if the patient did not receive any physiotherapy at all (Table 4). Medical specialists and health insurers alike mention that health care outcomes are most relevant when they can be compared with the aspects of professional practice, because they deal with the effect of the treatment.
In contrast with medical specialists, allied health professionals mention more frequently that they are more interested in feedback on the professional practice of their own discipline. However, both groups agree that feedback on all health outcomes and aspects of professional practice is needed, because they also form part of the health care pathway of the patient.

| Frequency and timing
Patients prefer to receive feedback when the specific health outcomes and aspects of professional practice have become relevant in their disease process. They prefer to receive this feedback by e-mail or through a patient portal. In terms of frequency, patients mentioned that, for all indicators (including PROs and PREs), feedback once a year would be sufficient (Table 5). Patients would prefer to receive feedback for the first time after the diagnostic phase, because then their stress level will be lower compared with during the diagnostic phase.
Both medical specialists and allied health professionals agree that feedback should be given more often in the start-up phase of a quality registration. In this way, users will get used to receiving feedback and will act on it.
Medical specialists and allied health professionals differ on the frequency of feedback: medical specialists prefer to receive feedback on process indicators (1-4 times a year) more often compared with outcome indicators (1-2 times a year). However, for allied health professionals, this is exactly the opposite.

| Report form
Patients mentioned that feedback should be well balanced and an explanation of the figure or graph should be given. There is no information available on how the health insurers prefer to receive feedback because they prefer to receive raw data to develop their own figures. Professionals agree that the report should be simple as well as giving an overview of the indicators, followed by more in-depth information. In addition, they are all in doubt about displaying average scores or specific scores of hospitals in public. They fear that it could result in reputational damage when the hospital is pictured as a lesser performing hospital. Professionals agree that feedback should preferably be given by e-mail.
In contrast to medical specialists, allied health professionals prefer to receive the indicators of their own discipline first, followed by the remaining indicators. In addition, allied health professionals would prefer a meeting around the feedback with more background information.
Medical specialists prefer to discuss feedback within their hospital before asking for more background information.
Furthermore, prior to giving feedback on PROs and PREs to patients, medical specialists feel that professionals should question the preferences of the patient regarding receiving their own results or the results of the general population. Professionals should also ask patients whether results on PROs and PREs might be consulted by professionals.

| Transparency
Patients and professionals alike are cautious about transparency of data.
They are worried about the quality of data and the risk of misinterpretation. Medical specialists suggest organizing a committee to decide on issues concerning transparency. In contrast, allied health professionals are in favour of making data public and have less stringent requirements for making data public compared with medical specialists. Health insurers mention that they feel a duty to take responsibility to the population. In order to improve care, it is important to visualize delivered care.
F I G U R E 1 Quotes from different stakeholders on the main research questions

| Type of graph for feedback on indicators
Patients mentioned that feedback figures for professional practice are difficult to read for patients in general (Table 6). In contrast, figures for health outcomes are easier to read for patients. Professionals also confirm that patients might not be able to read the feedback on health outcomes and professional practice.
For both health outcomes and professional practice, patients as well as professionals prefer bar graphs because they are easy to read. Other preferred graphs for medical specialists are Kaplan-Meier graphs and box plots for survival indicators and process indicators, respectively. Allied health professionals mention that box plots, Kaplan-Meier graphs and funnel plots give a less clear overview and are more difficult to interpret.

| Type of graph for feedback on PROs and PREs
Patients mention that figures for this kind of feedback are easier to read compared with figures for process and structure indicators (  Our study confirms that feedback is a method for reflection and for creating awareness, resulting in a change in behaviour. 1,22 Also, patients and professionals mention that knowing the hospital's scores on PROs and PREs can improve the quality of care. Greenhalgh showed already that the use of PROs in clinical practice is valuable in improving the discussion and detection of health-related quality of life problems. 38 In line with previous literature and irrespective of the stakeholder, simple formats, such as bar graphs, were generally preferred to more complex graphical information. 20,39,40 Regarding PROs and PREs, our study shows that both a pie chart and a bar graph are preferred by patients. Professionals have a slight preference for a pie chart over a bar graph. Hildon et al. 41 described that patients often prefer a bar graph, because it is a clearer graph visually. Moreover, patients' preferences for a bar graph are in line with Kuijpers et al. 20 In addition, Hildon et al.

| DISCUSSION
described that a funnel plot was difficult to read for patients, which our study confirms as well. 41 Although our patient population prefers a figure over plain text, they would also prefer an explanation to go with the figure. This is in line with Brundage et al., who stated that patients did not wish to receive HRQL information out of context or without explanation, 39 and also with Tufte, who gave an overview of the characteristics that a well-readable graph should have. 42

| Limitations
The fact that only three health insurers participated in the study could be considered a limitation. This is probably too small to reach saturation (the point at which no new information was mentioned in the interviews). 34 However, the health insurers shared the same thoughts on the topics discussed. Bias may have occurred when selecting the patients, because it is possible that patients with a higher social status and adequate communication skills were selected by each hospital, resulting in a less representative patient population. HNC is associated with poor socio-economic circumstances. 43 In the interviews, it became clear that it was difficult for patients to understand the feedback regarding health outcomes, such as recurrence rates. Therefore, the interviews with patients were directed mainly towards the use of feedback on PROs and PREs, when we spoke about "health outcomes". Questions for the health insurers merely focussed on the goal of feedback, because the insurers mentioned that they prefer raw data instead of receiving a complete report.
Lastly, there might be an overvalue of positive preferences for feedback. This study shows that all stakeholders are positive about receiving feedback on professional practice and health care outcomes.
However, if this view would manifest itself in action, you would expect that the literature on implementation of audit and feedback would show much larger and more consistent effect sizes. This is similar to the situation in which adherence to clinical guidelines is still low and clinicians often overstate their adherence to the guidelines. [44][45][46] Knowing how stakeholders prefer to receive audit and feedback does not assure that they will actually respond to it. Therefore, it is necessary to test the response in practice.

| CONCLUSION
This exploratory study shows that preferences for receiving feedback between patients, professionals and health insurers differ regarding content but not regarding layout. Therefore, reports tailored to these preferences are recommended. Using this information, the effect of audit and feedback can be improved by adapting the feedback format and contents to preferences of stakeholders. As a result, this could potentially improve quality of care. A next step is to test in practice to what extent professionals actually respond if audit and feedback suit their preferences.