A coproduced patient and public event: An approach to developing and prioritizing ambulance performance measures

Abstract Background Patient and public involvement (PPI) is recognized as an important component of high‐quality health services research. PPI is integral to the Pre‐hospital Outcomes for Evidence Based Evaluation (PhOEBE) programme. The PPI event described in detail in this article focusses on the process of involving patients and public representatives in identifying, prioritizing and refining a set of outcome measures that can be used to support ambulance service performance measurement. Objective To obtain public feedback on little known, complex aspects of ambulance service performance measurement. Design The event was codesigned and coproduced with the PhOEBE PPI reference group and PhOEBE research team. The event consisted of brief researcher‐led presentations, group discussions facilitated by the PPI reference group members and electronic voting. Setting and participants Data were collected from eighteen patient and public representatives who attended an event venue in Yorkshire. Results The results of the PPI event showed that this interactive format and mode of delivery was an effective method to obtain public feedback and produced a clear indication of which ambulance performance measures were most highly favoured by event participants. Discussion and conclusions The event highlighted valuable contributions the PPI reference group made to the design process, supporting participant recruitment and facilitation of group discussions. In addition, the positive team working experience of the event proved a catalyst for further improvements in PPI within the PhOEBE project.


| INTRODUCTION
Patient and public involvement (PPI) is recognized as an important component of good-quality health services research internationally and in the UK is viewed as central to national health research policy by the Department of Health (DH), National Health Service (NHS) and National Institute of Health Research (NIHR) [1][2][3] The Research Governance Framework (RGF) for Health and Social Care 2 states that research should be "pursued with the active involvement of service users and carers including, where appropriate those from hard to reach groups" and that patients should be involved at every stage of the research process where appropriate. "Hard to reach groups" (also termed "seldom heard") may be defined as those from minority or socially disadvantaged groups for example minority ethnic, LGBT (lesbian, gay, bisexual, transgender) or homeless people, people with chronic mental illness, drug users or criminal offenders. 4 Patients may be involved in a consultation role (researchers seek the views of patients and public on key aspects of their research); a collaborative role (an on-going partnership between researchers, patients and the public throughout the research process); or publicly led (public and patients design and undertake the research). As most NHS-related research is publicly funded, patients and public have a right to be involved to help improve the NHS and their own health-care outcomes and experiences. Thus, patients must move from being "mere users and choosers to being makers and shapers of health services." 5 There is a compelling argument that patients offer unique insights and knowledge of a clinical condition or experience of care that researchers may not possess. In this way, patients can help researchers to focus on meaningful and relevant issues, improving the overall quality and credibility of research. There is still considerable debate around the best methods to incorporate PPI into high-quality research. Here, we describe one way this was attempted.

| Aims of this investigation
The aims were (i) to assess whether a coproduced, face-to-face PPI prioritization event was an effective method of obtaining public feedback and (ii) to find out whether outcome measures considered by health service professionals in a Delphi study were also important to patient and public representatives.
The focus here was the PPI event design and execution. Andy Irving, the PPI lead for the research team, and the PhOEBE PPI reference group members, who provided direct quotes, were the main authors of this article.

| PhOEBE research programme
The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5-year National Institute for Health Research (NIHR) funded research programme which aimed to develop new ways of measuring the quality, performance and impact of pre-hospital care provided by ambulance services. PPI played an important part in the programme: PPI representatives were coapplicants and involved with design of the programme which involved two ambulance services,

| Patient and public involvement in PhOEBE
The PhOEBE PPI group met on a regular basis with a named PPI lead from the PhOEBE research team (Andy Irving, AI), working to an agreed "terms of reference" document (Supporting Information). One PPI representative was also a member of the Project Management Group (PMG) and Study Steering Committee (SSC). This ensured a lay perspective on significant decisions within the project was considered and so acted as a link between the research team and PPI group.
At the beginning of the PhOEBE project, potential ambulance performance and quality measures were identified from two systematic reviews of related policy and evaluation research. These were then prioritized using a three-stage consensus process: Stage

| Codesign phase
The broad aim of the codesign phase was to develop a more interactive way to listen to those who used and cared about ambulance services beyond a mere "tick-box" exercise while also meeting the requirements of the PhOEBE research programme.
Our specific objectives for the event were for participants to:

| Setting and participants
As everyone was considered a potential patient of the ambulance service, the PPI reference group wanted a representative and diverse sample of participants, ensuring that measures and indicators developed would be relevant, of value and understandable to any patient or members of the public who might wish to interpret them. Efforts were made to invite patients and the public from diverse backgrounds to represent the various potential ambulance service users, particularly those "hard to reach" groups who might not traditionally access such an event.
Participants were recruited through publicizing the event via email letter and flyer to over 20 PPI groups and networks (Supporting Information). The PPI reference group cascaded the invitation via their own networks to other patient and public groups in the Yorkshire, Humber and Lincolnshire areas.
There were no explicit inclusion or exclusion criteria as we wished the event to be accessible and open to all and were fully prepared to make any reasonable adjustments to enable participants to attend and engage. A non-academic venue, with good transport links, was thought to be the best option; travel reimbursement and a monetary gift in line with INVOLVE good practice were offered to all PPI participants. 7  At the end of the event, a summary of the results from the nine voting rounds was presented. To achieve objective 5 and 6, a final researcher-led PowerPoint presentation explained how these results would feed forward into the next steps of the PhOEBE project.

| Event format
Finally, participants were also given the opportunity to provide feedback about the event itself both on paper feedback forms and using anonymous voting.

| Key results on the process of the event
Overall 16 individuals registered to attend the event and 14 attended (88%) representing people from three broad participant categories: "hard to reach" groups (n=3), students and aged under 25 years (n=3), and representatives from local and regional patient involvement and advocacy networks (n=8). A full list of event invitees and attendees is presented in an Supporting Information. Participants answered a brief set of evaluation questions at the end of the day using the Turning Technologies voting method to see whether the event had met its objectives. A member of the PPI Reference Group participated in the feedback increasing the numbers from 14 to 15 (Table 1) .
Feedback via voting at the end of the event confirmed the first four objectives were fully met. Participants were not asked to consider objective 3 as it was evident this objective had been met as votes had recorded participant's views on measures considered most important.
Extra questions confirmed that all participants had enjoyed the day and that 14 of 15 (93%) felt that such an event was a useful model for future PPI work. To give participants further opportunity to give feedback, paper forms were also used ( Table 2).
In "additional comments" boxes, participants also made some very positive statements about the event: "Good use of voting technology" "Fascinating group discussion. Very good way of choosing answers.
Great level of expertise" "Good provision of resources" "Good balance of debate, reflection and voting" "Aimed at just the right level for me" Clearly, the vast majority of participants felt they had understood the aims of the day, felt the objectives had been met and enjoyed this method of involvement, specifically that the format and mode of delivery made this an effective method to obtain public feedback.
Participants also raised issues around things that could have been improved F I G U R E 1 Voting round 1 of 9, the pain measures voting slide "Difficult to choose some points as ideally you would measure everything" "More regional spread of the general public" "Try to spread to youth services, eg young carers" There were several costs associated with the development and delivery of this event outlined in Table 3 below.

| Key results on the outcomes of the event
The PPI event produced a clear indication of which measures were most highly favoured by participants (see Table 4 voting results from the PPI event). The highest-ranking measures are presented according to the percentage of votes achieved (see Table S1, S2 and S3 for a full list of measures and votes).
Alongside other key considerations, the results of the PPI event guided the research team to select 5 of the 9 most highly voted measures to be included in the final measures for further development.

| DISCUSSION
The PPI event provided a clear indication of measures preferred by event attendees using a format that was considered useful, informative and relevant. It also added value in other ways. The PPI reference group had an opportunity to extend their influence and involvement particularly in relation to participant recruitment, discussion facilitation and content of resource packs provided to event participants.
Closer working with the PhOEBE PPI reference group and research team at all stages of the event proved a catalyst for further improvements in PPI in the project. Increased contact and communication with the PPI lead also created closer collaborative relationships between the research team and PPI reference group members that helped support further PPI activities.
Following the success of the event, the PPI reference group were inspired to codesign a poster to share best practice from their experiences. The poster was presented by PPI members at two national con- The PhOEBE PPI reference group demonstrated willingness to be "makers and shapers" as research collaborators. This was made possible by mutual respect, commitment and positive attitudes between the research team and PPI reference group, meaning the latter were willing and able to take on this task. Developing trust and teamwork of this nature takes time and resources. Without this, there was a danger that disingenuous attempts to co-opt members of the public and patients into pseudo-collaborative roles, while maintaining total control of the research process would only reinforce and replay divisions between researchers and patients. 10 We can be proud of what has been achieved since [2014] and how things have definitely become more PPI focused and co-collaborative Each of the three-stage consensus process provided a key function: the Multistakeholder consensus event identified key concepts related to ambulance service quality and performance; the Delphi process was used to develop and refine measures related to these concepts; the PPI event then allowed PPI members to engage with and provide an input into the prioritization process.

| Strengths and limitations
Venue hire, catering, PPI payments for event attendance, travel expenses and the appointment of a paid external independent facilitator in total cost just over £4k. We acknowledge that these are funds that not all projects have. However, as research funders are often proactive in ensuring PPI is properly funded, it is the researchers' responsibility to appropriately consider and budget for such activities within grant applications. Marsden and Bradburn 11 recommend that an external facilitator is used in such involvement activity, as being independent of the subject of enquiry may help in developing collaborative working. The experience of the research team, including the PPI reference group, suggested that the external facilitator was a particular strength and helped the PPI and research team to deliver a successful event.
Staniszewska et al. 12  Emergency pre-hospital care is defined by its short-term transitory nature. Everyone is a potential user of ambulance services but few people would identify themselves as regular users and those who do may be atypical. This can make involving patient and the public in emergency care research challenging if no one identifies themselves as potential beneficiaries of such research or is willing to speak up on behalf of patients who use emergency care. Groups like the SECF 16 have enthusiastic and committed members like Maggie and Dan, with wide-ranging knowledge of pre-hospital and emergency care who provide critical patient perspectives within research and are not afraid to advocate on behalf of this patient group. 17 The PPI event benefited from service users of a local addiction treatment service attending. Such groups are typically hard to access and may not ordinarily attend such a research event despite being potential users of the ambulance service and so of direct relevance to them. In this way, as endorsed through the feedback and evaluation process at the event, the added value of our carefully considered PPI friendly methodology served to empower disadvantaged or typically stigmatized groups in society. This was made possible using the researchers' (AI) contact with local drug and alcohol services and service users, highlighting the value in building good relationships with local community groups.
The primary objective of this event was to obtain feedback from a wider PPI audience on ambulance service performance measurement. We did not set out to "do research" on the PPI participants themselves; therefore, demographic or other data were not collected from the PPI event participants. As a result, the representativeness of the participants in terms of age, gender, disability/impairment, ethnicity, etc. cannot be commented upon. Despite efforts to invite PPI from diverse backgrounds, no claim to have achieved a representative sample can be made.

| CONCLUSION
While there is no single correct method for involvement, there are some key ingredients that researchers and PPI may wish to adopt.
The PhOEBE PPI reference group was instrumental in the design and execution of the PPI event but to achieve this took time, patience and teamwork. We should be clear that to deliver such an event also takes significant staff resources. The role of the PPI lead was important in building relationships, developing trust, communicating and in maintaining momentum for involvement within the PhOEBE project.
The RAPPORT 14 PPI evaluation concluded that developing good relationships and having a dedicated PPI coordinator, either internal or external of the team, is significant in providing effective PPI.
In conclusion, this article has presented a method of involvement, which proved effective in obtaining patient and public feedback on complex, little known aspects of ambulance service performance measurement and in building capacity for further PPI within the PhOEBE project.