Priorities for methodological research on patient and public involvement in clinical trials: A modified Delphi process

Abstract Background Despite increasing international interest, there is a lack of evidence about the most efficient, effective and acceptable ways to implement patient and public involvement (PPI) in clinical trials. Objective To identify the priorities of UK PPI stakeholders for methodological research to help resolve uncertainties about PPI in clinical trials. Design A modified Delphi process including a two round online survey and a stakeholder consensus meeting. Participants In total, 237 people registered of whom 219 (92%) completed the first round. One hundred and eighty‐seven of 219 (85%) completed the second; 25 stakeholders attended the consensus meeting. Results Round 1 of the survey comprised 36 topics; 42 topics were considered in round 2 and at the consensus meeting. Approximately 96% of meeting participants rated the top three topics as equally important. These were as follows: developing strong and productive working relationships between researchers and PPI contributors; exploring PPI practices in selecting trial outcomes of importance to patients; and a systematic review of PPI activity to improve the accessibility and usefulness of trial information (eg participant information sheets) for participants. Conclusions The prioritized methodological research topics indicate important areas of uncertainty about PPI in trials. Addressing these uncertainties will be critical to enhancing PPI. Our findings should be used in the planning and funding of PPI in clinical trials to help focus research efforts and minimize waste.


| INTRODUCTION
Growing awareness of the importance of patient centeredness in  3,4 The emphasis on patient centeredness in research stems from a belief that involving patients in decisions about how studies are designed and conducted improves research, making it more relevant to end users 3,5-7 and reducing waste. 8,9 Patient involvement is also believed important for moral reasons, based on the principle that the people whose lives are most affected by research should have a say.
In the UK, patient involvement is known as patient and public involvement (PPI). 5,10 In clinical trials, PPI tends to involve a small number of patients or members of the public (known as PPI contributors). 11 Some PPI contributors will have direct personal experience of the condition being investigated, whilst others bring general experience of being a patient or service user. A key consideration is that PPI contributors are in a position to offer a distinctive perspective to researchers or clinicians. Many UK funders require researchers seeking funding to provide evidence of how PPI will inform their studies. [12][13][14] Despite the emphasis on PPI in the UK and internationally, there are uncertainties about how best to implement it, 15 about the purpose of PPI and whether it actually does improve research. 10,12,15,16 Concerns have been raised about tokenism and resourcing in PPI, about the difficulty of ensuring diversity and avoiding professionalization among PPI contributors, 10,17,18 complexities with researchers and patients sharing power, 19 and inadequacies in training and support for both PPI contributors and researchers. 20 Problems with the conceptualization and meaningful assessment and measurement of PPI have also been identified. 21 Each of these concerns points to different priorities for methodological research on PPI. Reviews of PPI in research and other contexts identify many topics for future research. 4,[21][22][23][24] Although not all reviews focus specifically on clinical trials, trials are regarded as particularly likely to benefit from PPI 20,25 by helping to address the many methodological issues that arise within trials. 5 Most reviews of PPI echo similar concerns to those identified in the above paragraph, pointing to the need for: agreed tools for measuring PPI and its impact across the different phases of research, 15,24,26,27 for investigations of how best to support PPI 6,23,28 and for optimal models of implementing PPI. 29,30 However, many of these topics have been identified by PPI researchers and it is unclear whether these priorities are shared by the wider community of trialists and PPI stakeholders. Given the diversity of stakeholders involved in PPI, there is considerable potential for divergence in the prioritization of topics to investigate, and therefore for dilution of research efforts in investigating how to improve PPI in research.

In the METHODs for Patient and Public Involvement In Clinical
TriALs (METHODICAL study), we conducted a modified Delphi process to identify the priorities of a broad range of PPI stakeholders for methodological research to resolve uncertainties about PPI in clinical trials, as well as to help improve to the design of future PPI research and avoid unnecessary duplication of research effort.

| MATERIALS AND METHODS
Delphis are used in health and social science research as a means of involving participants with relevant experience, via a multistaged study, to achieve consensus on a given topic. 16,31,32 This involves conducting Participants: In total, 237 people registered of whom 219 (92%) completed the first round. One hundred and eighty-seven of 219 (85%) completed the second; 25 stakeholders attended the consensus meeting.
Results: Round 1 of the survey comprised 36 topics; 42 topics were considered in round 2 and at the consensus meeting. Approximately 96% of meeting participants rated the top three topics as equally important. These were as follows: developing strong and productive working relationships between researchers and PPI contributors; exploring PPI practices in selecting trial outcomes of importance to patients; and a systematic review of PPI activity to improve the accessibility and usefulness of trial information (eg participant information sheets) for participants.

Conclusions:
The prioritized methodological research topics indicate important areas of uncertainty about PPI in trials. Addressing these uncertainties will be critical to enhancing PPI. Our findings should be used in the planning and funding of PPI in clinical trials to help focus research efforts and minimize waste.

K E Y W O R D S
clinical trials, Delphi, patient and public involvement, research priorities sequential anonymous surveys to collect, collate and present results back to the group. To help achieve consensus, participants can view and revise their own responses in the light of group responses. 32 The process can be modified to include opportunities for feedback or a consensus meeting so that participants can discuss their views. 33,34 We designed a modified Delphi, comprising a literature review to identify topics for research on PPI, followed by a two round online survey and stakeholder consensus meeting.
We established a study team of 17 PPI stakeholders from across the UK to oversee the METHODICAL project, including: four PPI Coordinators, eight PPI researchers, one PPI planner, two PPI contributors, one non-lay reviewer and one lay reviewer. Seven members of the team had secondary PPI-related roles.

| Patient involvement
Our study team included three patient partners who were involved in all aspects of study design and conduct, including development of protocol, pilot topics and accompanying text, survey recruitment, interpretation of study findings and review of this manuscript.
Approximately half of the consensus meeting places were allocated to patients. We will send participants a summary of the study findings.
The summary will also be placed on the study website and promoted through social media platforms used by patients.

| Recruitment
To help maximize the utility of our findings, we aimed to include all key paid and unpaid roles of people who co-ordinate, support and contribute to PPI in trials. Individuals were eligible to participate in the Delphi process if they had at least 12 months' experience within a PPI role in clinical trials. Study team members did not participate in the survey. As definitions of roles in PPI vary, the study team identified seven stakeholder groups to inform recruitment, consulting with our PPI partners to select terminology to define each group (Table 1). We provided this list of stakeholder groups and accompanying definitions in recruitment materials. A free text field was included at registration so participants could elaborate on their role/s and self-identify their role if they felt this was not included in the list. The study team agreed that for the feedback of results in round 2 to be meaningful at the level of stakeholder group, approximately 10 participants per group would be required.
We used snowball sampling to identify stakeholders 35 using personal contacts and Internet searches to develop a database of individuals, organizations and networks under each of the seven stakeholder groups. The METHODICAL researcher (AK) sent emails to the identified organizations, networks and individuals (Table S1) with study information. The email included a request to invite potential survey participants by distributing the study invitation to their members or contacts. AK also placed an advert and link to the survey on the "People in Research Forum" (www.peopleinresearch.org).

| Development and pilot of topics
We used online search engines (eg Google Scholar and OVID (MEDLINE), organizational databases (eg INVOLVE library) and hand searches of citations within key articles to identify literature that systematically evaluated the scope and impact of PPI within health research 15,20,24 to develop a list of potential methodological research topics for round 1 of the Delphi. This was supplemented by reviewing recent publications assessing PPI specifically within clinical trials. 22,23,27,36 For each topic, we developed accompanying descriptive text to help explain these. The study team, including PPI partners reviewed the list of topics and accompanying descriptions to ensure they were distinct and clearly communicated challenges associated with PPI in clinical trials. Methodological research in this context was described to participants in study information materials as: "methods, practices and procedures of PPI in clinical trials." We piloted the list of topics with a small group of lay (n=2) and non-lay (n=3) PPI stakeholders to check clarity and understanding and then refined the list of topics and descriptive text ( Figure 1).

| Online survey
The online Delphi was conducted between November 2015 and  In round 1, stakeholders registered for the study by indicating their name, email address, which of the seven stakeholder groups they had the most experience in, years of PPI experience, consent to participate, interest in attending the consensus meeting and interest in receiving a copy of the published findings. We assigned each registered user a unique identifier to ensure anonymity and enable linking of scores between rounds. Participants then scored the importance of each of research topic using a scale of 1-9, with scores 1-3 being not critical or low importance, 4-6 important but not critical and 7-9 of critical importance. 37 Selecting a score of 10 indicated an abstention from scoring an individual topic. Participants were also invited to suggest additional topics to be added to round 2. Participants who registered but did not start the survey, or partially completed round 1 questions, were excluded from the analysis and not invited for round 2. The study team reviewed additional topics suggested by participants in round 1 for inclusion in round 2.
In round 2, we showed participants bar charts summarizing the distribution of the percentage of scores 1-9 for each topic from each stakeholder group. We then invited participants to revise or keep their own score from the previous round. The email invitation for round 2 indicated that responses received within 10 or 17 days would be entered into prize draws for a £50 voucher or a F I G U R E 1 Overview of the Delphi process £30 voucher, respectively. AK sent email reminders periodically to non-responders.

| Consensus meeting
We allocated thirty places to equal numbers of lay and non-lay stakeholders with broad representation across the seven stakeholder groups. The METHODICAL study team were invited to attend and participate in the consensus meeting. Three study team members helped to facilitate the meeting and did not take part. Ten other study team members registered to attend as participating stakeholders and were allocated either lay or non-lay places based on their primary PPI roles. We invited survey participants at random within their stake-  (Table S4). PW began by asking attendees if any clarification of the topic was required. Comments and discussion were then encouraged before PW asked attendees to consider whether or not the topic should be prioritized for future research. Where more than 70% of round 2 participants in any one stakeholder group had indicated a topic was of high importance (scored it 7-9), we invited attendees to raise opposing arguments. A similar approach was followed for those topics where less than 50% of round 2 participants in any one group had indicated a topic to be of less importance (scored it 1-3), with views requested if a participant felt strongly that a topic should be con- AK circulated a written report to meeting attendees seven weeks after the meeting, which included notes from meeting discussions and any changes made to the topic description text.

| Statistical analysis
We pre-defined consensus as 70% or more participants scoring from 7 to 9 and less than 15% participants scoring from 1 to 3 on a particular topic. 38,39 All statistical analysis was performed in R version 3.2 (R Foundation for Statistical Computing , Vienna, Austria. www. R-project. org/). We ranked final research topics from the METHODICAL consensus meeting according to the percentage of participants scoring a research topic as critically important (scores 7-9) and then by ascending order of the percentage of scores 1-3.

| Online survey
Of the 237 people who registered for the survey, 219 (92%) completed round 1. Twelve individuals registered but did not start the survey and six provided partial responses ( Figure 1)

| Consensus meeting
Of the 30 people registered, 25 attended and were eligible to vote ( acteristics of PPI which lead to a successful trial) was considered to be too broad. We made changes to three topic titles and nine descriptive texts after group discussion in order to clarify the topic before voting (Table S2). Table S3 provides the final ranked list of all research topics. Sixteen topics achieved consensus with greater than 70% of participants scoring them 7-9 and less than 15% scoring them 1-3. As shown in Table 3, the top 10 prioritized research topics were varied, covering PPI processes, resources, practices and relationships between stakeholder groups. Three topics shared joint "first place" with 96% of meeting attendees rating each as critically important: developing strong and productive working relationship between researchers and PPI contributors; PPI practices in selecting trial outcomes of importance to patients; and a systematic review of PPI activity in improving the accessibility and usefulness of trial information (eg leaflets and information sheets) for clinical trial participants.
As discussed previously, an additional topic, regarding the definition of PPI and people's understanding of it, was added to round 2. to increase diversity and capture wider patient or public perspectives on clinical trial designs, such as online surveys and social media.
Some of the top ten topics focus on the impact of PPI and particularly the need to review PPI in specific trial processes, such as: the development of trial information for patients; recruitment and retention of patients; choice and measurement of outcomes; and the dissemination of results. Two of these (Topic 28, strategies to recruit and retain patients, and Topic 29, the selection of trial outcomes) align with existing methodological research agendas for clinical trials. 38 Conceptually, PPI should have a substantial role in addressing these issues. However, our results demonstrate that further work is needed to map and formally evaluate current PPI practices to help make these more relevant to trials, 3,5-7 and help to reduce research waste by targeting resources more effectively. 8,9 For example, it is common to involve patients in developing information materials for prospective trial participants, 43  During the consensus meeting some prioritized topics were revised to define a research method to be used to explore that particular topic, such as Topic 31: "A systematic review of PPI activity in improving accessibility and usefulness of trial leaflets and information sheets for clinical trial participants'", whilst others, such as Topic 20 "Developing strong and productive working relationships between researchers and PPI contributors" are more wide ranging and relate to challenges in PPI. Such wider topics may contain multiple components, and further consideration will be needed to develop these topics into formal research questions and to identify the most appropriate research methods for addressing these questions. 32

| Strengths and limitations
Our study had several strengths. The METHODICAL team included representation of all stakeholder groups including lay and non-lay members, who oversaw all stages of the project, including the recruitment strategy. The survey sample size was also relatively large compared to other Delphi studies, and attrition was low. Comparison of round 1 mean scores between those who did and did not complete round 2 indicate that our study was not affected by attrition bias ( Figure S1).
We took several steps to help ensure that all stakeholder groups were represented at every stage of the Delphi and that all groups and individuals felt able to contribute freely. We sampled stakeholders purposively for the survey stage. For the consensus meeting a random selection of participants within groups ensured balance and fairness in the allocation places for lay and non-lay stakeholders across all seven of stakeholder groups.
High and low priority topics identified in our study are cited in international literature on public and patient involvement in research. [46][47][48] However, further research is required to explore the level of priority given to these topics in international settings.
The study also had some limitations. As the potential sample was large and diverse we were unable to fully define the sampling frame and used snowball sampling to try to make sure all stakeholder groups were included in the sample. As a result, our study was subject to selfselection bias among those who registered for the study. To minimize the burden of survey participation, we chose not to collect social or demographic information, such as ethnicity or socio-economic status; therefore, the diversity of participants and the potential impact of socio demographic characteristics upon the prioritization process cannot be evaluated.
Some study team members participated in the discussion and voting within the consensus meeting, which meant that a subset of attendees was not independent from the project. We reasoned that they would bring valuable experience and expertise to the discussion 49  Delphis are dependent upon the participants having time to commit to the process to completion. 50 To reduce the potential burden on participants and minimize attrition bias, we pre-specified a two round, rather than a three or four round survey. 33

| CONCLUSIONS
In conclusion, the prioritized methodological research topics identified by the Delphi process highlight key uncertainties about PPI in trials. Addressing these uncertainties will be critical to enhancing PPI. Our findings should be used by those involved in planning and funding of PPI in clinical trials to help focus research efforts and minimize waste.

ACKNOWLEDGEMENTS
The online survey was conducted using software designed and developed by Richard Crew and Duncan Appelbe at the CTRC, University of Liverpool. We would like to thank all the patients and researchers who took part in this study.

CONFLICT OF INTEREST
All authors have completed the ICMJE uniform disclosure form and declare: no support from any organization for the submitted work;