Agreeing on global research priorities for medication safety: an international prioritisation exercise

Objectives Medication errors continue to contribute substantially to global morbidity and mortality. In the context of the recent launch of the World Health Organization’s (WHO) Third Global Patient Safety Challenge: Medication Without Harm, we sought to establish agreement on research priorities for medication safety. Methods We undertook a consensus prioritisation exercise using an approach developed by the Child Health and Nutrition Research Initiative. Based on a combination of productivity and citations, we identified leading researchers in patient and medication safety and invited them to participate. We also extended the invitation to a further pool of experts from the WHO Global Patient Safety Network. All experts independently generated research ideas, which they then independently scored based on the criteria of: answerability, effectiveness, innovativeness, implementation, burden reduction and equity. An overall Research Priority Score and Average Expert Agreement were calculated for each research question. Findings 131 experts submitted 333 research ideas, and 42 experts then scored the proposed research questions. The top prioritised research areas were: (1) deploying and scaling technology to enhance medication safety; (2) developing guidelines and standard operating procedures for high-risk patients, medications and contexts; (3) score-based approaches to predicting high-risk patients and situations; (4) interventions to increase patient medication literacy; (5) focused training courses for health professionals; and (6) universally applicable pictograms to avoid medication-related harm. Whilst there was a focus on promoting patient education and involvement across resource settings, priorities identified in high-resource settings centred on the optimisation of existing systems through technology. In low- and middle-resource settings, priorities focused on identifying systemic issues contributing to high-risk situations. Conclusions WHO now plans to work with global, regional and national research funding agencies to catalyse the investment needed to enable teams to pursue these research priorities in medication safety across high-, middle- and low-resource country settings.


VIEWPOINTS PAPERS
Most studies of the frequency and nature of medication error have come from high-resource country settings. These have found that around 2-3% of clinical encounters in primary care settings and 10% involving hospital in-patients will result in errors [3][4][5][6][7][8][9][10]. Not all will be clinically important, but medication errors have been found to be the commonest resulting in harm [11]. Research in low-resource health care settings suggests a broadly comparable frequency of medication-related harm [6][7][8][9][10][11][12]. This body of work is challenging to interpret, because of the heterogeneity in the definitions used, patient populations studied and methodologies employed. Despite these challenges, the over-riding message from this body of evidence is very clear: medication errors affect patients of all ages, both sexes, occur globally across all health care settings, and, most importantly, they are largely preventable.
The World Health Organization (WHO) established two previous Global Patient Safety Challenges. Each Challenge sought to highlight a major patient safety problem impacting all countries and health systems. The first of these, Clean Care is Safer Care (2005) focused on reducing health care-associated infections and the second [13], Safe Surgery Saves Lives (2007), aimed to improve the safety of surgical processes [14]. Building on the success of these global initiatives, WHO launched the third WHO Global Patient Safety Challenge: Medication Without Harm in 2017 [15]. This seeks to facilitate a range of strategic initiatives with the aim of improving medication safety globally. Research is fundamental to these global, regional and national efforts. To inform these deliberations, we sought to define research priorities for medication safety using an inclusive, systematic and replicable process.

Origins
We established a management team to identify global research priorities for medication safety. After reviewing available approaches for research prioritisation, we decided to employ the Child Health and Nutrition Research Initiative (CHNRI) method [16]. This research priority setting technique was introduced in 2007 and has now successfully been applied in over 100 different exercises, a number of which have also been led by WHO [17] (see Box 1). Revised guidelines for application of the method, based on the experience of its use, have recently been published [18][19][20][21][22][23]. This followed extensive deliberation on what was learnt from working with funders, researchers and other stakeholders over the last decade, as well as the development of a revised conceptual framework and validation of the key concepts that CHNRI relies on. In June 2017, we developed a protocol to guide the process for setting priorities in medication and patient safety research globally. A small management team (including the authors of this report) coordinated the steps of the WHO priority setting exercise.

Expert input
We identified 598 experts in medication safety from across the world. We did so by searching the Web of Science' s Core Collection for the most productive authors in the preceding five-year period, or those who Box 1. The CHNRI method for setting research priorities The CHNRI method uses the principle of crowdsourcing to score ideas against a pre-defined set of criteria. This enables funders and policymakers to view the strengths, the weaknesses, and relative ranking of each proposed research idea, based on submitted opinions of a larger number of experts. This method uses a systematic, transparent, and democratic approach to priority setting. While it allows researchers to independently generate and score research questions, it also involves funders, policymakers, and other stakeholders at an early stage of the process, ensuring their ownership of the outcomes. The CHNRI method has thus far been implemented in about 100 studies led by multilateral organisations (eg, WHO, United Nations International Children' s Emergency Fund (UNICEF)), national governments (eg, India, South Africa), and funders (eg, The Bill and Melinda Gates Foundation) to set research priorities in areas ranging from the reduction of global child mortality, dementia, or disability to the efficient execution of national health plans (eg, in China). The recognised advantages of this method include its systematic nature, transparency and replicability, clearly defined context and criteria, involvement of the funders, stakeholders and policy makers, a structured way of obtaining information, informative and intuitive quantitative outputs, studying the level of agreement over each proposed research idea, and independent scoring of many experts, thus limiting the influence of individuals on the rest of the group [18][19][20][21][22][23][24].

VIEWPOINTS PAPERS
were lead authors of the top 1% most cited research articles. The key words used to identify the experts were "medication safety" and "patient safety". After removing duplicate names, we invited the resulting 457 researchers to participate. Each expert was invited to generate up to three research ideas and then systematically rank these using pre-agreed criteria (detailed below). We in addition approached 190 persons who expressed interest in medication safety through the WHO' s Global Patient Safety Network. A total of 131 invitees agreed to participate and submitted their research ideas.
The management team then scrutinised the submitted ideas and ensured that the wording of each idea fitted the format for the scoring process. This led to a consolidated list of 333 unique research ideas, which were then thematically organised into 33 broader categories, each containing between three and 30 research ideas.
The larger participating scorer group, comprising 42 experts, scrutinised the list of questions and agreed on the context and the criteria for scoring. The context was defined as "global." This meant that some proposed research ideas were scored differently because they were not feasible in specific settings. We asked the scorers to state whether the context they considered for the scoring process was for high-resource or low-resource settings. The resulting data were used for sub-analyses (see later).

Criteria
The timeframe within which the results were expected from proposed research was 5-10 years. Six independent criteria were agreed and used to discriminate between the many proposed research questions identified: (i) Answerability: Is this research question likely to be answered using the proposed methods and approaches?
(ii) Effectiveness: Is this research question likely to lead to interventions that will effectively reduce the burden of medication-related harm?
(iii) Innovativeness: Is this research question truly novel, making good use of overall technological and scientific progress?
(iv) Implementation: Is this research question likely to lead to interventions or solutions that could be readily implemented?
(v) Burden reduction: Is this research question likely to lead to a significant reduction in medication-related harm?
(vi) Equity: Is this research question likely to reduce inequity in the population?
All invited contributors were asked to score each submitted research question using these pre-defined criteria.

Scoring
Experts were offered four response options for scoring: 0 (unlikely to meet the criterion); 0.5 (not sure if it can meet the criterion); 1 (likely to meet the criterion); or left blank if the expert felt insufficiently informed to make a judgment. The scores for each criterion ranged from 0-100%, and the overall research priority score (RPS) assigned to each research question was a simple mean of all six criteria-specific scores. Average expert agreement (AEA), defined as the level of agreement among scorers, was also calculated for each research question, as the frequency of the mode (ie, the most common score divided by the total number of scores).

RESULTS
We received scores for the 333 proposed research ideas from 42 experts (see Table S1 in Online Supplementary Document). Most experts (n = 27) scored with a high-resource setting in mind. Five experts did not indicate which setting they had in mind while scoring. We present the scores from the overall pool of 42 experts in Table 1 and Table S2 in Online Supplementary Document, and the results by resource context in Table 2 and Table S3 in Online Supplementary Document (high-resource context), and in Table 3 and Table S4 in Online Supplementary Document (low-resource context). Table 4 shows the 10 research questions with the most divergent scores based on the measure of AEA, which has a maximum theoretical range of 25%-100%. Table 5 shows the lowest ranked research priorities. RPS -research priority score, AEA -average expert agreement *Specific scores, ranging from 0-100, are presented for each of the 6 priority-setting criteria: answerability, effectiveness, innovativeness, implementability, potential for burden reduction and equitability. Questions are ranked according to their overall research priority scores (RPS), which also has a maximum theoretical range of 0%-100%. Average expert agreement, which can theoretically range from 25%-100%, is also provided for each question.

VIEWPOINTS PAPERS
The overall three highest-ranking research questions (with RPS 86.7-89.2) focused on: • Assessing how the incidence of harm due to prescribing errors can be reduced by different interventions in low-resource settings; • Assessing the prevalence, main factors responsible and effective interventions for preventing severe avoidable medication-related patient harm in low-resource settings through pilot studies; • Identifying affordable and effective methods of improving medication literacy among patients in resource-limited settings. Figure 1 illustrates the key priorities for low-and high-resource settings.
Below, we consider the emerging themes, areas of overlap of priority areas, and discrepancies in more detail.

Recognised importance of low-resource settings
Our analysis shows that almost half (9 out of 20, 45%) of the top research priorities across experts related to identifying and addressing medication-related harm in low-resource settings (see Table 1). The prioritised questions included identifying existing incidence and prevalence of harm, training staff, identifying potential interventions (patient education/involvement, technology, effective medication reconciliation), and developing indicators. Three of the top 10 (30%) priorities identified by experts working in high-resource settings concerned tackling issues in low-resource settings (see Table 2). RPS -research priority score, AEA -average expert agreement *Specific scores, ranging from 0-100, are presented for each of the 6 priority-setting criteria: answerability, effectiveness, innovativeness, implementability, potential for burden reduction and equitability. Questions are ranked according to their overall research priority scores (RPS), which also has a maximum theoretical range of 0%-100%. Average expert agreement, which can theoretically range from 25%-100%, is also provided for each question.

Exploring how to optimise existing systems through technology
Investigating how technology can be used to optimise existing efforts aimed at reducing medication-related harm was identified as a priority, particularly in high-resource settings ( Table 2). The top 20 overall priorities comprised the development of predictive algorithms and risk scores (see Table 1), and included human factors considerations, and information presentation. Of the top 10 priorities identified in high-resource settings ( Table 2), four (40%) included technology (predictive algorithms and risk scores), technology in low-resource settings, and decision support systems. Pictorial information presentation featured in 20% of identified priorities, but it is unclear if these related to digital systems. Top priorities in low-resource settings did not include technology ( Table 3).

Focus on organisational environments, learning and people
Patient education and involvement was mentioned in 25% (5 out of 20) top overall research priorities ( Table 1). National strategies and policies were also mentioned, but only in 5% of overall research priorities ( Table 1).
30% of the highest ranking priorities in high-resource settings tackled patient education and involvement, but wider organisational and health system issues did not feature in the top 10 ( Table 2).
Priorities in low-resource settings focused on wider questions around organisational environments in which issues surrounding medication safety are situated at a facility, system and national level ( Table 3).
Here, the top 10 research questions tackled exploring differences between settings with varying levels of income (30%), identifying systemic issues contributing to high-risk situations (20%), and promoting patient education and involvement (20%).

Overlap between lowest priority and most divergent perspectives
The research questions with the most divergent scores centred on research topics in specific settings eg, investigating health care-associated infections, antibiotics, generic medicines, over-the-counter (OTC) medicines ( Table 4). Lowest-ranked research questions focused on single study designs in specific settings and medicines, but also included procedural issues such as identifying high-risk patients (see Table 5).  Table 3. Top 10 research priorities among the 333 proposed research questions based on the scores from 10 experts in medication safety who were scoring mainly with a low-resource context in mind, and who represent a subset of the 42 scorers* RPS -research priority score, AEA -average expert agreement *Specific scores, ranging from 0-100, are presented for each of the 6 priority-setting criteria: answerability, effectiveness, innovativeness, implementability, potential for burden reduction and equitability. Questions are ranked according to their overall research priority scores (RPS), which also has a maximum theoretical range of 0%-100%. Average expert agreement, which can theoretically range from 25%-100%, is also provided for each question.

EQUI-
There was substantial overlap between six research priorities that were included in both the 10 research questions (60%) with the most divergent scores and the 20 lowest priority research questions (30%).

DISCUSSION
Our priority setting exercise has shown that medication safety is a truly global problem. Medicines are inadvertently killing and harming patients on a scale that is unacceptable. When viewed globally, and not from the perspective of some leading health care providers in high-resource countries, there has been relatively little improvement since the subject was first documented. This means that neither research, nor its translation into practice, has played a part in protecting patients. Taking a fundamental look at research priorities in medication safety is a necessary step to create opportunities to transform this situation.
There was clear recognition by the experts who participated in our study of the need for research commissioners and funders to support ideas in scope to serve most of the world' s population, but also those that are innovative and with realistic prospects for implementation. At the opposite end of the priority spectrum were ideas either formulated too specifically or too focused on a specific context, or simply not clear enough. The significant overlap between the most controversial and the lowest priority areas suggests that the notion of lowest priority is somewhat contested. To conduct a study investigating the impact of procurement based on clinical efficacy and safety, with the use of longitudinal data analytics thereby optimising benefits and minimising harm. The scorers also were of the view that research must encompass low-resource settings, where the burden of medication-related harm is likely to be highest. Patient education and involvement were judged as priority areas across all care contexts. Priorities in high-resource settings focused on optimising existing systems through technology, whilst in low-resource settings priorities reflected the need for wider organisational changes.
The method we used to prioritise research needs is based on a systematic, well-documented process that is transparent and replicable. It is based on a collective opinion of experts. This eliminates the risk of individual scorers unduly influencing the overall score. We engaged many scorers (42 international experts), so there is a very high degree of confidence (>95%) that the ranking of priorities would not greatly change with a different group of scorers, unless major self-selection bias was present [23]. Differences in the contexts used to score are unlikely to have affected the scoring of most proposed research questions. They could however have had an important effect on proposed ideas that were considered not feasible in low-resource settings. This exercise also efficiently discriminated between a very large number of research questions, with RPS spread over a very wide range (27.4-89.2, see Table S2 in Online Supplementary Document).
In comparison to previously conducted CHNRI exercises, this work was characterised by a rather low response rate at all stages of scoring. The initial response rate in a typical CHNRI exercise is around 60%, while the response rate at the second stage tends to be around 50% [16]. The scoring of this study was, compared to other CHNRI exercises, burdensome for scorers, so it is possible that this may have turned away many of the invited scorers. That said, simulations have shown that a sample size of 42 scorers should still result in stable and replicable scores and rankings [22,23]. To support this, research questions that were similar in nature were ranked very closely together on the final list, which is one of the useful indicators of robustness of the CHNRI process.
Another limitation relates to the search strategy and sample characteristics of experts. The search strategy for experts may not be representative as the first-author, with the exception of major trials, is seldom the VIEWPOINTS PAPERS RPS -research priority score, AEA -average expert agreement *Specific scores, ranging from 0-100, are presented for each of the 6 priority-setting criteria: answerability, effectiveness, innovativeness, implementability, potential for burden reduction and equitability. Questions are ranked according to their overall research priority scores (RPS), which also has a maximum theoretical range of 0%-100%. Average expert agreement, which can theoretically range from 25%-100%, is also provided for each question.
VIEWPOINTS PAPERS most experienced author or "expert". There was also clearly an overrepresentation of participants from high-resource settings (31 out of 42), which is likely to have influenced the results in favour of concerns important to those working in these settings. There was further a potential overrepresentation of pharmacists/pharmacologists (18 out of 42), and medical doctors (11 out of 42).
Our findings provide detailed and actionable recommendations for policymakers and research funders to begin addressing the significant burden associated with medication-related harm internationally and in countries with different levels of resources (see Tables S2-S4 in Online Supplementary Document). Furthermore, the CHNRI process offers research funders and policymakers an opportunity to understand potential strengths and weaknesses of many research ideas when evaluated against pre-defined priority-setting criteria. This can lead to a multitude of follow-up actions, depending on individual preferences.
Across countries, research priorities reflected a need to identify incidence/prevalence of harm and associated interventions (particularly in low-resource settings) as a necessary first step to improve medication safety. This is aligned with the current efforts of WHO to identify the prevalence of medication errors in low-resource settings [15]. These settings have received limited attention in the literature [25]. It also reflects issues surrounding a lack of agreement on ways to define and measure medication-related harm [26,27].
Prioritised interventions tackled the alignment of informational requirements of the various stakeholders involved in the medication management process (including the patient). Whilst patient involvement reflects a wider international drive towards the increasingly active role of patients in the management of their own health and illness to improve quality of care [28], patients may also be viewed as part of a wider health information infrastructure surrounding medication-related information which is created and maintained by a wide variety of stakeholders [29].
This infrastructure may include both paper-based and electronic systems, depending on context. For instance, technology-based approaches to optimising existing medication management through secondary uses of data are more immediately relevant to high-resource settings that already have basic technological infrastructures in place [30]. Conversely, identified priorities in low-resource settings show that these countries need to focus on wider organisational changes to allow the emergence of basic informational infrastructures (technology-based or otherwise). This indicates that certain pre-requisites need to be in place before optimisation through technology can be considered [31].

CONCLUSIONS
Our method has identified and agreed research priorities for improving medication safety globally and by resource setting. WHO now plans to work with global, regional and national funding bodies to commission work pursuing these research priorities and through doing so contribute to efforts to reduce the unacceptable burden of medication-related harm.