Improving medication safety in the Intensive Care by identifying relevant drug-drug interactions - Results of a multicenter Delphi study Journal of Critical Care

Purpose: Drug-drug interactions (DDIs) may cause adverse outcomes in patients admitted to the Intensive Care Unit (ICU). Computerized decision support systems (CDSSs) may help prevent DDIs by timely showing relevant warning alerts, but knowledge on which DDIs are clinically relevant in the ICU setting is limited. Therefore, the purpose of this study was to identify DDIs relevant for the ICU. Materials and methods: We conducted a modi ﬁ ed Delphi procedure with a Dutch multidisciplinary expert panel consisting of intensivists and hospital pharmacists to assess the clinical relevance of DDIs for the ICU. The procedure consisted of two rounds, each included a questionnaire followed by a live consensus meeting. Results: In total the clinical relevance of 148 DDIs was assessed, of which agreement regarding the relevance was reached for 139 DDIs (94%). Of these 139 DDIs, 53 (38%) were considered not clinically relevant for the ICU setting. Conclusions: A list of clinically relevant DDIs for the ICU setting was established on a national level. The clinical value of CDSSs for medication safety could be improved by focusing on the identi ﬁ ed clinically relevant DDIs, thereby avoiding alert fatigue. the CC (http://


Introduction
Drug-drug interactions (DDIs) can lead to preventable adverse clinical outcomes [1]. A DDI occurs when one or more drugs affect the pharmacokinetics and/or pharmacodynamics of one or more other drugs [2]. Patients in the Intensive Care Unit (ICU) are particularly susceptible for DDIs, due to often-present impaired absorption, diminished renal and hepatic function, and polypharmacy [3]. In the ICU, DDIs cause 16% of all adverse drug events (ADEs), including both preventable and non-preventable ADEs [4]. ADEs in the ICU are associated with higher morbidity and mortality, prolonged length of stay, and increased hospital costs [5]. Computerized Decision Support Systems (CDSSs) that show DDI alerts provide an opportunity to reduce risks related to DDIs. Typically, DDI alerts warn prescribers about potential DDIs and provide advice to change the medication regiment and/or plan necessary additional monitoring. However, lack of a fit between the general knowledge base on which the DDI alerts are based and the clinical setting may cause alert fatigue and high override rates [6][7][8][9].
In the ICU, up to 90% of the DDI alerts are overridden and around 84% of these overrides are appropriate [10,11]. In contrast to other clinical wards, ICU patients are under continuous monitoring which enables timely detection and risk management of potential adverse effects of DDIs. Furthermore, the critical condition of ICU patients may require administration of interacting medications, despite the risk of adverse effects. Therefore, some DDI alerts may be of no clinical value or even unjustified in the ICU setting. A list of clinically relevant DDIs for the ICU setting, agreed upon by ICU experts, may help tailoring DDI alert content of CDSSs to the ICU setting. We hypothesize that such a list can result in diminishing alert fatigue and, in turn, decrease the risk of (unintentionally) overriding relevant alerts, and thus improve medication safety.
Therefore, the objective of this study was to identify which DDIs are considered clinically relevant for the ICU setting by conducting a modified Delphi procedure with a national multidisciplinary expert panel consisting of intensivists and hospital pharmacists.

Materials and methods
This Delphi study is reported in accordance with the CREDES reporting standard for Delphi studies [12] (Supplementary file 1). The Delphi method is an often used method to obtain consensus from experts on medication related topics [13][14][15]. In a modified Delphi procedure, the judgment and opinions from experts are collected using an iterative process that involves a questionnaire and a live consensus meeting [16]. Our modified Delphi procedure was based on the RAND Appropriateness Method [17], a type of modified Delphi procedure, and consisted of two rounds of a questionnaire and a live consensus meeting (Fig. 1). During both rounds experts were asked to judge each presented DDI on its clinical relevance for the ICU setting.
We did not ask our experts to make judgments about the severity and evidence level of each DDI, since this information is already present in the G-standard. This standard was originally developed for the primary care setting and nowadays serves as input for CDSSs in all Dutch hospitals. A national committee assigns severity and evidence levels to DDIs based on the Summary of Product Characteristics and review of literature [18]. This situation is unique, as in most countries the development and maintenance of DDI knowledge bases are not organized at a national level.
The DDIs were selected according to their severity level and frequency in the ICU. For that purpose, we extracted retrospective medication administration data over a period of seven years from six Dutch ICUs consisting of both small and large ICUs, situated in teaching and non-teaching hospitals and geographically well spread over the Netherlands.
After data-extraction, we screened the medication administration data for the occurrence of DDIs using a computerized algorithm based on the G-standard DDI knowledge base. For the first Delphi round, we selected DDIs that occurred at least a hundred times. For the second Delphi round, the remaining DDIs that have a severity level of D (severe consequences reported), E (life threatening consequences reported) or F (lethal consequences reported) were selected.

Expert panel
Experts were recruited from all fourteen Dutch ICUs participating in the SIMPLIFY study [19] evaluating the effectiveness of decision support on the number of DDIs in the ICU. All 14 ICUs were adult, closed format mixed medical surgical ICUs. Together, these ICUs represent 311 beds (mean: 22; SD: 13.2) and 23,452 yearly admissions (mean: 1675; SD: 1029). From each ICU, intensivists and hospital pharmacists were invited to participate in the expert panel. In addition, two hospital pharmacists and one clinical geriatrician were asked to participate because of their expertise in medication safety in ICU patients. In total 30 experts were invited.

Questionnaire
For both Delphi rounds, we used an online questionnaire (SurveyMonkey), presenting DDIs in subsections based on a similar mechanism of action. To prevent bias due to the serial position effect [20], we created two versions of both questionnaires by randomly changing the order of the DDI subsections. Half of the experts were randomized to receive one version; the other half received the other version. The questionnaire was tested on one expert before distribution to all experts. To support the experts during assessment, each DDI was linked to its online risk analysis for background information. An example of the questionnaire is available on request.
The questionnaire used in the first round consisted of three additional sections: respondent characteristics (twelve questions), respondent's knowledge and opinion about DDI alerts on the ICU (six questions), and feedback on the questionnaire (two questions). Experts were invited via e-mail containing a link to the questionnaire. Experts had one month to complete the questionnaire. When necessary, e-mail reminders were sent.

Live consensus meeting
During both live consensus meetings, DDIs with a low agreement score derived from the questionnaire were discussed. The purpose of the meetings was to determine reasons for disagreement and see whether the reasons represented genuine differences in opinion or differences in interpretation. In the last case, agreement could still be reached. A senior intensivist involved in the study (DD) chaired the meetings. At the meeting, expert panel members received a paper version of their questionnaire, to see how they had scored the DDIs. Each DDI with a low agreement score was discussed for approximately five minutes, and during the discussion a PowerPoint slide representing the distribution of scores for the specific DDI was shown. After discussion of each DDI, the experts individually scored the DDI again. When these scores would still result in a low agreement score, we accepted this result. After the consensus meeting, participants received a book token and had the possibility to charge their travelling expenses.

Definitions of clinical relevance and agreement
For each DDI the experts were asked to score the clinical relevance level of the DDIs for the ICU setting on a scale of 1 to 5 (Table 1).
A DDI is considered clinically relevant when the mode score for clinical relevance (the most often selected score) of the DDI is three or higher, and the DDI does not have low agreement. A DDI is considered not clinically relevant when the mode score is less than three, and the DDI does not have low agreement. Based on the D7S method from RAND [17], we defined agreement as low when more than 10% of the scores are a 1 or 2 and more than 10% of the scores are a 5. DDIs with low agreement were discussed during the live consensus meetings.

Outcomes
Our main outcome was the number and percentage of clinically relevant DDIs and not clinically relevant DDIs. We categorized the number and percentage of clinically relevant DDIs by type of increased risk related to the DDI. We also categorized by type of advised monitoring strategy. Both categorizations are based on Uijtendaal et al. [21] and information in the G-standard. Medication categories are based on the Anatomical Therapeutic Chemical (ATC) Classification System [22]. Differences between expert panel member characteristics in the first and second round were tested by a Chi squared or Fisher exact test where appropriate. All analyses were performed with the R statistical environment (3.4.1) [23].

Results
Screening the medication administration data of six Dutch ICUs for DDIs resulted in a total occurrence of 216 unique DDIs. Of these, 73 DDIs occurred at least one hundred times and were selected for the first Delphi round. Seventy-five DDIs occurred less than one hundred times but have a severity level of D, E or F were selected for the second Delphi round. The remaining 68 DDIs were excluded (Fig. 2).

Round 1: frequently occurring DDIs
For the first questionnaire, 30 experts were invited to participate and assess the 73 frequently occurring DDIs. Twenty-seven experts responded, resulting in a response rate of 27/30 (90%). Of these, 13 participated in the first live consensus meeting. See Table 2 for expert characteristics. No significant differences were found between characteristics of the experts participating in the first questionnaire and the experts participating in the first live meeting.
Calculating the agreement scores of 73 DDIs assessed in the first questionnaire, seventeen (23%) DDIs scored low on agreement. These seventeen DDIs were discussed during the first live meeting and rated again. Five DDIs remained with low agreement, while for twelve DDIs agreement was reached. For 56 DDIs, agreement was already reached based on the answers provided in the questionnaire. Overall, in the first round agreement was reached for a total of 68/73 (93%) DDIs. Of these 68 DDIs, 25 (37%) were considered as clinically relevant for the ICU, while 43 (63%) were considered as not clinically relevant. Fig. 2 shows a flow diagram of the results of the first and second round of our Delphi procedure.

Round 2: less frequent but severe DDIs
For the second questionnaire, the same 30 experts were invited to assess the remaining 75 DDIs. Twenty-six experts responded, resulting in a response rate of 26/30 (87%). Of these, nine participated in the second live consensus meeting. See Table 2 for expert characteristics. No significant differences were found between the characteristics of the experts participating in the second questionnaire and the experts participating in the second live meeting. Furthermore, no significant differences were found between the experts participating in the first and second Delphi round.
In the second questionnaire, 24 (32%) DDIs scored low on agreement. These 24 DDIs were discussed during the second live meeting and rated again. Four DDIs remained with a low agreement score, while for 20 DDIs agreement was reached. For 51 DDIs agreement was reached based on the answers provided in the questionnaire. Overall, in the second round, agreement was reached for a total of 71/75 (95%) DDIs. Of these 71 DDIs, 61 (86%) were considered as clinically relevant for the ICU, while 10 (14%) were considered as not clinically relevant (Fig. 2).
Summarizing results from both Delphi rounds, agreement was reached for a total of 139/148 (94%) DDIs. For a total of 9 DDIs (6%) low agreement remained. Of these 139 agreed upon DDIs, 86 (62%) were considered as clinically relevant for the ICU, while 53 (38%) were considered as not clinically relevant. Supplementary file 2 provides a full overview of all 148 DDIs and their mode scores.
In Table 3a, the proportion of clinically relevant DDIs per type of increased risk is shown. All DDIs increasing the risk of hematologic disturbances were considered clinically relevant for the ICU setting, as well as all DDIs with a risk of decreased efficacy of immunomodulators, antibiotics, antimycotics, antipsychotics or anti-epileptics. Most DDIs potentially causing cardiac arrhythmias or neurologic disturbances were clinically relevant. On the other hand, none of the DDIs potentially causing electrolyte disturbances, masking hypoglycemia, or decreasing the efficacy of antithrombotics or lipid-modifying agents were clinically relevant. In general, DDIs potentially influencing blood pressure or increasing the risk of bleeding were not clinically relevant.
Considering monitoring strategies (Table 3b), all DDIs for which therapeutic drug monitoring or liver function monitoring is advised were clinically relevant. In the category of risk-modifying strategies, DDIs for which a time interval between administrations is advised were mostly clinically relevant. At the same time, none of the DDIs for which monitoring of glucose, potassium or sodium levels is advised were clinically relevant. Furthermore, few of the DDIs for which blood pressure or blood clotting time monitoring is advised were clinically relevant.

Discussion
By conducting a modified Delphi procedure, for 148 DDIs the clinical relevance in the ICU setting was assessed by a national multidisciplinary expert panel. For 139 DDIs (94%) agreement on relevance was reached. Overall, 53 of these 139 DDIs (38%) were assessed as not clinically relevant. In the group of frequently occurring DDIs, 63% were assessed as Table 1 Answer options to score the clinical relevance level of DDIs for the ICU setting.
Answer options to score the clinical relevance of DDIs for the ICU setting 1 Not clinical relevant, since adverse effects of this DDI are negligible. 2 Clinically relevant, the adverse effects of this DDI will however be limited since routine monitoring to timely identify adverse effects is present. 3 Clinically relevant, the adverse effects of this DDI can however be limited by additional monitoring and/or changes in dosage/frequency/timing. 4 Clinically relevant, the adverse effects of this DDI on the patient can be substantial, however these effects are acceptable and treatable. 5 Clinically relevant, the adverse effect of this DDI on the patient should preferably be avoided.
not clinically relevant. In the group of less frequent but potentially severe DDIs, 14% of the DDIs was assessed as not clinically relevant.

Strengths and limitations
This study has several strengths. First, to our knowledge this is the only multicenter study that assessed the clinical relevance of DDIs for the ICU setting through a Delphi procedure with a multidisciplinary expert panel. By using the Delphi method, opinions of experts can be collected anonymously without a senior or powerful expert dominating the outcome. The Delphi method is often used successfully to obtain consensus from experts on medication related topics [16]. Also, discussion in the live consensus meeting allowed the experts to reflect and change opinions when necessary. Second, in both Delphi rounds the response rates of the questionnaires were high. Third, selection of DDIs was based on multicenter analysis of the DDI frequency on the ICU and we combined this data with severity levels based on extensive risk analyses.
This study also has limitations. First, the judgment of clinical relevance of DDIs by experts is inherently subjective. Therefore, the results of this study may not be agreed upon by all intensivists and hospital pharmacists. However, this is inherent to each research area without a gold standard. In a sensitivity analysis, we compared the assessment of intensivists to the assessment of hospital pharmacists and we found no differences. In addition, ICUs can have different treatment population and consequently prescribe different medication, resulting in possible differences in clinical relevance of specific DDIs. Accordingly, the results we found may not be an exact fit for each ICU. However, the experts who participated in our Delphi have considerable pharmacotherapy expertise and experience on the ICU and represent teaching and nonteaching ICUs from fourteen different ICUs. Additionally, various patient populations including cardio surgical, neurosurgical, transplant, trauma and oncological patients are treated in these fourteen ICUs. Therefore we are confident that the various types of ICU levels and appropriate expertise were sufficiently represented. In addition, we used a definition for agreement of the clinical relevance level of a DDI based on the established D7S definition of agreement from RAND [17].
Second, our results are based on DDIs included in the Dutch G-standard DDI knowledge base assessed by Dutch experts. Nevertheless, since frequently occurring DDIs in the ICU setting seem comparable between countries [24], our result could be generalizable to ICUs from other countries. Furthermore, in our study we did not.
Assess all DDIs included in the DDI knowledge base of the G-standard. However, we have identified characteristics of DDIs (e.g. presence of routine monitoring to recognize potential adverse effects) that could be used as a decision aid to assess whether or not a DDI might be turned off in the ICU. This knowledge can be used in the Netherlands but is also generalizable to ICUs in other countries.

Comparison to literature
Our results are in line with findings from similar studies in which a Delphi procedure was used to assess clinical relevance of DDIs [25][26][27]. In the only study we found in the ICU setting by Askari et al. [25], a single-center expert panel consisting of intensivists and pharmacists assessed the relevance of 53 severe DDIs that occurred in one ICU. Compared to Askari et al., our study assessed more DDIs, both severe and less severe, by a larger multicenter expert panel. Furthermore, our selection of DDIs was based on a multicenter dataset as compared to a single-center dataset used by Askari et al. Weingart et al. [26] assessed clinical relevance of DDIs in the primary care setting. They found that low and medium severe DDIs in general were not considered relevant while severe DDIs were. Another study assessed the clinical relevance of DDIs in the outpatient setting [27], they found that about half of the DDIs were considered clinically relevant. As there are important differences between settings, such as differences in monitoring intensity between the ICU and other settings, it is important to determine clinical relevance of DDIs for specific settings and using the obtained assessment to improve clinical value of CDSSs.

Interpretation of our results
Comparing clinically relevant and clinically irrelevant DDIs, DDIs for which nonstandard monitoring, such as therapeutic drug monitoring, is required to timely detect potential adverse effects are more often clinically relevant according to ICU experts. Furthermore, most DDIs that decrease the efficacy of medication therapy are considered clinically relevant. This may be explained by the fact that success of ICU treatment depends heavily on medication efficacy. Finally, infrequent DDIs are more often considered clinically relevant than frequent DDIs, probably due to the unfamiliarity with the medication. It is important to realize that frequently occurring DDIs produce most DDI alerts and therefore have a larger share in the development of alert fatigue in comparison to less frequent DDIs.

Implications and future research
Our findings can be used to tailor a DDI knowledge base used in CDSSs in the ICU setting. Since the majority of the frequently occurring DDIs are considered not clinically relevant, the number of DDI alerts produced by a CDSS may be reduced substantially by turning off alerts for these DDIs. By doing so, alert fatigue and high override rates may markedly decrease, help focus on the relevant alerts and improve patient safety.
Besides using CDSSs, during the live consensus meetings experts mentioned several other measures to reduce risks related to DDIs on the ICU. For example, some ICU experts said to refrain from using certain medication classes such as oral vitamin K antagonists and NSAIDs, thereby preventing interactions that occur with these medication classes.
Our results may encourage caregivers from other settings and developers of CDSSs, to establish a DDI knowledge base for a specific setting of patient group such as children or frail elderly. Future research is needed to examine whether tailoring DDI alerts based on clinical relevance for the ICU results in improved effectiveness of CDSS.

Conclusion
Using a modified Delphi procedure, we found that experts assessed about 40% of the DDIs occurring in the ICU as not clinically relevant for the ICU setting. This indicates that the clinical value of CDSSs for medication safety could be improved by focusing on the identified clinically relevant DDIs.

Ethical approval and consent to participate
The study protocol was reviewed by the Medical Ethics Committee of the Amsterdam Medical Center, the Netherlands. This committee provided a waiver from formal approval (W16_391 # 17.001) and informed consent since this study does not fall within the scope of the Dutch Medical Research (Human Subjects) Act.

Consent for publication
Not applicable.

Availability of data and material
The questionnaire data will be available upon request with the corresponding author.

Funding
This work was supported by ZonMw (dossier number: 80-83600-98-40140). The funder had no role in the design of the study or writing the manuscript.