Development of a Standardized Rating Tool for Drug Alerts to Reduce Information Overload^[*]

 

 Buy Article Permissions and Reprints

Summary

Background: A well-known problem in current clinical decision support systems (CDSS) is the high number of alerts, which are often medically incorrect or irrelevant. This may lead to the so-called alert fatigue, an over -riding of alerts, including those that are clinically relevant, and underuse of CDSS in general.

Objectives: The aim of our study was to develop and to apply a standardized tool that allows its users to evaluate the quality of system-generated drug alerts. The users’ ratings can subsequently be used to derive recommendations for developing a filter function to reduce irrelevant alerts.

Methods: We developed a rating tool for drug alerts and performed a web-based evaluation study that also included a user review of alerts. In this study the following categories were evaluated: “data linked correctly”, “medically correct”, “action required”, “medication change”, “critical alert”, “information gained” and “show again”. For this purpose, 20 anonymized clinical cases were randomly selected and displayed in our customized CDSS research prototype, which used the summary of product characteristics (SPC) for alert generation. All the alerts that were provided were evaluated by 13 physicians. The users’ ratings were used to derive a filtering algorithm to reduce overalerting.

Results: In total, our CDSS research prototype generated 399 alerts. In 98 % of all alerts, medication data were rated as linked correctly to drug information; in 93 %, the alerts were assessed as “medically correct”; 19.5 % of all alerts were rated as “show again”. The interrater-agreement was, on average, 68.4 %. After the application of our filtering algorithm, the rate of alerts that should be shown again decreased to 14.8 %.

Conclusions: The new standardized rating tool supports a standardized feedback of user-perceived clinical relevance of CDSS alerts. Overall, the results indicated that physicians may consider the majority of alerts formally correct but clinically irrelevant and override them. Filtering may help to reduce overalerting and increase the specificity of a CDSS.

^* Supplementary material published on our website http://dx.doi.org/10.3414/ME16-01-0003

^** These authors contributed equally to this work

• References

• 1 Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998; 279 (Suppl. 15) 1200-1205.
  CrossrefPubMedGoogle Scholar
• 2 Wester K, Jönsson AK, Spigset O, Druid H, Hägg S. Incidence of fatal adverse drug reactions: a population based study. Br J Clin Pharmacol 2008; 65 (Suppl. 04) 573-579.
  CrossrefPubMedGoogle Scholar
• 3 WHO.. Medicines: safety of medicines – adverse drug reactions. Fact sheet N°293 2008. Available from: http://www.wiredhealthresources.net/resources/NA/WHO-FS_MedicinesSafetyAdverseReactions.pdf
  PubMedGoogle Scholar
• 4 Schnurrer JU, Frölich JC. [Incidence and prevention of lethal undesirable drug effects]. Internist (Berl) 2003; 44 (Suppl. 07) 889-895. German.
  CrossrefPubMedGoogle Scholar
• 5 Schrappe M, Lessing C, Albers B. Agenda Patientensicherheit 2007. 2008 Available from: http://www.aps-ev.de/fileadmin/fuerRedakteur/PDFs/Agenda_Patientensicherheit/Agenda_2007_mit_Titelblatt.pdf
  PubMedGoogle Scholar
• 6 Sedlmayr B, Patapovas A, Kirchner M, Sonst A, Müller F, Pfistermeister B. et al. Comparative evaluation of different medication safety measures for the emergency department: physicians’ usage and acceptance of training, poster, checklist and computerized decision support. BMC Med Inform Decis Mak 2013; 13: 79.
  CrossrefPubMedGoogle Scholar
• 7 Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (Suppl. 10) 1223-38.
  CrossrefPubMedGoogle Scholar
• 8 Ojeleye O, Avery A, Gupta V, Boyd M. The evidence for the effectiveness of safety alerts in electronic patient medication record systems at the point of pharmacy order entry: a systematic review. BMC Med Inform Decis Mak 2013; 13: 69.
  CrossrefPubMedGoogle Scholar
• 9 Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014; 23 (Suppl. 09) 773-780.
  CrossrefPubMedGoogle Scholar
• 10 Pauly A, Wolf C, Busse M, Strauß A-C, Krebs S, Dörje F. et al. Evaluation of eight drug interaction databases commonly used in the German health-care system. Eur J Hosp Pharm 2014; 22 (Suppl. 03) 165-170.
  PubMedGoogle Scholar
• 11 Ammenwerth E, Hackl WO, Massari P, Darmoni S. Validation of completeness, correctness, relevance and understandability of the PSIP CDSS for medication safety. Stud Health Technol Inform 2011; 166: 254-259.
  PubMedGoogle Scholar
• 12 Spina JR, Glassman PA, Belperio P, Cader R, Asch S. Primary Care Investigative Group of the V.. A. Los Angeles Healthcare System. Clinical relevance of automated drug alerts from the perspective of medical providers. Am J Med Qual 2005; 20 (Suppl. 01) 7-14.
  CrossrefPubMedGoogle Scholar
• 13 Fritz D, Ceschi A, Curkovic I, Huber M, Egbring M, Kullak-Ublick GA. et al. Comparative evaluation of three clinical decision support systems: prospective screening for medication errors in 100 medical inpatients. Eur J Clin Pharmacol 2012; 68 (Suppl. 08) 1209-1219.
  CrossrefPubMedGoogle Scholar
• 14 Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G. et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 (Suppl. 01) 29-40.
  CrossrefPubMedGoogle Scholar
• 15 Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B. et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006; 13 (Suppl. 01) 5-11.
  CrossrefPubMedGoogle Scholar
• 16 van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008; 15 (Suppl. 04) 439-448.
  CrossrefPubMedGoogle Scholar
• 17 Pfistermeister B, Sass A, Criegee-Rieck M, Bürkle T, Fromm MF, Maas R. Inconsistencies and misleading information in officially approved prescribing information from three major drug markets. Clin Pharmacol Ther 2014; 96 (Suppl. 05) 616-624.
  CrossrefPubMedGoogle Scholar
• 18 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (Suppl. 02) 138-147.
  CrossrefPubMedGoogle Scholar
• 19 Jung M, Riedmann D, Hackl WO, Hoerbst A, Jaspers MW, Ferret L. et al. Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems. BMC Med Inform Decis Mak 2012; 12: 111.
  CrossrefPubMedGoogle Scholar
• 20 van der Sijs H. Drug safety alerting in computerized physician order entry – unraveling and counteracting alert fatigue. Rotterdam: Erasmus University; 2009. Available from: http://repub.eur.nl/pub/16936
  Google Scholar
• 21 Hsieh TC, Kuperman GJ, Jaggi T, Hojnowski-Diaz P, Fiskio J, Williams DH. et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004; 11 (Suppl. 06) 482-491.
  CrossrefPubMedGoogle Scholar
• 22 Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-Zéphir MC. Usability Flaws in Medication Alerting Systems: Impact on Usage and Work System. Yearb Med Inform 2015; 10 (Suppl. 01) 55-67.
  Article in Thieme ConnectPubMedGoogle Scholar
• 23 Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform 2013; 82 (Suppl. 06) 492-503.
  CrossrefPubMedGoogle Scholar
• 24 Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL. et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (Suppl. 06) 1243-1250.
  CrossrefPubMedGoogle Scholar
• 25 Ammenwerth E, Aly AF, Bürkle T, Christ P, Dor-mann H, Friesdorf W. et al. Memorandum on the use of information technology to improve medication safety. Methods Inf Med 2014; 53 (Suppl. 05) 336-343.
  Article in Thieme ConnectPubMedGoogle Scholar
• 26 Riedmann D, Jung M, Hackl WO, Stühlinger W, van der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak 2011; 11: 35.
  CrossrefPubMedGoogle Scholar
• 27 Patapovas A, Pfistermeister B, Tarkhov A, Terfloth L, Maas R, Fromm MF. et al. The effect object paradigm – a means to support medication safety with clinical decision support. Stud Health Technol Inform 2014; 205: 1065-1069.
  PubMedGoogle Scholar
• 28 Hahn M, Braus DF. [Psychiatric polypharmacy: hazard through drug-drug-interaction and possibilities for prevention]. Versicherungsmedizin 2012; 64 (Suppl. 03) 127-131. German.
  PubMedGoogle Scholar
• 29 DiPoto JP, Buckley MS, Kane-Gill SL. Evaluation of an automated surveillance system using trigger alerts to prevent adverse drug events in the intensive care unit and general ward. Drug Saf 2015; 38 (Suppl. 03) 311-317.
  CrossrefPubMedGoogle Scholar
• 30 Medizinische Medien Informations GmbH. [cited 2015 July 23]. Available from: http://www.mmi.de/
  PubMedGoogle Scholar
• 31 Qualtrics LLC. [cited 2015 July 24]. Available from: http://www.qualtrics.com/de
  PubMedGoogle Scholar
• 32 Miyo K, Kawazoe Y, Yamaguchi I, Tatsukawa A, Ohe K. Evaluation of a context-based prescription alert system: a clinical perspective. Stud Health Technol Inform 2013; 192: 1030.
  PubMedGoogle Scholar
• 33 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (Suppl. 21) 2625-2631.
  CrossrefPubMedGoogle Scholar
• 34 Taylor LK, Tamblyn R. Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform 2004; 107 Pt 2 1101-1105.
  PubMedGoogle Scholar
• 35 Andersson ML, Böttiger Y, Bastholm-Rahmner P, Ovesjö ML, Vég A, Eiermann B. Evaluation of usage patterns and user perception of the drug-drug interaction database SFINX. Int J Med Inform 2015; 84 (Suppl. 05) 327-333.
  CrossrefPubMedGoogle Scholar
• 36 Tamblyn R, Eguale T, Buckeridge DL, Huang A, Hanley J, Reidel K. et al. The effectiveness of a new generation of computerized drug alerts in reducing the risk of injury from drug side effects: a cluster randomized trial. J Am Med Inform Assoc 2012; 19 (Suppl. 04) 635-643.
  CrossrefPubMedGoogle Scholar
• 37 Koutkias VG, McNair P, Kilintzis V, Skovhus Andersen K, Niès J, Sarfati JC. et al. From adverse drug event detection to prevention. A novel clinical decision support framework for medication safety. Methods Inf Med 2014; 53 (Suppl. 06) 482-492.
  Article in Thieme ConnectPubMedGoogle Scholar
• 38 Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM. et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013; 13: 111.
  CrossrefPubMedGoogle Scholar
• 39 Cruz MR, Martins C, Dias J, Pinto JS. A validation of an intelligent decision-making support system for the nutrition diagnosis of bariatric surgery patients. JMIR Med Inform 2014; 2 (Suppl. 02) e8.
  CrossrefPubMedGoogle Scholar
• 40 Seidling HM, Phansalkar S, Seger DL, Paterno MD, Shaykevich S, Haefeli WE. et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011; 18 (Suppl. 04) 479-484.
  CrossrefPubMedGoogle Scholar

• Supplementary Material