https://orcid.org/0000-0001-6628-8949
Figures
Abstract
The relationship between the Naranjo scaling system and pediatric adverse drug reactions (ADR) is poorly understood. We performed a retrospective review of 1,676 pediatric ADRs documented at our hospital from 2014–2018. We evaluated patient demographics, implicated medication, ADR severity, calculated Naranjo score, associated symptoms, and location within the hospital in which the ADR was documented. ADR severity was poorly correlated with Naranjo interpretation. Out of the 10 Naranjo scale questions, 4 had a response of “unknown” greater than 85% of the time. Cardiovascular and oncological/immunologic agents were more likely to have a probable or definite Naranjo interpretation compared to antimicrobials. Further strategies are needed to enhance the causality assessment of pediatric ADRs in clinical care.
Citation: Murali M, Suppes SL, Feldman K, Goldman JL (2021) Utilization of the Naranjo scale to evaluate adverse drug reactions at a free-standing children’s hospital. PLoS ONE 16(1): e0245368. https://doi.org/10.1371/journal.pone.0245368
Editor: Paula Schaiquevich, Hospital JP Garrahan, ARGENTINA
Received: October 15, 2020; Accepted: December 29, 2020; Published: January 13, 2021
Copyright: © 2021 Murali et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data may be found in supporting information.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
According to the World Health Organization, an adverse drug reaction (ADR) is “a response to a medicine which is noxious and unintended, and which occurs at doses normally used in man” [1,2]. Together, the frequent use of off-label medications along with developmental changes that occur in children from infancy to adolescence result in an elevated risk for ADRs [3,4]. Despite the known ADR risk, pediatric ADRs often are underrecognized and underreported [4,5]. Moreover, even when ADRs are documented in the electronic medical record (EMR), the information often lacks the details required for medical providers to determine how to safely prescribe medications related to the implicated drug. Effectively doing so requires the determination of key factors such as ADR phenotype and severity, as many ADRs are tolerable side effects with low risk of recurrence while others can be severe and life threatening [6]. Yet, without available clinical tests to confirm or negate if a drug caused a reaction, the medical provider is challenged on how to interpret a potential ADR.
As a result, causality assessment tools (CATs) have been developed to provide an objective, standardized approach to aid in the determination of whether or not an administered drug caused an ADR [7,8]. CATs are typically comprised of questions focused on several themes including the drug exposure, adverse reaction presentation, and patient characteristics. Unfortunately, the results of CATs have been shown to be highly variable, depending on the tool used and the provider using the tool [9–11]. Moreover, as the majority of causality tools have been developed for adult populations, the utility in hospitalized children remains poorly understood [12].
Among such toolsets, the Naranjo scaling system is one of the most widely utilized CATs [8]. In the case of an adverse reaction, a Naranjo assessment can be completed. This assessment is comprised of 10 questions concerning the implicated medication and reaction phenotype. Each answered question has an individual score, which is then totaled to provide a final score that is associated with one of four categories of likelihood that the drug was associated with the reaction (unlikely, possibly, probably, or definitely). To date, it is unclear if and how Naranjo results are associated with pediatric ADR clinical factors.
Accordingly, the primary objective of this work was to understand how the Naranjo is used in the pediatric setting. To do so, we characterized Naranjo probability scores with respect to pediatric ADR phenotype, severity, and implicated drug class. We then took this one step further, in a secondary objective to identify specific clinical factors associated with probable or definite Naranjo scores.
Methods
Study setting
We performed a retrospective case series study of hospitalized patients with a documented ADR at Children’s Mercy Kansas City, a 354-bed free-standing academic pediatric hospital with an average of 15,000 admissions annually. In October 2010, an active hospital-wide pharmacovigilance program was started to identify and evaluate children hospitalized with an active or historical ADR. Details of this program have been described in previous works [6,13,14].
ADRs are classified and documented in the electronic medical record (EMR) by phenotype (hypersensitivity, side effect, other) and severity (mild, moderate, severe) (Table 1) [6]. Severity classification are based on the Hartwig’s Severity Assessment Tool [15] with the modification that we consider a reaction moderate if the implicated drug was discontinued [6]. Clinical pharmacists who have undergone pharmacovigilance training, also collect more detailed information including patient demographics, ADR phenotype, ADR treatment, completion of the Naranjo scale, and whether the reaction occurred within 30 days Naranjo assessment. All ADR information is documented in a form within the EMR. The Naranjo scale is a series of 10 questions. Each question can be answered as “Yes”, “No”, or “Unknown”, with point values assigned to each answer [8]. The final score interpretations are stratified into four categories with a score of ≥ 9 considered “definite”, 5 to 8 “probable”, 1 to 4 “possible”, and those ≤0 “doubtful” likelihood of the drug causing the ADR.
Study cohort and data collection
The data utilized for this study were obtained from the pharmacovigilance database between November 15, 2014 and December 31, 2018. From this complete set of documented and pharmacist reviewed ADRs, a series of exclusion criteria were applied to ensure a robust data set for analysis.
To begin, all children ≥21 years of age were removed. Next, all children with multiple ADRs or those with more than a single implicated drug for any reaction were excluded. Such a scenario can occur in two ways. First, there may be multiple drugs implicated for a single ADR, for which a Naranjo questionnaire would have been completed for each drug. Second, a child may have two or more ADRs documented in their record. In both cases, these data may be biased by extrinsic factors (such as time between reactions), and as such all children with multiple ADR/Naranjo records were excluded from this study. Finally, we ensured a complete record of ADR assessment by the pharmacovigilance program was available, requiring the calculated Naranjo score (and response to each Naranjo question), ADR phenotype, ADR mediation, as well as a complete set of patient demographics. The implicated drugs were further classified into general drug classes as defined by Shehab et al [16]. This study was approved by the local institutional review board.
Statistical analysis
Descriptive statistics were computed for cohort demographics (age at time of ADR documentation, sex, race/ethnicity), as well as the clinical characteristics of the documented ADR (reaction phenotype, clinical symptoms, and overall Naranjo score). Median and interquartile range (IQR) are reported for all continuous features, and frequency distributions are used for all nominal categories.
First, to evaluate the association between Naranjo scoring and ADR attributes, a series of bivariate statistical comparisons were performed with implicated drug class, phenotype, and severity. Given the low prevalence in the highest and lowest Naranjo categorization, non-parametric Fisher’s exact tests were performed to assess the distribution of the 4 Naranjo classes across each of the study variables. Due to the large number of drug class categories, the Fisher’s test was performed with a Monte Carlo simulation, using 10,000 sampling iterations and seeded for reproducibility [17]. Directionality and magnitude of the relationship between ADR severity and Naranjo categories were assessed by the Kendall rank correlation coefficient (tau). In all cases, exact p-values are reported. Results that reach significance at or above 95% confidence are highlighted in the results and discussion sections to follow.
Next with respect to the secondary objective, the identification of clinical factors associated with the Naranjo categorizations (probable / definite vs. possible / doubtful scores), multivariate logistic regression was performed adjusting for study variables found to be significant in the bivariate analyses: drug class, severity, and phenotype. In addition, regressions were adjusted for patient age, sex, and race/ethnicity. Reference categories were selected to represent the lowest severity level, largest drug class and race/ethnicity.
Finally, in an effort to better contextualize the data presented throughout this manuscript, we concluded with an analysis focused on the Naranjo questionnaire itself, where the frequency of “yes”, “no”, and “unknown” responses were calculated across each Naranjo score question. All analyses used python 3.7.6, pandas 1.0.1, numpy 1.18, scipy 1.4.1, and statsmodels 0.11.
Results
In total, 1746 children with a single ADR were identified by the pharmacovigilance program during the study period. Of these, 70 ADRs failed to meet the remaining inclusion criteria: starting with patients older than 21 years of age (n = 6), those with missing Naranjo scores (n = 31), missing severity classification (n = 14), or failure to document if the reaction occurred within past 30 days (n = 7), classifications other than hypersensitivity reactions or side effects (n = 2), missing and unknown race (n = 10), resulting in a total of 1676 ADRs included in analysis.
Of the 1,676 ADRs, the median age of a patient with a documented ADR was 10.5 years (IQR, 5–15), 822 (49%) of patients were male, and the majority of patients were Caucasian 1,286 (77%) (Table 2). Hypersensitivity reactions were documented 66% of the time as compared to side effects (34%). The most common ADR clinical symptoms were cutaneous 1197 (71%), neurological 228 (14%), and gastrointestinal 256 (15%). The Naranjo score classified 25 (1.5%) of cases as definite, 835 (50%) probable, 813 (49%) possible, and 3 (0.2%) doubtful (Table 2). The adverse reaction severity as classified by a pharmacist totaled 25 (1.5%) mild, 1489 (89%) moderate, and 162 (10%) severe reactions (Table 3).
ADR phenotype was found to be significantly different across the Naranjo interpretation categories (P < .001). A Fisher’s test showed no statistical difference between the 3 severity levels and interpretation (P = 0.459), and upon further evaluation the correlation between increasing severity and Naranjo score was extremely low (Tau: 0.0329, P: 0.1726).
With respect to drug class, the most common implicated drugs were systemic antimicrobial agents (1072; 64%) (Table 4). Monte Carlo simulation demonstrated the distribution of Naranjo scores was not uniform across drug classifications (P < .001). Cardiovascular agents had the highest percentage of a definite Naranjo score interpretation (6.3%), followed by hormone-modifying agents (5.9%). When controlling for age, gender, race, time of documentation, as well as adverse reaction severity and phenotype, cardiovascular and oncological/immunologic agents were more likely to have a probable or definite Naranjo interpretation compared to the reference category of antimicrobials (Table 5). Documented ADRs that were evaluated within 30 days of occurring were also more likely to be probable or definite.
Evaluation of each Naranjo question revealed an “unknown” response was frequently selected (Table 6). Specifically, questions regarding drug re-administration (question 4), administration of a placebo (question 6), detection of drug at toxic concentrations (question 7), and reaction in response to changes in dosing (question 8) had a response rate of “unknown” greater than 85% of the time.
Discussion
Determining whether there is a causal relationship between an undesired reaction and a drug is an important component of pharmacovigilance [12]. As diagnostic tests are not readily available to implicate a drug, CATs are used to assess the probability of a drug resulting in an ADR [18]. In this study, we found that 1) the Naranjo scoring system is not associated with ADR severity, 2) specific medication classes implicated in ADRs had increased likelihood of receiving higher Naranjo scores and 3) several questions in the Naranjo scale are answered unknown most of the time.
A probable or definite Naranjo score was not associated with a specific ADR severity or reaction phenotype. Our findings align with previous studies demonstrating that CAT results are often categorized as “possible,” which is difficult to interpret clinically [19,20]. The classification of “possible” suggests there may be another equally likely explanation for the event and/or there is uncertainty or lack of information [21]. The purpose of this work was to explore the relationship between ADR severity and causality because both are relevant clinically. When clinicians are making prescribing decisions for those children with an ADR, the clinician is typically assessing both the likelihood a medication caused the ADR and the severity of the reaction. Clinically, the utility of the Naranjo is limited, as a life-threatening reaction associated with a drug will likely be presumed too high risk to consider re-introduction regardless of probability score classification. Similarly, an ADR of mild severity classified as “definite” would not preclude use in the future. These findings further support limitations noted of utility of the Naranjo scoring system especially as related to clinical decision making [22]. Efforts to incorporate additional data and machine learning into causality assessment are underway, though not readily available for clinical pharmacovigilance.
We found that cardiovascular agents and oncological/immunologic agents had a higher likelihood of having a probable or definite Naranjo score relative to antimicrobials. Although the reason for this is unclear, our findings do suggest that different drug classes may require specific evaluations to better understand assessment. Antimicrobial agents are commonly implicated in pediatric ADRs though a reaction such as a rash can easily be confused with a viral exanthem making causality assessment challenging [23]. Studies have demonstrated antimicrobial re-challenge often reveals the drug is well-tolerated allowing for the removal of the ADR label [24,25]. Individualized causality assessment specific to a drug class may provide clinicians more helpful information for future prescribing.
The ability for a clinical pharmacist to answer the individual Naranjo questions on reported pediatric ADRs was challenging as demonstrated by the high percentage (>85%) of answers to 4 of the questions in the Naranjo questionnaire being “unknown”. The questions routinely answered “unknown” are likely the most important to determine causality. Drug challenges that would most likely confirm causality such as ADR recurrence with re-administration of the drug or with dose escalation rarely occur. In contrast, a question regarding whether or not the adverse event appeared after the drug was given was answered yes almost universally. Both types of questions are of limited use in deciphering ADR causality. These findings suggest that regardless of ADR severity, the data needed to help determine if the reaction was caused by the drug are challenging to gather by the medical provider. Previous work utilizing a modified Naranjo assessment that excludes questions 6 through 10 has been shown to effectively categorize ADRs with a high likelihood [26]. However, such studies have yet to be conducted within the pediatric population. Information specific to children could include the number of years that have elapsed since ADR occurrence as some reactions such as penicillin allergies diminish over time [27]. Also, concomitant infectious testing results would be of interest as children have viral infections that can mimic or trigger an ADR [28]. Several other CATs have been developed to enhance causality assessment, though to date, not a single tool is universally accepted or demonstrates consistent results [7,9,11].
Limitations of this study include that it is a retrospective chart review of patients at a single institution; future multi-institutional studies can help providers better understand the variance of ADR documentation practices across institutions as well as gain a broader understanding of the clinical application of causality assessment tools in pediatric populations. Based on the need to collect ADR specific data, our findings may only be applicable to institutions with pharmacovigilance programs. Additionally, results may differ at other institutions based on local pharmacovigilance practices. Finally, causality assessments performed by clinical pharmacists and Naranjo scores can vary depending on evaluator.
Conclusion
Establishing the association between drug exposure and an ADR is challenging. The ability to determine whether a drug did cause a reaction is important when making future medication prescribing decisions. The utility of the Naranjo scale in pediatric ADR causality is limited as the scores do not differentiate ADR severity and the responses to several of the Naranjo scale questions are “unknown” when completed by clinical pharmacists. Active pharmacovigilance programs are vital to accurately identifying and assessing ADRs in the clinical setting within pediatrics. The development of a pediatric specific ADR assessment tool that can account for drug class, reaction severity and additional clinical tests may enhance ADR causality assessment and clinical care.
References
- 1.
World Health Organization. Medicines: safety of medicines–adverse drug reactions. Fact sheet N 293. Updated October 2008. Available at http://whqlibdoc.who.int/hq/2002/WHO_EDM_QSM_2002.2.pdf. Last accessed Dec 21, 2020.
- 2. International drug monitoring: the role of national centres. Report of a WHO meeting. World Health Organ Tech Rep Ser. 1972;498:1–48. pmid:4625548
- 3. Rieder M. Adverse Drug Reactions in Children: Pediatric Pharmacy and Drug Safety. The journal of pediatric pharmacology and therapeutics: JPPT: the official journal of PPAG. 2019;24(1):4–9.
- 4. Smyth RM, Gargon E, Kirkham J, Cresswell L, Golder S, Smyth R, et al. Adverse drug reactions in children—a systematic review. PLoS One. 2012;7(3):e24061. pmid:22403604
- 5. Cammack B, Oschman A, Lewis T. Improving Recognition and Reporting of Adverse Drug Reactions in the NICU: A Quality Improvement Project. Pediatric quality & safety. 2019;4(5):e203. pmid:31745506
- 6. Yu D, Sheets J, Suppes S, Goldman J. Characterization of Severe Adverse Drug Reactions at a Free-Standing Children's Hospital. J Clin Pharmacol. 2019;59(12):1569–72. pmid:31309572
- 7. Arimone Y, Begaud B, Miremont-Salame G, Fourrier-Reglat A, Moore N, Molimard M, et al. Agreement of expert judgment in causality assessment of adverse drug reactions. Eur J Clin Pharmacol. 2005;61(3):169–73. pmid:15827761
- 8. Naranjo CA, Busto U, Sellers EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981;30(2):239–45. pmid:7249508
- 9. Goldman JL, Chung WH, Lee BR, Chen CB, Lu CW, Hoetzenecker W, et al. Adverse drug reaction causality assessment tools for drug-induced Stevens-Johnson syndrome and toxic epidermal necrolysis: room for improvement. Eur J Clin Pharmacol. 2019;75(8):1135–41. pmid:30918988
- 10. Theophile H, Arimone Y, Miremont-Salame G, Moore N, Fourrier-Reglat A, Haramburu F, et al. Comparison of three methods (consensual expert judgement, algorithmic and probabilistic approaches) of causality assessment of adverse drug reactions: an assessment using reports made to a French pharmacovigilance centre. Drug safety: an international journal of medical toxicology and drug experience. 2010;33(11):1045–54. pmid:20925441
- 11. Varallo FR, Planeta CS, Herdeiro MT, Mastroianni PC. Imputation of adverse drug reactions: Causality assessment in hospitals. PLoS One. 2017;12(2):e0171470. pmid:28166274
- 12. Vazquez-Alvarez AO, Brennan-Bourdon LM, Rincon-Sanchez AR, Islas-Carbajal MC, Huerta-Olvera SG. Improved drug safety through intensive pharmacovigilance in hospitalized pediatric patients. BMC pharmacology & toxicology. 2017;18(1):79. pmid:29216902
- 13. Goldman JL, Sullins A, Sandritter T, Leeder JS, Lowry J. Pediatric Pharmacovigilance: Enhancing Adverse Drug Reaction Reporting in a Tertiary Care Children's Hospital. Ther Innov Regul Sci. 2013;47(5):566–71. pmid:30235581
- 14. Tillman EM, Suppes SL, Feldman K, Goldman JL. Enhancing Pediatric Adverse Drug Reaction Documentation in the Electronic Medical Record. J Clin Pharmacol. 2020. pmid:32776356
- 15. Hartwig SC, Siegel J, Schneider PJ. Preventability and severity assessment in reporting adverse drug reactions. American journal of hospital pharmacy. 1992;49(9):2229–32. pmid:1524068
- 16. Shehab N, Lovegrove MC, Geller AI, Rose KO, Weidle NJ, Budnitz DS. US Emergency Department Visits for Outpatient Adverse Drug Events, 2013–2014. JAMA. 2016;316(20):2115–25. pmid:27893129
- 17.
Noutahi, Emmanuel. (2018, February 28). Fisher's exact test for MxN contingency table (Version 1.4.3). Zenodo. http://doi.org/10.5281/zenodo.2587757. Last accessed June 30, 2020.
- 18.
LiverTox: Clinical and Research Information on Drug-Induced Liver Injury [Internet]. Bethesda (MD): National Institute of Diabetes and Digestive and Kidney Diseases; 2012-. Adverse Drug Reaction Probability Scale (Naranjo) in Drug Induced Liver Injury. [Updated 2019 May 4]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK548069/. Last accessed July 14, 2020.
- 19. Gallagher RM, Kirkham JJ, Mason JR, Bird KA, Williamson PR, Nunn AJ, et al. Development and inter-rater reliability of the Liverpool adverse drug reaction causality assessment tool. PLoS One. 2011;6(12):e28096. pmid:22194808
- 20. Sassolas B, Haddad C, Mockenhaupt M, Dunant A, Liss Y, Bork K, et al. ALDEN, an algorithm for assessment of drug causality in Stevens-Johnson Syndrome and toxic epidermal necrolysis: comparison with case-control analysis. Clin Pharmacol Ther. 2010;88(1):60–8. pmid:20375998
- 21.
The Uppsala Monitoring Centre. The use of the WHO-UMC system for standardised case causality assessment. https://www.who.int/medicines/areas/quality_safety/safety_efficacy/WHOcausality_assessment.pdf. Last accessed Sept 28, 2020.
- 22. Avner M, Finkelstein Y, Hackam D, Koren G. Establishing causality in pediatric adverse drug reactions: use of the Naranjo probability scale. Paediatric drugs. 2007;9(4):267–70. pmid:17705565
- 23. Norton AE, Konvinse K, Phillips EJ, Broyles AD. Antibiotic Allergy in Pediatrics. Pediatrics. 2018;141(5).
- 24. Blumenthal KG, Peter JG, Trubiano JA, Phillips EJ. Antibiotic allergy. Lancet. 2019;393(10167):183–98. pmid:30558872
- 25. Vyles D, Adams J, Chiu A, Simpson P, Nimmer M, Brousseau DC. Allergy Testing in Children With Low-Risk Penicillin Allergy Symptoms. Pediatrics. 2017;140(2). pmid:28674112
- 26. Murayama H, Sakuma M, Takahashi Y, Morimoto T. Improving the assessment of adverse drug reactions using the Naranjo Algorithm in daily practice: The Japan Adverse Drug Events Study. Pharmacology research & perspectives. 2018;6(1). pmid:29417762
- 27. Shenoy ES, Macy E, Rowe T, Blumenthal KG. Evaluation and Management of Penicillin Allergy: A Review. JAMA. 2019;321(2):188–99. pmid:30644987
- 28. Dibek Misirlioglu E, Guvenir H, Bahceci S, Haktanir Abul M, Can D, Usta Guc BE, et al. Severe Cutaneous Adverse Drug Reactions in Pediatric Patients: A Multicenter Study. The journal of allergy and clinical immunology In practice. 2017;5(3):757–63. pmid:28351788