Explainable Artificial Intelligence for Patient Safety: A Review of Application in Pharmacovigilance

Explainable AI (XAI) is a methodology that complements the black box of artificial intelligence, and its necessity has recently been highlighted in various fields. The purpose of this research is to identify studies in the field of pharmacovigilance using XAI. Though there have been many previous attempts to select papers, with a total of 781 papers being confirmed, only 25 of them manually met the selection criteria. This study presents an intuitive review of the potential of XAI technologies in the field of pharmacovigilance. In the included studies, clinical data, registry data, and knowledge data were used to investigate drug treatment, side effects, and interaction studies based on tree models, neural network models, and graph models. Finally, key challenges for several research issues for the use of XAI in pharmacovigilance were identified. Although artificial intelligence (AI) is actively used in drug surveillance and patient safety, gathering adverse drug reaction information, extracting drug-drug interactions, and predicting effects, XAI is not normally utilized. Therefore, the potential challenges involved in its use alongside future prospects should be continuously discussed.


I. INTRODUCTION
The World Health Organization defines pharmacovigilance (PV) as the science and activities related to the detection, assessment, understanding, and prevention of adverse effects or other drug-related problems [1].
Recent artificial intelligence-based technologies can be an efficient complement to traditional PV methods, which can be costly and time-consuming and can result in adverse drug reactions (ADRs) that go unreported to healthcare professionals.
Artificial intelligence (AI) can improve PV, but its use in PV is still in the early stages of research. Various machine learning (ML) techniques, together with natural language The associate editor coordinating the review of this manuscript and approving it for publication was Jerry Chun-Wei Lin .
processing and data mining, can be applied to electronic health records, claims databases and social media data to improve the characterization of known drug side effects and reactions, and to detect new signals [2], [3].
AI-based technologies have been criticized for their inexplicable algorithms, despite their high predictive power. In critical decision areas such as healthcare, the reasoning behind a decision is as important as the decision itself, which is why there is growing interest in and research and development around Explainable Artificial Intelligence (XAI).
XAI was developed to improve the transparency of AI systems and generate explanations for them, and seeks to increase trust and understanding by assessing the strengths and limitations of existing models [4], [5], [6]. Approaches that extract information from a model's decision-making process, such as post-hoc explanations, can provide useful information for practitioners and users interested in caseby-case explanations rather than the internal workings of a model [7].
XAI increases the explainability and transparency of AI algorithms by making it possible to interpret the variables that influence decisions, complex internal features, and learned decision paths within a decision process [8], [9]. I.R. Ward et al. successfully quantified the importance of features using an XAI algorithm, further demonstrating the potential contribution of XAI to PV monitoring [10].
The importance of PV in medicine is relevant to all species affected by medical interventions, and ensuring medical safety requires attention and research into approaches such as drug safety reporting and the exchange of reliable and timely information on PV activities [11]. The global pharmacovigilance and drug safety software market size was valued at USD 6.9 billion in 2021 and is estimated to expand at a compound annual growth rate (CAGR) of 10.5% between 2022 and 2030 (Source: www.grandviewresearch.com).
The aim of this study was to review the literature on the use of XAI in PV by identifying publications related to ML/AI and drugs and the rationale for the reported findings. From the perspective of AI and XAI usage, these studies were analyzed, and the findings were summarized, in which the use of XAI in the field of PV is referred to as ''PV XAI''. The main contributions are highlighted and discussed below: -This study is clearly an early attempt to review XAI research in PV. Unlike other fields, we found that XAI research in PV is at an early stage of development, limited to a few articles and some methodologies. -Nevertheless, we have identified the positive potential of PV XAI for drug therapy, ADRs, polypharmacy and drug repurposing. -While safety issues in real-world healthcare settings may limit the growth of the field, we expect PV XAI research to expand as it has in other areas, and we encourage collaboration and ongoing research discussions with experts in the field.

II. LITERATURE SURVEY
In this study, the trend of XAI in the field of PV was examined. However, the trend was also explored broadly to more diverse aspects, including interpretable artificial intelligence. Although there is a clear difference between Explainable AI (knowledge about what different nodes represent and their importance to model performance) and Interpretable AI (ability to determine cause and effect in a machine learning model), based on the same aim, they were comprehensively reviewed.
There has been a surge in XAI studies in drug-related applications since 2019, with relatively few studies from 2013 to 2018 (Fig. 1). The limited number of publications indicates a demand for more research on XAI in PV applications. The selection of appropriate search terms for the exploration of XAI-related research in PV was not easy; we started manually with broad keywords. The following five searches were performed: pharmacovigilance XAI (47), pharmacovigilance ''explainable artificial intelligence'' (76), pharmacovigilance explainable AI (230), pharmacovigilance explainable ML (181), and pharmacovigilance explainable machine learning (213). These search terms were used in a Google Scholar search on 22 June 2022, and the numbers in parentheses are the number of articles returned from each search. Retrieved articles were first screened for titles and abstracts to exclude duplicates, then articles were added through a first full-text review for relevance and a second full-text review based on a selective methodology, resulting in a final selection of 25 unique publications (Fig. 2, Supplementary Table 1).

III. RESULTS
This highlight focuses on PV and reviews of recently developed machine-learning models to predict the explanatory potential and effectiveness of XAI (Fig. 3).
PV involves the collection and analysis of data from patient records or other sources to identify causal relationships between medicines and adverse drug reactions. While the potential for monitoring and preventing adverse drug reactions through clinical data is great, exploiting it can be time-consuming because clinical data is highly heterogeneous in structure and domain, and is often managed in multiple files. In addition, specific expertise is required to make sense of clinical information and often requires an understanding of appropriate related prescriptions, which in most cases requires a long preparation time with clinical researchers [36]. VOLUME 11, 2023 50831 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.  Data registries are used to evaluate and improve outcomes for populations defined by a condition, disease, or specific type of exposure. Data registries use observational research methods to collect and harmonize data on the treatment, outcomes, and well-being of patients receiving care over time. They also allow large data sets to be aggregated and analyzed for trends or patterns in treatment and outcomes [37].

c: KNOWLEDGE DATA
With the development of graph mining, knowledge-based data is emerging as a source of drug research, especially in the pre-marketing phase. Data on drug chemistry [38], drug targets [39], drug side effects [40], biological pathways [41], protein interactions [42], and drug interactions [43] are the most popular knowledge-based datasets used in PV research. H. Kwak et al. reported that integrating the graph structure of knowledge bases with real-world data (RWD) improves the causal interpretability of adverse drug event (ADE) detection [44]. These databases are also accessible through open platforms to be utilized in pharmacovigilance research (Supplementary Table 2).

2) MODELS a: TREE-BASED MODELS
Tree-based algorithms are conceptually simple but powerful ML methods that are effective on small and large datasets to solve linear and nonlinear modelling problems. Several tree-based ML algorithms were used in this study, including regression trees and ensemble learning, such as random forest, extreme gradient boosting (XGBoost) and adaptive boosting (AdaBoost) [45].
Tree-based ML models with XAI techniques can act as an 'early warning system' for per-patient disease-related adverse outcomes and harmful drugs in a PV monitoring system. With appropriate infrastructure and additional clinical data, these algorithms could provide an autonomous method for monitoring adverse outcomes from medications at the population level, providing a valuable addition to the existing statistical techniques, such as mean decrease of impurity (MDI) and mean decrease in accuracy (MDA), which are currently used and would be the next step in progressing towards a real-time PV monitoring system.

b: NEURAL NETWORK MODELS
Deep neural networks (DNNs) are the foundation of modern AI models and are used in many AI projects, including computer vision, speech recognition and robotics. They are based on the idea that the computation of a neuron involves a weighted sum of input values, where the weighted sum corresponds to the combination of these neuronal values and the value scaling performed by the synapses. Rather than simply outputting a weighted sum, the neuron performs a functional operation on the combined inputs within the neuron. To take advantage of this, researchers need to understand the key design considerations of DNNs and be able to evaluate the usefulness of different DNN design techniques for efficient processing [46].

c: GRAPH-BASED NEURAL NETWORK
In 2005, Gori et al. proposed a new neural network model capable of handling graph data structures: graph neural networks (GNNs). A pioneering study of deep learning methods in non-Euclidean spaces, GNNs generate random state embedding vectors, and the states of nodes are continuously updated according to the information propagation mechanism of the graph. Updates are made based on the state information of neighbouring nodes at a previous time [47]. GNNs are an alternative way to consider additional neighbourhood information in addition to the given data itself.
The local interpretable model-agnostic explanations (LIME) tool explains the model prediction using the most important contributors. It locally approximates the prediction by perturbing the input around the interest class until a linear approximation is obtained [53] and helps the decision maker to justify the behaviour of the model.
In some studies, there is a model that takes an individual's health information (e.g., medication history) as an input and predicts that an individual will have an adverse outcome from acute coronary syndrome (ACS). XAI was used to detect the contribution of specific drugs to the prediction of ACS. The post-doc use of SHAP and LIME successfully identified both important and relatively less important features [10]. Additionally, there are studies using text analysis to detect adverse drug reactions (ADRs) [15]. In ADR surveillance, the problem of class imbalance often has a negative impact on the outcome. A variant of weighted-CRF was proposed to solve the data imbalance, and LIME was applied to interpret the model results according to either the presence or absence of a weighted loss function.
Layer-wise relevance propagation (LRP) [54] is an explanatory technique applicable to models structured as neural networks, where inputs can be images, videos or text. LRP works by propagating the prediction backwards in the neural network using specifically designed local propagation rules. The propagation procedure implemented by LRP is subject to a conservation property, where what has been received by a neuron must be redistributed to the lower layer in equal amounts. Li et al. [20] proposed a CNN to continuously predict the cytotoxicity of compounds against a leukaemic lymphoblast CCRF-CEM cell line, and applied the LRP technology to a compound that had an important effect on predicting the biological properties of a compound. An attempt was made to identify the important parts of the structure. Compared to previous methods, such as chemical descriptors, it was confirmed that the structural formula of a compound can provide sufficient information for machine learning-based prediction of anticancer efficacy.

b: FEATURE IMPORTANCE
Permutation importance is often used to calculate feature importance: randomly rearrange the values of features and observe the predicted change in the model to determine which features contribute to model predictions. The importance weight is based on the predicted distance between the perturbed and original values of a feature. The importance of a feature can be interpreted as a weight and applied to all features [27]. The importance of a feature is measured by calculating the increase in the model's prediction error after permuting the feature. A feature is considered important if shuffling its values increases the model error because the model relies on that feature for prediction. A feature is considered unimportant if shuffling its values does not change the model error because the model ignores the feature for prediction. In particular, the permutation feature importance measure for random forests was introduced by Breiman and measures the increase in the model's prediction error after permuting feature values to disrupt the relationship between features and the actual outcome [55].

c: ATTENTION MECHANISM
The core concept of the attention mechanism is that the model is designed to pay attention only to the inputs that are most relevant to the predictive task [56]. The attention mechanism has been mainly proposed for natural language processing, including machine translation, and has been found to contribute to improving interpretability as well as technological evolution in the field of visualization [57].
Zhang et al. [12] considered that explainability is important so that stakeholders can trust the deep learning model in pharmacovigilance. Their research provides explainability and transparency by utilizing the attention mechanism for multitasking. Other studies have evaluated more clinically valid explanations for predicting drug side effects [13]. When the attention-based architecture and SHAP (a post-hoc surrogate model) were used together, it enabled appropriate local and global explanations complementary to each other; additionally, the SHAP method was evaluated as a more suitable for a real-time clinical explanation. Liu and Xie [35] developed their novel model, TranSynergy, for a synergistic drug combination prediction. As a knowledge-based and self-attention transformer model, TransSynergy has been demonstrated to improve predictability of synergistic drug combinations as well as improve interpretability. He et al. [21] used traditional machine learning methods such as boosting and Bayesian, as well as five deep learning prediction models including graph attention network and Attentive FP, to discover new drug candidates for the treatment of breast cancer. Attentive FP proposed by Bahdanau et al. [56] is a stateof-the-art graph-based neural network model for predicting molecular properties. It was confirmed that this model can focus on highly relevant parts of the input by applying the graph attention mechanism and can interpret the learned contents.

d: KNOWLEDGE DISTILLATION & GRAPH BASED EXPLANATIONS
Presently, in the field of deep learning, ''knowledge distillation'' refers to extracting knowledge from a trained model, as seen by the dictionary meaning of ''knowledge'' and ''distillation''. Several studies have been proposed to exploit this extracted knowledge. Knowledge distillation in deep learning can be summarized as a series of processes in which knowledge distilled from a large model (teacher model) is transferred to a small model (student network) [59]. Knowledge extraction is a form of model-specific XAI that extracts knowledge from a large, complex model to a more simplified model, and for many years model compression [60], tree regularization [61], and a combination of model compression and dimensionality reduction [62]. As distillation research has been fragmented, new possibilities have been identified with the development of artificial intelligence interpretability [63]. The rule extraction method, which is similar to knowledge extraction, is a useful XAI method and can be practically applied in the digital medical field [63].

B. ASPECTS OF XAI USE IN PHARMACOVIGILANCE 1) USING XAI TO PREDICT DRUG TREATMENT
The XAI system should be useful for supporting treatment decisions because it suggests a descriptive and visual approach to drug treatment decisions that can help doctors understand and apply recommendations confidently. These results suggest that, even in unguided clinical situations, the XAI system can promote accurate medication prescriptions by doctors [10].
By developing a post-interpretable framework based on an ensemble predictive model to interpret the importance of clinical features and genotypes in predicting daily drug doses, models with fewer variables can be built, and the complexity of the final predictive model can be reduced. Furthermore, it helps clinicians prescribe correct doses to patients using more effective clinical parameters [27]. In recent years, many studies have been conducted to determine an optimal model for accurately predicting drug treatments [10], [21], [27], [29], [30], [31], [32].
For medications associated with Alzheimer's disease, low to moderate doses of antipsychotics, moderate to high doses of antidepressants, and moderate to high doses of cardiovascular drugs have been identified as major features associated with Alzheimer's disease using SHAP [24]. Additionally, an interactive visualization tool has been designed and developed to assist domain users in GNN-based drug repurposing; in this, it was demonstrated how XAI can be used to examine treatments for Alzheimer's Disease (AD) [19]. In the field of cancer research, to gain a deeper understanding of the established models, SHAP was utilized to calculate the contribution of important structural fragments for breast cancer [21]. Additionally, the LRP technique was implemented to interpret the network and visualize the chemical groups predicted by a model that contributes to the toxicity of anticancer drugs with human-readable representations [20].

2) USING XAI TO DETECT ADVERSE DRUG REACTIONS
ADRs are statistically characterized in randomized clinical trials and post-marketing PV; however, their molecular mechanisms remain unknown in most cases. In addition to clinical trials, many elements of knowledge on drug ingredients are available in open-access knowledge graphs, such as their properties, interactions, and involvement in pathways [13]. Rapidly advancing AI technologies can facilitate data-driven estimation when screening multiple variables and capture non-linear relationships to accurately predict clinical outcomes. In recent years, many studies have been conducted to identify an optimal model for accurately predicting ADRs and taking effective preventive measures [17], [18], [25], [26], [27], [28].

3) USING XAI TO EXPLORE POLYPHARMACY
Drug combinations have demonstrated a significant potential for cancer treatment, as they relieve drug resistance and improve therapeutic efficacy. The rapidly increasing number of anticancer drugs has elicited costly and time-consuming experimental investigations of all drug combinations. However, computer technologies may improve the efficiency of drug combination screening. Despite recent advances in applying ML to predict synergistic drug combinations, much room for improvement remains. The XAI method can deconvolute genes contributing to synergistic drug combinations and improve model interpretability [35].

IV. OVERALL FINDINGS OF XAI USE IN PHARMACOVIGILANCE A. BENEFITS OF EXPLAINABLE PHARMACOVIGILANCE
PV XAI can provide clinicians and patients with a more comprehensible support system to ensure interpretability and explainability of AI models [64]. It can also simplify the model evaluation process while increasing the transparency and traceability of the model. There is great potential in systematically monitoring and managing XAI-based models and continuously collaborating with clinicians to enable fine-tuning to advance the system, enabling more efficient and accurate delivery of clinical decision support. The transparency of PV XAI can also manage regulations, risks and other requirements to alleviate PV model governance costs, minimize manual inspection overhead and costly errors, and reduce the risk of unintentional bias.

B. OPEN PROBLEMS, CHALLENGES, AND NEEDS OF EXPLAINABLE PHARMACOVIGILANCE
XAIs should be developed to enable clinicians to explain decisions in models so that patients can perceive and understand the explanations [65]. The field should consider developing a platform to support communication and linkage with more stable visualizations. This will enable clinicians to support clinical decisions that can build trust with patients and caregivers.
In order to develop a PV XAI model, collaboration with experts in related fields is very important but difficult; however, continuous consideration is required. An appropriate engineering basis for understanding the model used as well as a clinical understanding should be developed.
On the other hand, as models become more and more complex, the flaws of explainability must be accommodated. PV XAI cannot cover all aspects of pharmacovigilance. Therefore, all the life-cycle processes for data-method-result interpretation need to be considered that can balance and reduce bias in the results.

V. CONCLUSION
In this study, we reviewed PV XAI papers and discussed recent research trends and the need for XAI research. Unlike other areas where XAI and AI are developing together, PV XAI research is still in its infancy. There are not many papers on PV XAI and the methodology is limited to a few models. However, studies are slowly beginning to show the potential of XAI research for medication monitoring and patient safety, collecting ADR and ADE information, extracting drug-drug interactions, and predicting drug treatment effects.
As in other areas, as awareness of XAI methods grows, we expect to see AI used in pharmacovigilance and patient safety in many more ways in the coming years than those identified in this review, and the positive potential of XAI for drug therapy, ADRs and interactions is very promising. However, it is clear that the growth of this field may be limited by the lack of validated and established uses of XAI in real-world healthcare settings, and this is an area that requires further investigation. Therefore, the challenges and future prospects of XAIs in pharmacovigilance should be discussed with continued interest.
Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable.

Conflicts of Interest:
The authors declare no conflict of interest.