Towards Directive Explanations: Crafting Explainable AI Systems for Actionable Human-AI Interactions

1 RESEARCH BACKGROUND AND MOTIVATION

The utilisation of Artificial Intelligence (AI) systems has grown significantly in the past few years across diverse domains such as medical [12, 13, 36], finance [9, 11, 14], legal [43, 50, 53] and others [2, 6, 35]. Despite the success of AI systems across various applications, the “black-box” nature of AI models has raised several concerns related to lack of transparency [6, 28, 33] and appropriate trust [26, 46], particularly when predicted outcomes are biased, unfair, incorrect or misguiding [10, 21, 32]. Consequently, the field of Explainable AI (XAI) has gained much focus from AI practitioners as explanations can potentially help users to develop a clear mental model of such complex algorithms, eventually enhancing their trust for the adoption of AI systems [2, 17].

However, popularly adopted XAI methods [2, 6, 19, 29, 37] are predominantly designed for AI experts, neglecting the needs of a broader community of non-technical consumers of AI [9, 51]. Moreover, prior works have questioned the efficacy of current one-off static explanations for non-expert users, i.e. users who might be experts in a particular application domain but may lack technical AI knowledge [25]. Hence, there is a need for tailoring explanations for non-expert users and making XAI methods more useful, understandable, actionable and trustworthy through user-centric design principles [30].

Additionally, prior works have emphasised the need for directive explanations to non-expert users for enhanced human-AI collaborations [41, 44]. Directive explanations can assist users in obtaining their desired predictions by interacting with the prediction system for an actionable recourse [44]. Instead of static explanations, non-expert users have expressed the need for more interactive explanations to foster understanding and interpretation [1, 20, 23] of complex AI systems. Such explanations can enable them to understand why a certain prediction is generated and also indicate how to obtain their desired predictions without any intervention from AI experts [22, 28, 38, 45, 52]. Along with interactive explanations, prior research in Interactive Machine Learning (IML) has studied the collaborative interactions of users with AI systems and their vital role in model development, fine-tuning, debugging and evaluation [4, 15, 46, 49]. For instance, researchers of EluciDebug [23] have found that explanation-driven interactions can facilitate users to improve prediction models by helping them to identify bugs and issues and enhance their overall mental model of the system.

However, despite various explanation-driven interaction methods discussed in the literature [8, 23, 24, 48, 49], researchers have identified many challenges when non-technical consumers of AI have been involved in the AI prediction process [25, 40]. For example, non-expert users have expressed the need for more active involvement with AI models through interactive explanations of the underlying data rather than static explanations of the prediction algorithms [25].

Moreover, recent works have emphasised the need for Data-Centric AI (DCAI) [31] approaches for model improvement by improving the quality of the training data for building more reliable and trustworthy prediction models [5]. However, prior research has shown that improving the quality of training data is not straightforward without thorough domain knowledge [16]. Domain experts can identify certain types of biases in data to improve the training data quality. For example, instead of model developers, healthcare experts such as doctors or nurses have a better understanding of patient’s medical records, enabling them to identify biases and limitations in training data of prediction models for healthcare applications. Consequently, it is vital to capture the domain knowledge of domain experts through interactive explainability systems that give them more control over the training data [4, 18, 48].

2 RESEARCH OBJECTIVES AND METHODS

The author’s research focuses on crafting XAI systems that provide directive explanations for non-expert users and allow them to collaborate with the system for more reliable and useful predictions. The following are the primary research objectives:

The following high-level research questions further complement the preceding research objectives:

To address the above research questions, XAI systems are designed and developed following user-centric principles. Initially, exploratory studies are conducted to collect the needs of the target users for specific application areas. Then, in multiple iterations, low-fidelity prototypes are designed for initial evaluation with the target users. Then, high-fidelity prototypes are developed and evaluated through quantitative, qualitative or mixed-methods user studies. Finally, the data collected from these studies are analysed to extract valuable insights. These insights are instrumental in addressing the research questions and formulating design implications and guidelines for future research.

3 CURRENT STATUS

This section describes the current progress of the author’s research. To fulfil the research objectives, three research studies with two healthcare-focused interactive XAI systems have been conducted to date. The author has conducted six user studies in total involving healthcare experts and patients by following a user-centric design and development process.

Research Study 1 - In his first research study [7], the author designed and developed an explanation dashboard for monitoring the risk of diabetes onset. This interactive dashboard provides explanations for a diabetes risk prediction model by combining data-centric, feature importance and example-based local explanations. The dashboard aims to assist healthcare experts like nurses and physicians in monitoring patients and recommending measures to minimise their risk of type-2 diabetes. First, an exploratory user study was conducted through focus group discussion followed by a co-design session with 4 registered nurses from the Community Healthcare Centre dr. Adolf Drolc in Maribor, Slovenia, to collect the user requirements and understand their challenges in monitoring diabetic patients. Next, a qualitative user study was conducted through individual interviews with 11 healthcare experts to evaluate our low-fidelity click-through prototypical dashboard. Finally, a mixed-methods user study was conducted with 45 healthcare experts and 51 diabetic patients to evaluate our high-fidelity web application prototypical explanation dashboard. Through these user studies, the author measured the usefulness, understandability, actionability and trustworthiness of different types of explanations included in the dashboard. The results underscored the importance of the author’s representation of data-centric explanations that presented local explanations with a global overview of feature importance and example-based explanations. However, both healthcare experts and patients highlighted the importance of combining the different types of explanations for recommending risk-mitigating actions, indicating that any limitation of an individual explanation method can be complemented by other methods.

Research Study 2 - In the second research study [8], the author designed and developed an Explanatory Model Steering (EXMOS) system for healthcare experts. The EXMOS system provides different types of global explanations to support healthcare experts in improving prediction models through manual and automated data configurations. To evaluate this system, two between-subject user studies were conducted: one quantitative study with 70 healthcare experts and another qualitative study with 30 healthcare experts. The impact of diffrent types of global explanations on trust, understandability and model improvement was measured in this research study. The results highlighted the inefficiency of global model-centric explanations for guiding users during the configuration of the training data. Despite the benefits of data-centric global explanations in helping users comprehend the post-configuration system changes, it was found that a hybrid fusion of both data-centric and model-centric global explanations was most effective for steering prediction models.

Research Study 3 - In the latest research study, the author expanded his research with the EXMOS system to investigate the impact of different levels of control for model steering through data-centric approaches. More specifically, the latest research explored the effectiveness of manual and automated data configuration methods in healthcare-focused XAI systems. A between-subject mixed-methods user study was conducted with 74 healthcare experts to evaluate the effectiveness of different data configuration methods across multiple measures such as understandability, trust, model improvement, explanation goodness and satisfaction, feedback importance and usability and system interactions. It was found that the study participants could significantly improve the performance of the prediction model using the manual configurations instead of the automated method. The study findings highlighted the necessity of a higher involvement of domain experts for enhanced human-AI collaboration.

4 FUTURE WORK

In his previous research studies, the author has mostly focused on different types of visual explanations designed to elucidate prediction models built on structured datasets. However, with the recent advancements in Generative AI [47] and Large Language Models (LLMs) [54], he plans to research on designing XAI systems, which include AI models built on unstructured data such as text or images. The author aims to follow a similar research procedure as our previous studies to explore different approaches for actionable and human-friendly explanations. Furthermore, his current research has highlighted the need for more objective measures to evaluate different XAI methods instead of subjective metrics. Thus, as next steps of his current research, the author wants to focus on the following areas:

Explainability using LLMs - The author hypothesises that LLMs can be used to generate more natural and human-friendly explanations and foster more natural interaction approaches with AI models. More specifically, the author wants to include LLM-based chatbots [25, 39, 42] and allow non-expert users to interact with prediction models through conversation-based explanations and model steering approaches. He proposes to conduct randomised control experiments with non-expert users to measure the effectiveness of LLMs in generating more actionable, causal and contextual explanations. The author would also like to collect more qualitative data to understand the benefits and pitfalls of using LLMs in explanatory model steering.

Measuring diverse XAI methods - More recently, many researchers have questioned the efficacy of current quantitative user-centric metrics to compare different XAI methods [3, 27, 34]. Most of these metrics are highly subjective to the research participants and do not provide a generalised evaluation of different types of XAI methods for other users or applications. Thus, the author’s current research studies have raised a vital open question: “how to measure diverse XAI methods in a unified and fair approach as each method elucidates different aspects of an AI system?" Therefore, he proposes to conduct a systematic review of the research literature to compare the benefits and drawbacks of subjective user-centric XAI metrics and objective XAI metrics. After this systematic review, he aims to formulate a unified and fair evaluation approach for inter-system and intra-system comparisons of XAI methods from this research and evaluate this novel approach with XAI researchers to collect their feedback.

ACKNOWLEDGMENTS

The author would like to thank his supervisor, Prof. Katrien Verbert, for her constant support throughout his PhD. This work is supported by research grants: Research Foundation–Flanders (FWO, grant G0A3319N) and KU Leuven Internal Funds (grant C14/21/072). He would like to extend his gratitude to his collaborators Dr. Simone Stumpf (University of Glasgow, UK), Dr. Gregor Stiglic (University of Maribor, Slovenia) and the Augment HCI group, KU Leuven, for their support during this research work.