2504.00125

# LLMs for Explainable AI: A Comprehensive Survey **Authors**: Ahsan Bilal, David Ebert, Beiyu Lin > University of Oklahoma Norman OK USA ahsan.bilal-1, ebert, ## Abstract Large Language Models (LLMs) offer a promising approach to enhancing Explainable AI (XAI) by transforming complex machine learning outputs into easy-to-understand narratives, making model predictions more accessible to users, and helping bridge the gap between sophisticated model behavior and human interpretability. AI models, such as state-of-the-art neural networks and deep learning models, are often seen as ”black boxes” due to a lack of transparency. As users cannot fully understand how the models reach conclusions, users have difficulty trusting decisions from AI models, which leads to less effective decision-making processes, reduced accountabilities, and unclear potential biases. A challenge arises in developing explainable AI (XAI) models to gain users’ trust and provide insights into how models generate their outputs. With the development of Large Language Models, we want to explore the possibilities of using human language-based models, LLMs, for model explainabilities. This survey provides a comprehensive overview of existing approaches regarding LLMs for XAI, and evaluation techniques for LLM-generated explanation, discusses the corresponding challenges and limitations, and examines real-world applications. Finally, we discuss future directions by emphasizing the need for more interpretable, automated, user-centric, and multidisciplinary approaches for XAI via LLMs. Large Language Models (LLMs), Explainable AI (XAI), Model Interpretability, AI Trustworthiness, Natural Language-Based Explanations, Evaluation Techniques for Explanations copyright: none conference: ; ; conference: ACM Transactions on Intelligent Systems and Technology; March 2025; USA ccs: Computing methodologies Natural language generation ## 1. Introduction Recent advances in artificial intelligence (AI) have resulted in the development of more complex models, especially in the field of deep learning. AI models have demonstrated remarkable capabilities in various domains, from healthcare to finance (Khan et al., 2024), (Cao, 2020). However, as these AI models become more complex, it is challenging to understand how specific outputs are generated due to a lack of transparency (Barredo Arrieta et al., 2020). The lack of transparency in AI model decision-making is often referred to as a ”black box” problem, which hinders trust and limits widespread adoption, particularly in critical fields such as healthcare and finance (Bhatt et al., 2020). Despite ongoing efforts to enhance the explainability of the AI model (Hosain et al., 2024), many experts without a background in machine learning still struggle to understand how these systems generate decisions. Enhancing transparency is therefore crucial to increasing trust and facilitating broader deployment across various domains. This makes it challenging for them to use these explanations to make better choices. For example, in healthcare, a doctor might not fully understand why a machine learning model recommends a particular treatment, making it hard for them to trust or act on the model’s advice. In finance, a financial analyst may have difficulty interpreting how an AI system predicts market trends, which could lead to hesitation in relying on the model’s predictions. Explainable AI (XAI) includes methods to improve the interpretability of neural networks and other state-of-the-art AI models, such as Convolutional Neural Networks (CNNs) for image recognition, Recurrent Neural Networks (RNNs) for sequence data, and Generative Adversarial Networks (GANs) for image generation, aiming to enhance transparency without compromising performance metrics, such as accuracy (Barredo Arrieta et al., 2020). XAI helps balance the tradeoff between making models understandable and maintaining their effectiveness. Explainability remains a significant challenge across various applications (Ribeiro et al., 2016), (Xu et al., 2021), (Camburu et al., 2018a). By making model decisions understandable, XAI can build user trust, ensure accountability, and promote responsible and ethical use of these models. LLMs are becoming critical tools with broad applications across diverse domains. In medicine, they assist in tasks, such as diagnostics and personalized patient care (Thirunavukarasu et al., 2023). In finance, they support functions, such as risk assessment and market analysis (Wu et al., 2023). Additionally, in the field of natural language processing (NLP), LLMs are essential for tasks, such as text classification, summarization, and sentiment analysis. The extensive data on which LLMs are trained offers valuable potential for clarifying model architectures and the decision-making processes of AI models (Zhao et al., 2024; Amin et al., 2023). LLMs are an important link between complex AI models and XAI systems due to their natural language processing capabilities (Liu et al., 2023). For example, instead of just making a prediction in a medical imaging AI model, the model might use an LLM to explain why it flagged a lung scan as abnormal, including highlighting patterns associated with specific diseases. LLMs help in XAI in many ways, such as understanding user questions to generate the appropriate explanation (Nguyen et al., 2023), (Shen et al., 2023), and (Slack et al., 2023) also by directly explaining the complex ML model architecture and explanation to their output (Kroeger et al., 2023). For instance, (Bhattacharjee et al., 2024; Memon et al., 2024a) demonstrated the use of a simple prompting method to identify key features of an AI model’s predictions and generate counterfactual explanations using LLMs. This approach highlights the potential of LLMs to improve understanding and transparency in AI decision-making processes, making AI systems more interpretable and trustworthy across diverse fields. Our survey discusses various approaches to LLM-based Explainability, as shown in Figure 2 and in Table 1. We will specifically discuss three approaches: 1). The first approach, post-hoc explanations, corresponds to causal interpretability and focuses on analyzing why a specific input led to a particular output, providing explanations for output generated by the machine learning (ML) model; 2). The second approach, intrinsic explainability, corresponds with engineers’ interpretability and involves designing machine learning model architectures using LLMs to make machine learning models more explainable.; 3). The third approach, human-centered narratives, corresponds to trust-inducing interpretability by enhancing explanations for the outputs generated by ML models through natural language, making the outputs more understandable and fostering user trust. For example, the AI model predicts a high probability that a patient develops hypertension in the next five years due to their history of high cholesterol, a family history of hypertension, and contributing factors, such as age and weight. While their blood pressure is normal, these risk factors suggest future concerns. This narrative helps the doctor understand the reasoning behind the prediction. This survey will also explore evaluation techniques for these explanations and how explainability can be applied in real-world applications. Then, we discuss the challenges and limitations of achieving explainability in AI models using LLMs, as illustrated in Figure 3. We discuss this part in three topics: 1). understanding societal norms in terms of privacy; 2). managing the complexities of AI systems; 3). adapting LLMs to domain-specific tasks. We examine how various state-of-the-art LLM architectures emphasize different aspects of explainability using saliency maps, as shown in Figure 4. Finally, we outline future directions for improving explainability through LLMs. By integrating insights from both model architecture and narrative approaches, our survey aims to present a comprehensive understanding of how LLMs can improve explainability in AI systems. ## 2. Background The urgent need to interpret and trust AI systems has increased manifold in recent years. The evolution of XAI would help with this urgent need by making AI decisions transparent and easy for human beings to comprehend. Meanwhile, the tremendous advancements in LLMs have opened up new possibilities of generating detailed and dynamic explanations for AI systems. Additionally, LLMs provide experience in ML and AI to nontechnical and technical people for better understanding and effective acting on model decisions, opening up AI technologies to larger users. Merging XAI techniques with the capabilities of LLMs enables researchers and practitioners to create more interpretable AI across domains that are reliable, fair, and user-friendly. The next sections discuss XAI, explore how LLMs contribute to explainability, and identify areas for future research in AI and NLP. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: AI Explainability Enhancement Flowchart ### Overview This image is a flowchart diagram illustrating a process for enhancing the explainability of an AI system that predicts purchase likelihood. The diagram shows how a core "XAI (Explainable AI) Process" is augmented by a Large Language Model (LLM) to produce outputs that are understandable to end-users. The flow moves from left to right, starting with an AI prediction task and culminating in actionable insights for a user. ### Components/Axes The diagram is structured into three main horizontal sections, with a clear left-to-right flow indicated by arrows. 1. **Initial Input (Top-Left):** * **Icon:** A database cylinder with a magnifying glass. * **Text Box:** "The AI predicts purchase likelihood from browsing history." 2. **Left Section - "XAI PROCESS" (Orange Header):** * This section is enclosed in a dashed-line box and contains three stacked components. * **Component 1 (Top):** * **Title:** ACCURACY * **Description:** "Decision based on cosine similarity to past behavior." * **Component 2 (Middle):** * **Title:** TRANSPARENCY * **Description:** "Model uses features like time spent on page and click patterns." * **Component 3 (Bottom):** * **Title:** STAKEHOLDER NEEDS * **Description:** "Explanation highlights customer segments for marketing." 3. **Right Section - "LLM ENHANCING EXPLAINABILITY" (Orange Header):** * This section is also enclosed in a dashed-line box and shows the LLM's role. * **Central Component:** A box labeled "LLM" containing a stylized icon of stacked books or documents. * **Arrows:** Two arrows point from the "LLM" box to two output boxes on its right. * **Output Box 1 (Top):** * **Title:** ADDING CONTEXT * **Description:** "Prediction aligns with patterns from last quarter" * **Output Box 2 (Bottom):** * **Title:** ENHANCED UNDERSTANDING * **Description:** "Like a friend's recommendation based on your choices." 4. **Final Output (Bottom-Right) - "END-USER UNDERSTANDING" (Orange Header):** * A single, wide text box at the bottom of the diagram. * **Text:** "AI suggests targeting based on strong behavioral patterns, with a confidence score for clarity and actionable insights." 5. **Flow Connections:** * Three arrows originate from the right side of the "XAI PROCESS" box (one from each of its three components) and converge, pointing into the left side of the "LLM" box. * Two arrows originate from the right side of the "LLM" box, pointing to the "ADDING CONTEXT" and "ENHANCED UNDERSTANDING" boxes. * A final arrow flows from the "LLM ENHANCING EXPLAINABILITY" section down to the "END-USER UNDERSTANDING" box. ### Detailed Analysis The diagram presents a conceptual pipeline, not a data chart. The information is entirely textual and relational. * **Process Flow:** The core XAI process (focused on Accuracy, Transparency, and Stakeholder Needs) provides the foundational explainability. This output is then processed by an LLM. * **LLM Function:** The LLM acts as an enhancer and translator. It takes the technical explanations from the XAI process and adds two key layers: 1. **Contextualization:** It frames the prediction within recent historical patterns ("last quarter"). 2. **Analogy/Relatability:** It translates the logic into a human-understandable analogy ("like a friend's recommendation"). * **Final Transformation:** The combined output is synthesized into a final, user-centric form: a targeting suggestion that includes a confidence score, making it both clear and actionable. ### Key Observations * **Dual-Layer Explanation:** The diagram explicitly separates the technical, model-centric explanation (XAI Process) from the user-centric, contextualized explanation (LLM Enhancement). * **Stakeholder Focus:** The "STAKEHOLDER NEEDS" component within the XAI process indicates the initial explanation is already tailored for a specific audience (marketing), which the LLM then further refines for an end-user. * **Action-Oriented Output:** The final box emphasizes "actionable insights," moving beyond mere explanation to prescribed action. ### Interpretation This diagram argues that raw explainability from an AI model (the XAI Process) is insufficient for effective human understanding and decision-making. It posits that a Large Language Model is a crucial intermediary layer that can bridge the gap between technical model behavior and practical, business-ready insight. The flow suggests a value chain: **Data → Prediction → Technical Explanation → Contextualized & Analogical Explanation → Actionable Insight.** The LLM's role is not to explain the model's inner workings more accurately, but to make the existing explanation more **comprehensible, relevant, and useful** for a non-technical end-user. The inclusion of a "confidence score" in the final output is a key detail, as it adds a layer of metacognition—explaining not just *what* the AI recommends, but *how sure* it is, which is critical for trust and decision-making. The overall message is that effective AI deployment requires not just predictive accuracy, but a deliberate pipeline for transforming model outputs into human-centric understanding. </details> Figure 1. Explainable AI and Its Intersection with LLMs ### 2.1. Explainable AI Explainable AI (XAI) refers to a range of methods that make AI model outputs more transparent and understandable to humans, providing access to the inner workings of AI models so that users can understand the design and logic behind AI systems. XAI creates models that are more understandable, trustworthy, and ethically aligned with user expectations and regulatory requirements. XAI helps increase the interpretability of complex models by clarifying the most influential factors of AI predictions, such as the key input features identified in machine learning models (Ribeiro et al., 2016). XAI improves transparency, fairness, and reliability, to seamlessly align with the ethical principles. Regulatory bodies, such as the European General Data Protection Regulation (GDPR) emphasize transparency by encouraging AI systems to be understandable and accountable to both users and regulatory bodies (Bostrom and Yudkowsky, 2018). In fairness, XAI helps developers identify biases and limitations within models (Pahde et al., 2023), allowing software developers to debug algorithms and improve their decision-making processes. For reliability, industry professionals, such as product engineers, use XAI-driven insights to responsibly integrate AI solutions into applications (Weld and Bansal, 2019), ensuring that products are reliable. In short, XAI offers clear explanations that help end-users understand and trust AI outputs. ### 2.2. Leveraging Large Language Models for AI Model Explainability: Large Language Models (LLMs) have developed significantly, starting with the basic models, such as GPT and BERT, and then advancing to more complex and larger models, such as LLaMa (Touvron et al., 2023). Transformer architectures in LLMs enable LLMs to process large amounts of data efficiently. With billions of parameters, LLMs can highly capture complex and important features in the input (Touvron et al., 2023). LLMs are trained on massive datasets (Penedo et al., 2023) and fine-tuned to enhance their ability to follow instructions (Ouyang et al., 2022). LLMs can now tackle a wide range of tasks beyond text generation, providing valuable insights for model explainability. Traditional methods, such as Principal Component Analysis (PCA) rely on statistical techniques for feature selection. In contrast, large language models (LLMs) utilize extensive contextual knowledge to identify and emphasize important features based on input data dynamically. The adaptability of LLMs to dynamically focus on important features allows LLMs to offer context-sensitive explanations, making them particularly useful in complex domains. For example, in the communication domain, where during communication, understanding the contributing factors, such as signal interference, propagation loss, multipath fading, noise, and weather conditions, is a challenge (Mohsin et al., 2025c, b; Bhattacharya et al., 2025). LLMs help clarify the underlying complexities. By breaking down complex models and filling knowledge gaps, LLMs provide valuable insights into dynamic environments, demonstrating their potential to enhance AI model explainability in various fields, such as communication and cybersecurity (Zou et al., 2024; Erak et al., 2024) (Feng et al., 2023b). In the realm of cybersecurity, LLMs have proven effective for explainable AI. For example, LLMs generate human-readable insights to identify the detection patterns on very specific malware or phishing attack detection, network anomalies, and so on. LLMs improve the transparency of an Intrusion Detection System (IDS)’s decision-making process by helping users understand why specific alerts were triggered. For example, LLMs can provide explanations about patterns in network traffic, highlight anomalous behaviors, or detail the specific rules or thresholds that caused an alert (Elouardi et al., 2024). LLMs improve explainability as shown in Figure 1 and build trust by making the reasoning behind alerts and actions more accessible to users (Khediri et al., 2024). LLMs, therefore, generate explanations and facilitate the creation of more intuitive and interpretable AI systems. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: Taxonomy of Explainability Techniques for Large Language Models (LLMs) ### Overview The image is a hierarchical flowchart diagram illustrating a taxonomy of techniques for achieving explainability in Large Language Models (LLMs). It categorizes approaches into three primary methods, each with associated examples. The diagram flows from a central concept on the left to specific techniques and their implementations on the right. ### Components/Axes The diagram is structured with two header boxes at the top and a tree-like flow below. **Header Boxes (Top, Orange Background):** * **Left Header:** "Explainability Techniques" * **Right Header:** "Examples" **Main Flow (Left to Right):** 1. **Root Node (Far Left):** A single rounded rectangle labeled "Explainability in LLMs". 2. **Primary Technique Nodes (Center Column):** Three rounded rectangles branch from the root node, each representing a core technique category. The technique name is in bold at the bottom of each box. * Top: "Provide explanation on model's output" / **Post-Hoc Explanations** * Middle: "Design LLMs to be inherently interpretable" / **Intrinsic Interpretability** * Bottom: "Natural language explanations generated by LLMs" / **Human-Centered Explanations** 3. **Example Nodes (Right Column):** Five rounded rectangles provide specific examples, connected by arrows from their parent technique node. * Connected to *Post-Hoc Explanations*: "SHAP, LIME tools" * Connected to *Intrinsic Interpretability*: "Transparent model architecture" and "Attention-based interpretability" * Connected to *Human-Centered Explanations*: "Narrative-based explanations" and "Natural Language generation" **Visual Relationships:** Black arrows indicate the flow of categorization, originating from the "Explainability in LLMs" node and pointing to the three technique nodes. Further arrows connect each technique node to its corresponding example nodes. ### Detailed Analysis The diagram presents a clear, three-tiered classification system: 1. **Post-Hoc Explanations:** This technique focuses on analyzing a trained model after the fact. The description "Provide explanation on model's output" indicates these methods are applied externally to the model's decision process. The examples given are "SHAP, LIME tools," which are well-known model-agnostic interpretability frameworks. 2. **Intrinsic Interpretability:** This technique involves building interpretability into the model's design from the start, as stated by "Design LLMs to be inherently interpretable." It branches into two sub-approaches: * "Transparent model architecture": Suggesting models designed with simpler, more understandable structures. * "Attention-based interpretability": Leveraging the attention mechanism's weights to infer what parts of the input the model focuses on. 3. **Human-Centered Explanations:** This technique uses the LLM's own generative capability to explain itself, described as "Natural language explanations generated by LLMs." It also has two sub-approaches: * "Narrative-based explanations": Generating coherent stories or step-by-step reasoning. * "Natural Language generation": A broader category for producing explanatory text. ### Key Observations * The diagram establishes a clear hierarchy: a single problem ("Explainability in LLMs") is addressed by three distinct philosophical approaches (Post-Hoc, Intrinsic, Human-Centered), which are then grounded in concrete methods or tools. * The "Intrinsic Interpretability" and "Human-Centered Explanations" categories are further subdivided, indicating they encompass a wider range of strategies compared to the more tool-focused "Post-Hoc Explanations." * The visual layout uses consistent shapes (rounded rectangles) and arrow styles, with color used only in the header boxes to separate the conceptual labels ("Explainability Techniques", "Examples") from the content. ### Interpretation This diagram serves as a conceptual map for understanding the landscape of LLM explainability. It suggests that there is no single solution; rather, the field employs a multi-pronged strategy. * **The data suggests a progression in approach:** From external analysis (Post-Hoc), to internal design (Intrinsic), to collaborative dialogue (Human-Centered). This reflects an evolution from treating the model as a black box to be probed, to a transparent system, to an interactive partner. * **The elements relate to each other as a taxonomy.** The root defines the domain, the primary nodes define the strategic categories, and the leaf nodes provide actionable instances. This structure helps researchers and practitioners定位 (locate) specific methods within a broader framework. * **A notable insight is the inclusion of "Human-Centered Explanations."** This category acknowledges that for complex systems like LLMs, a technically perfect explanation may be less useful than one that is naturally understandable to a human user, even if it is generated by the model itself. This highlights a key tension in the field between mechanistic interpretability and practical utility. * The diagram implies that "Intrinsic Interpretability" might be the most challenging, as it requires fundamental changes to model architecture, whereas "Post-Hoc" and "Human-Centered" methods can often be applied to existing models. </details> Figure 2. Techniques for explainability in Large Language Models (LLMs). The diagram identifies three broad categories of approaches to explaining: Post-hoc Explanation, Intrinsic Interpretability, and Human-Centered Explanations, providing examples of methods in each category. ## 3. Overview of Explainability Techniques Techniques for explainability in AI systems using Large Language Models (LLMs) are categorized into 1) post-hoc explanations, 2) intrinsic interpretability, and 3) human-centered explanations (Zhang et al., 2022). Post-hoc explanations, such as SHAP and LIME, clarify model predictions by analyzing feature contributions after decisions are made. Intrinsic interpretability focuses on designing inherently explainable models, such as Chain of Thought (CoT) reasoning, which breaks tasks into logical steps. Human-centered approaches incorporate user feedback to ensure explanations are clear and tailored to human needs, enhancing the overall interpretability of AI models. A visual representation of explainability techniques is provided in Figure 2. ### 3.1. Post-hoc Explanations Post-hoc explanations clarify the decisions that large language models make after generating predictions. Post-hoc explanations help users to understand the reasoning behind a model’s decision by analyzing how the model arrived at a specific conclusion. The post-hoc technique provides two types of explanations (Setzu et al., 2021): 1) global explanations and 2) local explanations. Global explanations illustrate the overall behavior of the AI model, and local explanations focus on a single prediction and highlight the important input features that contributed to that result. For example, consider an email classification model that predicts whether an email is spam. A global explanation identifies features, such as keywords (e.g., ”free,” ”winner,” ”click here”) and patterns (e.g., excessive exclamation marks or links) that influence the model’s spam predictions across all emails. In contrast, a local explanation for a specific email focuses on the terms or phrases that contributed most to that particular prediction, such as ”claim your prize,” ”congratulations,” or a high count of hyperlinks, providing insight into why the model classified that email as spam. Post-hoc explanation tools interpret and explain a model’s decisions after the model makes a prediction. The methods and tools for post-hoc explanations include Integrated Gradients (IG), Local Interpretable Model-Agnostic Explanations (LIME), and Shapley Additive explanations (SHAP). Integrated Gradients (IG) (Bhat and Raychowdhury, 2023) explains individual predictions by identifying the most effective parts of the input concerning the output. IG reveals the features that have the most impact on specific predictions by analyzing how changes in the input affect the output. LIME focuses on local explanations by identifying the key features of a single prediction. LIME modifies the version of the dataset around the prediction and, analyzes the feature importance using techniques, such as LASSO or principal component analysis (PCA). In contrast, SHAP provides both local and global explanations. SHAP explains individual predictions by using Shapley values (a concept from game theory) to measure the importance of each feature (Park et al., 2021). In local explanations, SHAP works by analyzing changes in the model prediction by removing different features and determining the contribution of every feature to the prediction. The Shapley values from SHAP provide the relation between the specific prediction and the model’s behavior. For example, in a spam email detection model, SHAP shows that the presence of the word ”free” increases the likelihood of the email being spam, while the absence of words, such as ”meeting” reduces the likelihood and helps to explain the model that specifies email as spam (Bouke et al., 2024). SHAP extends its utility beyond local explanations by aggregating feature contributions across the entire dataset to provide a global explanation. For example, in the same spam email detection model, a global explanation reveals that certain features, such as frequent occurrences of promotional words (”offer,” ”discount”), and patterns, such as high density of hyperlinks, are consistently influential in the model’s spam classifications across all data points. Post-hoc explanations enhance model transparency by analyzing factors that influenced the output, such as specific input (local explanation) or general trends across the data (global explanation). The increase in transparency reduces the perception of the model as a ”black box” and reassures users that the model is making decisions based on understandable and reasonable criteria, rather than arbitrary or hidden processes. However, post-hoc methods have limitations. The explanations can be unstable, as small changes in the input data can result in inconsistent results. For example, a slight variation in the input features might significantly alter the explanation, even though the model’s prediction remains the same. Furthermore, post-hoc explanations focus on interpreting the model’s predictions rather than fully capturing model’s internal workings. For instance, in a spam email detection model, a post-hoc method might explain, why the model predicts spam by identifying relevant keywords or patterns, but may not reveal the detailed processes or computations the model used to make that decision. ### 3.2. Intrinsic Interpretability XAI has two main approaches: explaining predictions and designing interpretable model architectures. Post-hoc explanations focus on clarifying the model’s decisions after the model generates the prediction (the first approach), and intrinsic interpretability focuses on building models that are understandable by design (the second approach). The intrinsic interpretability approach involves structuring the model architecture using large language models (LLMs), to make the model’s behavior inherently explainable. For instance, in a sentiment analysis task, an LLM equipped with attention mechanisms can show phrases, such as ”excellent service” or ”poor quality” that directly contribute to a positive or negative sentiment classification. Attention weights act as built-in explanations, making the model’s decision-making process easier. With the help of built-in explanations, the model’s decisions are understandable by design, without needing external explanation tools (Tull et al., 2024). A notable technique in this category is Chain of Thought (CoT) reasoning (Chu et al., 2023), which breaks down complex problems into smaller, manageable steps, allowing models to follow a logical progression toward conclusions. The logical flow reduces error propagation and makes the model’s logic easier to follow. For example, in predicting a patient’s condition, CoT reasoning divides the decision-making process into sequential steps: first identifying symptoms, then evaluating potential diagnoses, and finally analyzing the relationships between symptoms and diagnoses. The sequential steps help to make the overall reasoning of AI model more interpretable and clear. Methods in the Chain of Thought (CoT) include 1) Guided CoT Templates and 2) ReAct (Reasoning and Acting). Guided CoT templates provide logical steps to guide the model’s reasoning process and ReAct incorporates task-specific actions to enhance the intrinsic interpretability of models (Wei et al., 2023). Guided CoT templates provide structured, predefined templates that help models follow a logical, multistep reasoning process. For example, in a math problem-solving task, a Guided CoT template prompts the model to break the problem into smaller steps (e.g., “First, calculate the sum, then subtract the result from the total”), to make the model’s internal reasoning more understandable, and improve the intrinsic interpretability by revealing how the model arrives at each step. ReAct enables models to perform iterative task-specific actions with step-by-step reasoning. For instance, in a navigation task, ReAct guides the model to reason about the best route, take a step (move forward), and then reason again based on the new position. The iterative reasoning and action increase the interpretability of AI model by real-time tracing of the decision-making process, effectively correcting the model’s explanation and revealing the internal actions and decisions of the model during execution. For example, in a question-answering model, ReAct involves the following steps: 1) prompting the model to read a question, 2) extracting relevant information from the text, 3) reasoning about the correct answer, and 4) providing an explanation for its choice. Intrinsic interpretability helps in architecture-based XAI by following a structured process. The first process involves LLMs generating explanations, training deep learning models on these explanations, and modifying the model architecture accordingly. Refining the architecture improves the accuracy and clarity of the generated explanation. The step proves particularly useful for tasks that demand both reasoning and interaction with external systems or calculations to ensure the necessary explanations are clear and precise. After generating the explanation, the second process checks the reliability of the explanation. Factual accuracy and logical correctness are two key aspects of a reliable architecture-based XAI. (Yang et al., 2023b). Factual accuracy ensures that the explanation aligns with real-world information and provides the correct output based on the model’s inputs. Logical correctness, on the other hand, focuses on the reasoning process, ensuring that the steps taken to conclude follow a coherent and consistent pattern. Factual accuracy alone does not guarantee reliable explanations; even incorrect outputs can be presented in convincing ways. For example, a model may explain a medical disease prediction, such as identifying diabetes risk based on patient symptoms, that is accurate but based on incorrect assumptions or unrelated data, such as a spurious correlation between age and unrelated dietary habits. Studies (Golovneva et al., 2022) show that factual accuracy combined with logical consistency leads to more reliable explanations, and helps build user confidence in the model’s outputs. For example, in legal case analysis, LLMs generate an explanation that accurately cites relevant laws (factual accuracy) while also outlining a logical step-by-step reasoning process to apply relevant laws to the case. The combination ensures that the explanation is both correct and easy for users to trust and understand.” The above discussion shows that Leveraging the LLMs for model generation, emphasizing both factual accuracy and logical correctness, and combining clear reasoning with interpretable outputs improve the architectural reliability of XAI. Table 1. Overview of XAI with LLMs | Background | XAI promotes transparency, trust, and regulatory compliance | (Bostrom and Yudkowsky, 2018), (Weld and Bansal, 2019), (Ribeiro et al., 2016), (Touvron et al., 2023), (Penedo et al., 2023), (Zou et al., 2024) | | --- | --- | --- | | Post-hoc explanations | LIME for local interpretations, SHAP for global and local insights | (Lundberg and Lee, 2017), (Park et al., 2021), (Setzu et al., 2021), (Bhat and Raychowdhury, 2023), (Bouke et al., 2024) | | Intrinsic interpretability | Chain of thought reasoning, guided templates for logical progression | (Chu et al., 2023), (Wang et al., 2024), (Golovneva et al., 2022), (Lanham et al., 2023), (Yeo et al., 2023), (Tull et al., 2024) | | Human-centered explanations | User-focused design, collaborative feedback methods | (Eiband et al., 2018), (Amershi et al., 2019), (Neerincx et al., 2019), (Ehsan and Riedl, 2020), (Wolf, 2019), (Jiang et al., 2024b) | | Evaluation of explanation | Qualitative and quantitative evaluation of LLM generated explanation | (Zhao et al., 2024), (Cao, 2023), (Nauta et al., 2023), (Dong et al., 2017), (Chen et al., 2021), (Sokol and Vogt, 2024), (Jacovi and Goldberg, 2020), (Burton et al., 2023), (Min et al., 2023), (Ichmoukhamedov et al., 2024) | | Benchmark datasets | Such as e-SNLI, CoS-E, and ECQA help in training the LLMs to generate clear, human-understandable explanations | (Camburu et al., 2018b), (Rajani et al., 2019), (Aggarwal et al., 2021), (Jansen et al., 2018), (Mihaylov et al., 2018), (Chen et al., 2024), (Friel et al., 2024), (Mathew et al., 2021). | | Application of XAI in different domains | Application of explainability using LLMs | (Umerenkov et al., 2023a), (Attai et al., 2024), (Dave et al., 2024), (Ehsan and Riedl, 2020), (Umerenkov et al., 2023b), (Yang et al., 2023a), (Benary et al., 2023), (Luz, 2024), (Feng et al., 2023a), (Thanh and Satoh, 2024) | | Challenges in XAI | Sensitive data, societal diversity, complex multi-source and other factors | (Huang et al., 2023), (Caliskan, 2023), (Klein, 2020), (Fang et al., 2023), (Wang et al., 2023), (Hua et al., 2024) | | Analysis of Feature importance | Feature importance using saliency map across LLMs | (Zhou et al., 2021), (Zytek et al., 2024b), (Jiang et al., 2023), (Bai et al., 2023), (OpenAI et al., 2023), (Koo et al., 2024), (Touvron et al., 2023), (Almazrouei et al., 2023), (Jiang et al., 2023), (Groeneveld et al., 2024) | | Future directions | Human-in-loop automation, visual explanations, interdisciplinary approach | (Do et al., 2024), (Zhang et al., 2024), (Selvaraju et al., 2019), (Ehsan and Riedl, 2020), (Schoonderwoerd et al., 2021), (Cabour et al., 2023), (Cai et al., 2019) | ### 3.3. Human-Centered Explanations Both post-hoc and intrinsic interpretability methods significantly improve XAI by cross-checking with factual ground truth. However, the challenge arises in presenting this explanation as understandable to the end user. Human-centered approach that designs XAI with the end user in the workflow (Eiband et al., 2018; Amershi et al., 2019) addresses the challenge of comprehensible XAI. The approch provides guidelines for researchers and developers to create explanations that cater to human needs. It also discusses how to apply human-centered design methods to better understand the social and technical aspects for better human-AI interaction (Neerincx et al., 2019; Ehsan and Riedl, 2020). A human-centered approach to XAI involves determining what, when, and how to explain information to users. Interviews, focus groups, and surveys are common ways to include users in the workflow of the model development process. In previous research, the approach of keeping the end user in the workflow plays an important role in improving the transparency of AI models. For example, research (Eiband et al., 2018) outlines ways to enhance transparency in intelligent systems by involving users in a step-by-step design process for developing explanations. Research (Wolf, 2019) illustrates the importance of using user scenarios early in the design process to identify users’ needs for explanations and to guide further development. Collaboration of end user with AI model, helps developers understand user values and the social context of human-AI interactions. The human-centered technique uses contexts of human-AI interaction to refine and improve the explanation. Specifically, the technique captures user preferences, comprehension levels, and relevant contextual factors to refine the explanation. The refined explanations are then tested with users in realistic work settings, focusing on important information for clear comprehensibility between user and AI model (Jiang et al., 2024b). For example, the DoReMi framework applies the four Human-centralized design principles (understand, define, design, evaluate) to XAI, enhancing existing methods by generating knowledge through design patterns that connect user requirements with design solutions. The researcher (Martens et al., 2023) generates human-centered explanations using counterfactual (CF) information in combination with large language models (LLMs). The study uses various prompts for the LLMs by giving model prediction as the context to generate a narrative based on the prompts. For example, if a model predicts a low credit score, a prompt might ask the LLMs to explain how a higher income or lower debt could change the prediction. Depending on users’ reactions to the generated explanation, different adjustments are made for efficient XAI. In short, a human-centered approach focuses on gathering essential information during model development and then providing explanations through LLMs while keeping humans in the workflow using the appropriate prompts. ## 4. Evaluating LLM-Generated Explanations In the previous section 3, the survey discussed several explanation techniques using LLMs. However, evaluating the aforementioned explanation techniques to capture the model’s reasoning processes remains a significant challenge (Zhao et al., 2024). Evaluation can be divided into two categories: 1) qualitative and 2) quantitative evaluation as shown in Table 2. Qualitative evaluation focuses on how easy the explanations are to understand and read, emphasizing aspects, such as 1) comprehensibility and human understanding, and 2) controllability. On the other hand, quantitative evaluation, looks at things, such as relevant, and efficient explanations, such as 1) faithfulness and 2) plausibility. Comprehensibility and human understanding focus on the ease with which the explanation can be understood and the clarity with which it conveys the model’s reasoning to humans (Lim et al., 2019; Sokol and Vogt, 2024). For example, if an LLM explains the prediction by highlighting certain words in the sentence, the explanation should be easy for a human to follow. For example, in the case of sentiment analysis, a comprehensible explanation would read something, such as: ”Review was classified as positive mainly for three key phrases: ’masterful direction’ indicates the high quality of filmmaking, ’compelling performances’ indicate that the acting was great, and ’innovative storytelling’ indicates creative merit. Such phrases occur frequently in positive movie reviews within our training data. The explanation is comprehensible because it breaks down the model’s reasoning into distinct components, links each to the corresponding evidence, and connects distinct components to commonly understood concepts in film criticism. Controllability is another important qualitative evaluation aspect that refers to the interactivity and adjustability of explanations (Cao, 2023). For example, enabling a user to set the focus of an explanation, such as highlighting select parts of the model’s decision process that are interesting or of interest to users, would increase the controllability of the explanation. In LLM-generated explanations, controllability allows users to provide feedback to improve the explanation. For example, a user identifies the unclear parts of the model and ask the LLM to improve its response. One way to test how well this works is through user studies, where participants give feedback on explanations generated by the model, such as highlighting sections that don’t make sense or aren’t relevant. Based on this feedback, the model’s ability to adjust its explanations and how satisfied users are with the updated versions would be used to measure its flexibility/controllability. A few works study this process iteratively, providing feedback and improving explanation quality by measuring the effects of human input on explanations (Nauta et al., 2023; Dong et al., 2017). Some researchers argue that explanations that look understandable to humans might not be accurate or true to the model’s actual logic (Jacovi and Goldberg, 2020). For example, an explanation might seem accurate but could still be wrong because the model was trained on flawed data, leading to incorrect relationships (Nauta et al., 2023). Researchers stress the importance of using more objective, quantitative methods to evaluate explanations to address the issue of misleading explanations. For example, checking whether an explanation is truly faithful to the model, rather than just looking reasonable, can help prevent mistakes. Research (Burton et al., 2023) focuses on the quantitative evaluation of explanations generated by large language models (LLM), by assessing aspects: 1) faithfulness and 2) plausibility. Faithfulness is the degree of accuracy with which an explanation represents the model’s actual decision-making process (Kadir et al., 2023). A faithful explanation should align with at least the following: 1) internal logic and cause-and-effect relationship established by the model; 2) how the model has been designed and the data the model has been trained on; and 3) whether the decision of the model is reproducible using features represented in the explanation. A faithful explanation in a medical diagnosis system, for instance, clearly states the features in the patient that help in model decisions, such as his or her age, symptoms, test results, previous case history, and the likes. A faithful explanation shows the transparent cause-and-effect relationships regarding how particular symptoms or positive test results add to the diagnosis. It should be consistent with what the model has learned; that is, it uses the same patterns as picked up during training, such as known associations between symptoms and conditions. Confidence scores or weights on each factor are meant to give an idea about the influence each feature in the decision-making process resulted in the diagnosis and further to demonstrate that the given decision can be reproduced using these explained features (Burton et al., 2023). Plausibility, by contrast, evaluates the logical coherence and domain consistency of explanations (Nauta et al., 2023). The plausible involves 1) aligning explanations with domain knowledge, 2) ensuring logical consistency, 3) verifying causal relationships, and avoiding contradictions or impossible claims. For instance, a plausible explanation from a climate prediction model would be based on established meteorological principles and valid cause-effect relationships (Jacovi and Goldberg, 2020). The difference between faithfulness and plausibility is important: an explanation can be plausible, it makes logical sense to the human not faithful, and it does not represent the model’s internal reasoning. For example, a cat prediction model might predict the presence of a cat in an image because it detected whiskers and a tail. However, the prediction may be based on unrelated features, such as texture patterns. Table 2. Comprehensive Evaluation for LLM Explanations | Qualitative | Comprehensibility and human understanding (Lim et al., 2019; Sokol and Vogt, 2024) | Ease of understanding and clarity in delivering the model’s reasoning to humans | Explaining sentiment analysis by highlighting key phrases, such as ”masterful direction” and ”innovative storytelling” | | --- | --- | --- | --- | | Controllability (Cao, 2023; Chen et al., 2021) | Interactivity and adjustability of explanations, enabling users to provide feedback to refine explanations | Users highlighting unclear parts of the explanation for improvement | | | Quantitative | Faithfulness (Burton et al., 2023) | Accuracy in representing the model’s internal decision-making process, including causal relationships and alignment with training data | A medical diagnosis system showing specific patient features and their weights that informed the decision | | Plausibility (Jacovi and Goldberg, 2020; Nauta et al., 2023) | Logical coherence and domain consistency, ensuring alignment with established knowledge | A climate prediction model providing explanations consistent with meteorological principles | | ## 5. Benchmark Datasets for Explainable AI with LLMs Researchers need special data to test these explanation generation features of LLMs and make them better (Zytek et al., 2024a). Such datasets can be used for training LLMs to explain AI model’s decision by providing structured sentences (Liu et al., 2024a). The goal is to generate an explanation that is clear, transparent, and easy to understand for humans. Benchmark datasets include examples, such as human-written explanations, steps of reasoning of prediction, and structured knowledge that aligns with factual information (Chen et al., 2024). These examples enable AI systems to become skilled in explaining their thought processes. Benchmark datasets are important because they give researchers a standard way to train and measure the quality of explanations produced by LLMs. For example, datasets allow researchers to measure whether the explanations are accurate, complete, easy to understand, and consistent across similar situations. By studying these datasets, researchers can better understand how to make AI systems that are not only clear but also more transparent and user-friendly. Each dataset has been designed to focus on specific aspects of explanation, such as clear reasoning, scientific knowledge (detailed annotations and structured reasoning examples), or ethical decision-making (containing scenarios with diverse ethical considerations).As shown in Table 3, the datasets include 1) e-SNLI (Enhanced Stanford Natural Language Inference) (Camburu et al., 2018b), 2) CoS-E (Common Sense Explanations) (Rajani et al., 2019), 3) ECQA (Explanation CommonsenseQA) (Aggarwal et al., 2021), 4) WorldTree (Jansen et al., 2018), 5) OpenBookQA with Explanations (Mihaylov et al., 2018), 6) XplainLLM (Chen et al., 2024), 7) RAGBench (Friel et al., 2024), and 8) HateXplain (Mathew et al., 2021). The overview of each dataset is given below. First we have the e-SNLI (Enhanced Stanford Natural Language Inference) dataset (Camburu et al., 2018b), which is an extension of the Stanford Natural Language Inference dataset that includes sentence pairs annotated with entailment labels (e.g., ”entailment,” ”contradiction,” or ”neutral”) and human written explanations. For example, if the first statement is ”A dog is running through the park” and the second statement is ”An animal is outside,” the two statements match, then the label is ”entailment,” and the explanation is ”A dog is a type of animal, and running through a park implies being outside.” The second dataset is the CoS-E (Common Sense Explanations) dataset (Rajani et al., 2019), which provides the multichoice common-sense questions with steps with human written explanations, to enable the model to generate everyday reasoning. For example, a question in the dataset might ask, ”Why would someone wear sunglasses?” with multiple-choice answers, such as ”To see better at night,” ”To block the sun,” or ”To look fashionable.” The correct answer, ”To block the sun,” is accompanied by a detailed explanation: ”Sunglasses are designed to protect the eyes from bright sunlight.” The third dataset is ECQA (Explanation CommonsenseQA) dataset (Aggarwal et al., 2021), which extends the CommonsenseQA dataset (Talmor et al., 2018) by adding the detailed explanation for the answers to the commonsense reasoning questions. For example, a question, such as ”Why would someone lock their car?” with answer choices such as ”1) To prevent theft,” ”2) To keep it clean,” and ”3) To make it lighter” is combined with the correct answer, ”To prevent theft.” ECQA provides detailed explanations with positive reasons, such as ”Locking the car secures the doors and prevents unauthorized access” and negative reasons, such as ”Leaving the car unlocked makes it easier for thieves to steal belongings.” These reasons help models identify the correct answer, explain it clearly, and prevent incorrect options, to improve the reasoning and explanatory abilities of models. Additionally, WorldTree dataset (Jansen et al., 2018), organizes scientific reasoning questions into structured explanation graphs to train the large language models for step-by-step scientific reasoning. For example, for the question, ”Why does a metal spoon feel hotter than a plastic spoon in hot water?” with the correct answer, ”Because metal is a better conductor of heat,” WorldTree dataset provides an explanation graph linking facts like ”Metal conducts heat efficiently,” ”Conductors transfer heat more effectively,” and ”Plastic is a poor conductor.” These explanation graphs combine domain knowledge and logical steps to help the models articulate clear and scientific reasoning processes. Similarly, OpenBookQA (Mihaylov et al., 2018) dataset improves elementary science questions with the help of knowledge-based explanations. For example, the question ”Which of these would let the most heat travel through?” the correct answer is ”a steel spoon in a cafeteria”. The answer is explained using the scientific fact ”Metal is a thermal conductor” and the common knowledge ”Steel is made of metal” and ”Heat travels through a thermal conductor.” Training the LLMs on this openbookqa dataset can enhance the ability to explain the relation between the science questions and answers. XplainLLM dataset (Chen et al., 2024) combine the question-answers with the knowledge graphs in the form of triplets. These triplets ensure transparency by using factual grounded reasoning in the knowledge graphs. For example, for the question ”The people danced to the music, what was the music like for them?” with options such as ”soothing,” ”vocal or instrumental,” ”loud,” ”universal,” and ”enjoyable,” the correct answer is ”enjoyable.” XplainLLM provides explanations grounded in knowledge graphs, detailing why ”enjoyable” is the correct choice while justifying why the other options are less suitable. Training the LLM on this dataset enhances transparency and reliability by linking the LLM’s reasoning to structured and factual knowledge. The other dataset is the RAGBench (Friel et al., 2024), which is a domain-specific dataset for using Retrieval-Augmented Generation (RAG) systems with a focus on explainability. Each sample in the RAGBench includes a query, the retrieved evidence, and the generated response using a query. For instance, in a technical support domain, a query such as ”How do I reset my device?” would be paired with relevant text from a user manual with the details of the reset process. This structure allows RAG systems to generate accurate and contextually accurate responses, with the retrieved evidence providing transparency into the model’s decision-making process. Training on RAGBench can enhance the ability of the model to produce explainable and reliable outputs in domain-specific applications. The last dataset is the HateXplain dataset (Mathew et al., 2021) provides explanation in the hate speech detection with annotations for an unbiased generation. Each sample of the HateXplain dataset includes hate speech classification and a rationale explaining the classification. For example, a post labeled as ”hatespeech” might target a specific community, with annotators highlighting specific words or phrases that justify this classification. The detailed annotation aids in understanding the model’s decision-making process and helps in reducing unintended bias towards target communities. Table 3. Overview of Datasets for LLM-Based Explainability | e-SNLI (Camburu et al., 2018b) | Human-written explanations for tasks to find logical relation. | Sentence and label pairs, such as ”entailment,” ”contradiction,” or ”neutral,” e.g., ”A dog is a type of animal, and running through a park outside.” | | --- | --- | --- | | CoS-E (Rajani et al., 2019) | Multichoice common-sense reasoning with explanations. | Example: ”Why wear sunglasses?”, pair with the answer: ”To block the sun,” with explanation: ”Sunglasses protect eyes from bright sunlight.” | | ECQA (Aggarwal et al., 2021) | Commonsense reasoning with detailed explanations. | ”Why lock a car?” with answer: ”To prevent theft,” with explanation: ”Locks secure doors, preventing unauthorized access.” | | WorldTree (Jansen et al., 2018) | Scientific reasoning using structured explanation graphs. | Example: ”Why does a metal spoon feel hotter than plastic in hot water?” with the answer: ”Metal conducts heat efficiently,” and explanation: Graph with domain facts, such as ”Metal is a conductor.” | | OpenBookQA (Mihaylov et al., 2018) | Elementary science questions with knowledge-based explanations. | Example: ”Which lets heat travel best?” with answer: ”Steel spoon,” and explanation: ”Metal is a thermal conductor, and steel is made of metal.” | | XplainLLM (Chen et al., 2024) | Factual reasoning with knowledge graphs in triplet form. | Example: ”Why did people dance to music?” with answer: ”Enjoyable,” and explanation: Knowledge graph linking ”enjoyable” with relevant reasoning. | | RAGBench (Friel et al., 2024) | Retrieval-Augmented Generation with explainability for specific domains. | Example: ”How to reset a device?” with explanation: Retrieved evidence from a user manual supports the generated response. | | HateXplain (Mathew et al., 2021) | Hate speech detection with unbiased explanations. | Example: A flagged post with highlighted words or phrases explaining the ”hate speech” classification. | ## 6. Real-World Applications of LLM Explainability LLMs are playing a vital role in enhancing explainability and confidence in AI-driven decisions across a wide range of domains, ranging from healthcare to finance. Applications in healthcare, for instance, include the use of LLM in explainability to trust AI-driven medical decisions and predictions (Attai et al., 2024; Dave et al., 2024; Umerenkov et al., 2023a). Suppose, for instance, that a doctor makes a diagnosis using an AI system. An LLM can break down the relationship between fever and chronic cough that leads to pneumonia, enable the doctor to consider the AI’s explanation and make treatment decisions based on it. In finance, LLMs contribute to explainability in complex decision-making processes, such as in credit risk assessment or fraud detection. For example, if an AI system flags a transaction as potentially fraudulent, an LLM can give a detailed breakdown of the various factors influencing this decision, such as unusual amounts of transactions, atypical locations, or patterns inconsistent with the user’s historical behavior (Feng et al., 2023a). In this way, explainability allows financial institutions to review the flagged cases thoroughly and ensures that decisions are made according to regulatory standards and customer trust. In healthcare, LLMs improve the process of drug discovery (Zheng et al., 2024; Zhavoronkov et al., 2019) by explaining the interactions of molecules at chemical and biological levels (Liu et al., 2024b). For example, in antibiotic development, LLMs analyze how a drug molecule binds to bacterial enzymes, and explain why certain compounds effectively kill bacteria while others do not. The explanation generated by LLMs ensures that scientists understand the combination of specific molecules to develop a drug and its impact on the human body. In drug discovery, LLMs help identify the molecule that works best against a disease (Blanchard et al., 2022). For example, in the design of new antiviral medications, LLMs process vast biomedical datasets to determine which molecules can bind to a virus’s proteins and, therefore, block its ability to replicate. By breaking down these interactions step by step, LLMs help select the most promising drug candidates for further testing. In addition to the design of new antiviral medications, LLM-generated explanations also help predict the effects of drugs on the human body (Chaves et al., 2024). For instance, if a new drug for heart disease is developed, an LLM can explain how its molecular structure will interact with heart cells and whether it may cause side effects such as high blood pressure. This transparency allows researchers to make informed decisions early in the drug development process, saving time and resources. In finance, explainability builds transparency into credit scoring systems and fraud detection (Luz, 2024; Feng et al., 2023a). An LLM might point out the load application refusal and potential factors behind it, such as a low credit score, high debt-to-income ratio, or missed payments. Such transparency not only increases trust among customers but also aids users in taking correct measures during loan applications. Similarly, in predicting currency trends, LLMs improve explainability in AI models by sentiment analysis of news articles and reports. LLMs analyze and explain particular sentiments, such as general optimism about the performance of an economy or concern over geopolitical instability, that influence the prediction of currency trends. Then, the historical price data is combined with trading patterns for clear and accurate forecasts of the currency movement; therefore, financial analysts use it to make better-informed decisions (Limonad et al., 2024). In legal applications, LLMs enhance the interpretability of AI-driven legal analysis and document review systems (Billi et al., 2023; Thanh and Satoh, 2024). When analyzing contracts or legal precedents, LLMs can clearly express the reasoning behind flagging specific clauses as potentially problematic or explain the reason behind the relevancy of certain cases to current litigation. For example, during a contract review by the lawyer, an LLM can explain why a particular compensation clause is risky by underlining that it is too broad compared to industry norms. Explainability provides lawyers with better decision-making and raises confidence in AI-assisted legal research. Manufacturing and industrial applications benefit significantly from LLM-enhanced explainability in predictive maintenance systems (Trivedi et al., 2024; Garcia et al., 2024). For example, suppose an AI system predicts an imminent failure of a machine. In that case, the LLM can explain how certain vibration patterns, temperature readings, and other operational data contribute to this prediction, and enable the maintenance team to make informed decisions about preventive actions. In educational technology, Explainability using LLMs help in personalized learning recommendation. (Laak and Aru, 2024; Ng and Fung, 2024). For example, when a high school student struggles with algebra, LLMs such as GPT-4 and Gemini design a custom study plan using the explanation based on the learning preferences and weaknesses of students. The learning preference includes simple explanations, practice problems, and quizzes that adjust based on the student’s progress to make learning easier and more personalized. Additionally, generating explanations using LLMs makes the learning process more understandable and interesting. Researchers (Abu-Rasheed et al., 2023) explain that large language models provide graph-based explanations to organize the whole syllabus for students, and allow students to follow each step systematically for better understanding. With these organized step-by-step explanations, students will know why they study particular topics, thus making them interested. In urban planning and smart city applications, LLM explainability helps break down complex urban questions into smaller, manageable factors and provide detailed, clear answers. Take the case of traffic congestion arising in the city’s downtown area. As described in (Jiang et al., 2024a) explanation, using LLMs breaks down traffic congestion issues into contributing factors by factors such as traffic flow patterns, road capacities, and peak travel times. Then, the language model provides detailed answers, such as optimizing traffic lights, creating bike lanes, or extending public transport routes, thus making the whole urban planning process more effective and efficient. ## 7. Challenges and Limitations related to Explainable AI (XAI) Explainable AI (XAI) aims to improve AI-based decision-making processes by building user trust with fairness, accountability, and transparency. Significant challenges exist as shown in Figure 3, especially within large language models (LLMs) for XAI, due to issues including 1) sensitive data handling, 2) accommodating societal diversity and norms, 3) the use of multisource data and algorithms, 4) complexity of AI model and 5) bias and fairness within LLMs ### 7.1. Sensitive Data One of the key challenges in XAI is the limited access to sensitive data (e.g., personal or private information) required to explain certain decisions, especially in post-hoc approaches (Section 3.1) and human-centered approaches (Section 3.3). Sensitive data include personal or private information, such as medical histories, financial transactions, or social media activity. For example, suppose that an AI model predicts that a patient in the ICU is at high risk of mortality due to sepsis (a severe and life-threatening condition caused by the body’s response to infection that damages its tissues and organs) (Singer et al., 2016). In the case of predicting diseases, such as sepsis, the decision may require disclosing specific details from the patient’s medical record, including the progression of vital signs, biomarkers (such as lactate levels), or infection history. While the disclosure of such records is necessary for clinical decision-making, it could risk the patient’s privacy. Robust security systems are required to protect sensitive data from potential risks (Farayola et al., 2024), such as unauthorized access and data breaches. Failure to provide security systems would have serious repercussions (Kuźniacki et al., 2022), such as legal penalties, loss of user trust, financial losses, and reputational damage. XAI systems must establish clear guidelines on data handling, such as implementing data encryption, anonymizing sensitive information, and restricting access through authentication protocols (Jaff et al., 2024; Yao et al., 2024), to ensure reliability with privacy standards. XAI can help detect and mitigate security vulnerabilities, ensure that sensitive data is handled responsibly, and minimize risks such as unauthorized access to data by making operations of AI models transparent as discussed in section 3.2. ### 7.2. Societal Diversity and Norms Another significant challenge is accommodating societal diversity and varying norms in models for the XAI. Diversity varies across and within different groups (Anik and Bunt, 2024), so it is important to design AI models that respect and reflect differences in diversity. For example, in medical science, an individual of African descent is more susceptible to sickle cell anemia due to genetic reasons (Williams, 2016), whereas an individual of European descent is more prone to cystic fibrosis (Cardoso et al., 2004). An AI system that predicts disease risk would need to take these demographic-dependent differences into account to ensure the results will be accurate and fair. Similarly, societal norms vary across cultures and communities (Gerston, 2010), so it is important to design AI models that carefully give attention to culturally specific norms and expectations to remove unintentional confusion or harm (Peters and Carman, 2024). For example, in the system of content moderation, what is considered appropriate or offensive can vary greatly between cultures. For example, the ”thumbs up” gesture means ”good job” or ”I agree” in many Western countries. But in some Middle Eastern countries, it can be seen as rude or insulting. The above example highlights the need to understand and respect cultural differences in XAI systems. ### 7.3. Multi-Source Data XAI also faces the challenge of combining multiple data sources. The final explanation may be skewed because AI systems use various types of data, such as social media activity, financial transactions, and online behavior, and each type of data provides a different perspective (de Bruijn et al., 2022). Using multi-source data and giving less importance than the required for optimal performance can distort the results (Masyutin, 2015). For example, in a credit scoring system (a method used to evaluate someone’s creditworthiness), if social media data, such as tweets, are given more importance than financial transaction history, the system might overestimate and underestimate the applicant’s creditworthiness (Day, 2022). Misalignment occurs because social media data may lack the accuracy or direct relevance of financial transaction history in predicting credit behaviors. Consequently, the resulting credit score fails to predict and explain the financial stability of the applicant. In such cases, the real information needed to predict the score should come only from financial transaction data, making the use of social media data unnecessary or misleading. A similar issue occurs in stock prediction models, where predictions may be influenced by Twitter posts about rumors, even when the company’s financial data suggest that the stock is stable. <details> <summary>x3.png Details</summary>  ### Visual Description ## Diagram: Challenges and Limitations in AI Systems ### Overview The image is a hierarchical flowchart or tree diagram illustrating the primary "Challenges and Limitations" associated with a complex system, likely related to Artificial Intelligence (AI) or Large Language Models (LLMs). The diagram uses a top-down structure with one central parent node branching into five distinct child nodes. ### Components/Axes * **Main Node (Parent):** A single, light blue rectangular box positioned at the top-center of the diagram. It contains the title text: **"Challenges and Limitations"**. * **Sub-Nodes (Children):** Five orange rectangular boxes arranged in a horizontal row below the main node. Each represents a specific challenge. * **Connectors:** Five black arrows originate from the bottom edge of the main blue box. Each arrow points downward to one of the orange sub-node boxes, indicating a direct, one-to-many relationship. The arrows are evenly spaced, creating a symmetrical fan-out pattern. ### Detailed Analysis The diagram explicitly lists five key challenges. The text within each component is transcribed below: 1. **Main Node (Top-Center):** * Text: `Challenges and Limitations` 2. **Sub-Nodes (Left to Right):** * **Node 1 (Far Left):** `Data Sensitivity Handling` * **Node 2:** `Variability of Societal Norms` * **Node 3 (Center):** `Multi-Source Data` * **Node 4:** `Complexity of AI Model` * **Node 5 (Far Right):** `Bias in LLMs` ### Key Observations * **Structure:** The diagram employs a simple, clear hierarchical tree structure to categorize challenges under a single umbrella concept. * **Visual Coding:** Color is used to distinguish hierarchy: light blue for the overarching category and orange for the specific items within that category. * **Layout:** The arrangement is symmetrical and balanced, with the central node ("Multi-Source Data") aligned directly beneath the main title box. * **Content Scope:** The challenges span technical, data-related, and socio-ethical domains, indicating a holistic view of the problem space. ### Interpretation This diagram serves as a high-level taxonomy, breaking down the broad, abstract concept of "Challenges and Limitations" into five concrete, manageable areas of concern for AI development and deployment. * **Relationship of Elements:** The five sub-nodes are presented as co-equal components of the main challenge. They are not shown as interdependent or sequential, but as distinct facets that must be addressed individually. The arrows signify that each is a direct sub-category of the parent topic. * **Implied Context:** The specific mention of "LLMs" (Large Language Models) in the final node strongly suggests the diagram's context is focused on the challenges inherent in modern generative AI systems. The other nodes support this: "Data Sensitivity" and "Multi-Source Data" relate to training data issues; "Complexity of AI Model" addresses technical opacity; and "Variability of Societal Norms" highlights the difficulty of aligning AI behavior with diverse human values. * **Notable Absence:** The diagram is purely categorical and does not provide data, metrics, or prioritization (e.g., which challenge is most severe). It functions as a conceptual map or agenda-setter for discussion rather than an analytical chart. * **Underlying Message:** The structure implies that overcoming the limitations of advanced AI is not a single problem but a multi-front effort requiring solutions across data governance, model architecture, ethical alignment, and technical interpretability. </details> Figure 3. Summary of the challenges of XAI ### 7.4. Complexities of AI Model Understanding the complexity of AI is very essential for the improvement in the explainability of the predictions (Gil et al., 2021) and fostering trust in the decision-making process of these AI models, both at the theoretical and application level. At a theoretical level, XAI emphasizes the importance of breaking down the decision-making process into simpler and more understandable parts. For example, in a neural network with five layers, the first layer might focus on recognizing basic features, such as edges and textures in an image. The second layer then combines simple features to detect more complex patterns, such as shapes and parts of objects. The difficulty is understanding how the decisions made in the first layer influence the decisions made in the second layer, and how each subsequent layer builds on the previous one. To fully understand the model, it’s essential to see how each layer works together to make the final decision or prediction (Räuker et al., 2023). The complexity of the AI model (Umer et al., 2023; Mohsin, 2023) poses challenges not only at the theoretical level but also in real-world applications. Designing user interfaces that help people interact with and understand complex AI systems becomes an additional challenge (Nguyen et al., 2024). For example, interfaces can be implemented at different levels of the model. At the first layer, the interface could focus on explaining low-level features, such as edges or textures. As the model progresses through the layers, the interface would show how these low-level features are combined into more complex patterns, such as shapes and object parts, in intermediate layers. In the final layer, the interface would explain how these combined features lead to the model’s final decision or prediction. For the application level, it is important that interfaces cater to different users based on their level of expertise, autonomy, and trust in the AI system. For example, a beginner might need a simpler explanation of the basic features, while an expert might want more in-depth insights into how the model combines information across all layers. By addressing the complexity of AI model, we can provide clear, effective explanations tailored to each user’s needs (Lim and Dey, 2009) Furthermore, after designing the interface, deploying complex AI models for predictions and utilizing LLMs for their explainability can be challenging. There are two main reasons for this: one is the limited power of the computers, such as GPUs needed to run complex AI models, and the other is the restrictions on smart devices, such as smartphones, which have less storage and processing ability. (IBM, 2024). For the first issue, computers may not have enough power or the right tools to run heavy AI models. The tools to run a heavy AI model can be expensive, and getting access to better computers might not always be possible. For the second issue, if edge devices, such as smartphones are used to reduce costs, the devices themselves often lack the storage space or processing power to handle large models locally. Techniques, such as Knowledge Distillation, have been developed to overcome the issue of limited processing and computation ability. Knowledge Distillation involves transferring knowledge from a large model (e.g., BERT, which is a large and powerful language model used for understanding text) to a smaller model (e.g., DistilBERT, a smaller version of BERT that performs similarly but requires less power and resources). In Knowledge Distillation, the smaller model (DistilBERT) learns to mimic the performance of the larger model (BERT) while being more efficient. The same method, as highlighted by (Cantini et al., 2024), reduces the computational constraint of AI models, making them more suitable for resources-constrained environments, including XAI applications. ### 7.5. Bias and Fairness within LLMs Another limitation is the bias and fairness issues presented within LLMs (Huang et al., 2023). Addressing the biases and fairness in large language models (LLMs) is important to ensure that the outputs of AI model are equitable and accessible for diverse users with different contexts. Biases include 1) social, 2) language, and 3) representation biases in the responses. Social biases are often related to characteristics, such as age, gender, and lifestyle, impacting individuals across different social groups unequally, as noted in (Huang et al., 2023) and (Caliskan, 2023). For instance, it might recommend men for leadership positions, such as managers and executives more frequently than women, even when both have similar qualifications. Social bias stems from historical data that overrepresents men in such roles, perpetuating stereotypes and unfairly disadvantaging female candidates. Language biases emerge when LLMs treat different languages unequally, creating social disparities in areas, such as communication and education (Mohsin et al., 2025a; Fang et al., 2023). For instance, the model might give better and more accurate responses in English than in less commonly spoken languages, which could make its outputs less useful or accessible to non-English speakers. Biases in LLMs-generated content, such as recommendation letters and news articles, occur due to language biases. Language biases can even affect translation tasks, potentially leading to misrepresentation or inaccessibility for different language groups. Representation biases occur when the data used to train the model lack balance, such as underrepresenting certain communities, cultures, and viewpoints. The skewed data can make the model generalize unfairly or completely overlook specific groups. For example, suppose an LLM is tasked with explaining a decision related to job recruitment but has been trained on data that overrepresent male candidates in leadership roles. In that case, its explanations might unintentionally favor male applicants or use language that perpetuates gender stereotypes. Biases in LLMs would increase the challenges in using LLMs for model explainability. For example, when explaining a recruitment decision, social biases might lead to stereotyped justifications favoring male candidates, language biases could make the explanation less comprehensible in non-English contexts, and representation biases might exclude perspectives of underrepresented groups, resulting in unfair and inaccessible explanations. Recent studies (Wang et al., 2023) reveal that even state-of-the-art LLMs, such as ChatGPT show unfairness in critical fields, such as education and healthcare. Researchers have proposed different methods, such as in-context learning (Fei et al., 2023) and refining prompt (Kamruzzaman and Kim, 2024; Memon et al., 2024b), for the problems of biases and fairness (Hua et al., 2024). For example, if an AI is asked about a sensitive topic, providing some background about the topic can help the model respond more thoughtfully. Similarly, researchers might avoid asking a question about a person’s age, gender, or ethnicity, or they might rephrase a question to make sure the answer doesn’t favor one group over another. <details> <summary>x4.png Details</summary>  ### Visual Description ## Horizontal Bar Chart: Salience Scores by AI Model and Parameter Size ### Overview This image is a horizontal bar chart comparing the "Salience Large" and "Salience Small" scores for six different large language models (LLMs). The chart visually contrasts the performance of each model across these two metrics, with models listed vertically and their corresponding scores represented by horizontal bars. ### Components/Axes * **Chart Type:** Horizontal grouped bar chart. * **Y-Axis (Vertical):** Lists six AI models, each followed by its approximate parameter count in parentheses. From top to bottom: 1. ChatGPT (175B) 2. Llamav2 (70B) 3. Cohere (54B) 4. Vicuna (13B) 5. Mistral (7B) 6. Olmo (7B) * **X-Axis (Horizontal):** Represents the numerical salience score. The axis is not explicitly labeled with a title or scale markers, but the score values are printed directly at the end of each bar. * **Legend:** Positioned at the top center of the chart. * A blue square is labeled **"Salience Large"**. * An orange square is labeled **"Salience Small"**. * **Data Series:** Two bars are plotted for each model: * **Blue Bar:** Represents the "Salience Large" score. * **Orange Bar:** Represents the "Salience Small" score. ### Detailed Analysis The chart presents the following specific data points for each model (Salience Large / Salience Small): 1. **ChatGPT (175B):** * Salience Large (Blue): 0.84 * Salience Small (Orange): 0.56 * *Trend:* The blue bar is significantly longer than the orange bar. 2. **Llamav2 (70B):** * Salience Large (Blue): 0.75 * Salience Small (Orange): 0.53 * *Trend:* The blue bar is longer than the orange bar. 3. **Cohere (54B):** * Salience Large (Blue): 0.71 * Salience Small (Orange): 0.56 * *Trend:* The blue bar is longer than the orange bar. Notably, its "Salience Small" score (0.56) is equal to ChatGPT's. 4. **Vicuna (13B):** * Salience Large (Blue): 0.57 * Salience Small (Orange): 0.51 * *Trend:* The blue bar is slightly longer than the orange bar. 5. **Mistral (7B):** * Salience Large (Blue): 0.68 * Salience Small (Orange): 0.50 * *Trend:* The blue bar is longer than the orange bar. Its "Salience Large" score is higher than the larger Vicuna model. 6. **Olmo (7B):** * Salience Large (Blue): 0.45 * Salience Small (Orange): 0.29 * *Trend:* The blue bar is longer than the orange bar. This model has the lowest scores in both categories. ### Key Observations * **Consistent Pattern:** For every model listed, the "Salience Large" score (blue bar) is higher than the "Salience Small" score (orange bar). * **Top Performer:** ChatGPT (175B) achieves the highest score in both categories (0.84 Large, 0.56 Small). * **Lowest Performer:** Olmo (7B) has the lowest scores in both categories (0.45 Large, 0.29 Small). * **Parameter Size vs. Performance:** While the largest model (ChatGPT) performs best, the relationship is not perfectly linear. For example, Mistral (7B) outperforms the larger Vicuna (13B) on the "Salience Large" metric (0.68 vs. 0.57). * **Score Equality:** Cohere (54B) and ChatGPT (175B) share the same "Salience Small" score of 0.56. ### Interpretation This chart likely evaluates how well different LLMs identify or generate salient (important, relevant) information, with "Large" and "Small" possibly referring to the scale or granularity of the salience task (e.g., document-level vs. sentence-level importance). The data suggests a general correlation between larger model size (parameter count) and higher salience scores, but with notable exceptions. The consistent gap between "Large" and "Small" scores for each model indicates that the task measured by "Salience Large" yields systematically higher scores across all models, or that models are better at the "Large" variant of the task. The performance of Mistral (7B) is particularly interesting, as it surpasses the larger Vicuna (13B) on the "Salience Large" metric, suggesting that factors beyond raw parameter count—such as training data, architecture, or fine-tuning—significantly impact this specific capability. The chart serves as a comparative benchmark, highlighting that model size alone is not the sole determinant of performance on salience-related tasks. </details> Figure 4. Saliency Comparison of Models by (Koo et al., 2024) ## 8. Feature Importance Across LLMs used for Explanation Using Saliency Maps The comparison of saliency values (see Figure 4 across various large language models (LLMs) highlights their emphasis on critical features, a key aspect of explainable AI. Saliency techniques help us identify the parts of a generated explanation, such as certain words and phrases that are most aligned with the ground truth (Cerekci et al., 2024). The saliency techniques typically involve 1) gradient-based methods (which measure the change in the output by a small change in the input by identifying important features in the input), and 2) attention mechanisms (which allocate focus on the parts of the input that are most relevant to the model). The saliency techniques help to measure the importance of the features, such as specific work or phrases in the explanation of the decision or response generated by the large language models. Highlighting the important parts of the word or phrase offers a clearer view of the prioritization of features by different large language models during the explanation generation and the alignment of the generated response by the ground truth. For example, findings by (Koo et al., 2024), as shown in Figure 4, provide saliency scores for each LLM under two conditions: one with a larger and one with a smaller input context. This is to ensure that LLMs maintain a consistent focus on important features regardless of the length of the provided input context. In this, ChatGPT (175 Billion parameters) (Achiam et al., 2023) has the highest saliency values, with 0.84 for large inputs and 0.56 for small inputs, indicating its strong ability to identify and prioritize important features. Similarly, LLAMAV2 (70B) (Touvron et al., 2023) and COHERE (54B) (Almazrouei et al., 2023) also show significant saliency values (0.75 and 0.71 for large inputs, respectively), highlighting their effectiveness in recognizing important elements. In contrast, models, such as VICUNA (13B) (Chiang et al., 2023) and MISTRAL (7B) (Jiang et al., 2023) display lower saliency values, suggesting they place less emphasis on critical features, especially in smaller input contexts. OLMO (1B) (Groeneveld et al., 2024) has the lowest saliency scores (0.45 for large inputs and 0.29 for small inputs). The above result is presented by (Koo et al., 2024), which indicates that the disparity among models emphasizes the importance of size, architecture, and training strategies in saliency performance. <details> <summary>x5.png Details</summary>  ### Visual Description ## Diagram: Future Research Directions Flowchart ### Overview The image displays a simple, hierarchical flowchart diagram illustrating three proposed future research directions stemming from a central theme. The diagram uses a top-down structure with one primary category branching into three distinct sub-categories. ### Components/Axes * **Central Node (Top):** A single, light blue rectangular box with rounded corners. It contains the text "Future Directions". * **Sub-Nodes (Bottom):** Three orange rectangular boxes with rounded corners, arranged horizontally in a row below the central node. * **Connectors:** Three solid black arrows originate from the bottom edge of the central "Future Directions" box. Each arrow points directly downward to one of the three sub-node boxes, indicating a direct relationship or derivation. ### Detailed Analysis / Content Details * **Text Transcription:** * Central Node: "Future Directions" * Left Sub-Node: "Automate with human feedback" * Middle Sub-Node: "Visual-Text Integration" * Right Sub-Node: "Cross-Disciplinary Research" * **Spatial Grounding & Layout:** * The "Future Directions" box is centered at the top of the diagram. * The three sub-node boxes are positioned in a horizontal line, centered beneath the main box. They are evenly spaced from left to right. * The left arrow connects to the "Automate with human feedback" box. * The middle arrow connects to the "Visual-Text Integration" box. * The right arrow connects to the "Cross-Disciplinary Research" box. * **Visual Design:** The diagram uses a clean, minimal aesthetic with a two-color scheme (blue for the parent category, orange for child categories) to visually distinguish hierarchy. All boxes have a subtle drop shadow, giving a slight sense of depth. ### Key Observations * The structure is strictly hierarchical and non-cyclical, showing a one-to-many relationship. * The three future directions are presented as parallel, equally weighted concepts, as indicated by their identical styling and horizontal alignment. * The language is concise and conceptual, suggesting these are high-level strategic goals rather than detailed technical plans. ### Interpretation This diagram outlines a strategic framework for future development or research. The central theme, "Future Directions," is decomposed into three complementary pillars: 1. **Automate with human feedback:** This suggests a focus on developing automated systems (likely AI/ML models) that are not fully autonomous but are instead refined and guided by continuous human input. This approach prioritizes safety, accuracy, and alignment with human judgment. 2. **Visual-Text Integration:** This points to research into multimodal systems that can process, understand, and generate both visual data (images, video) and textual data in a unified manner. This is a key area for advancing AI that can comprehend the world more holistically. 3. **Cross-Disciplinary Research:** This emphasizes the importance of collaboration across different fields of study (e.g., computer science, cognitive science, ethics, domain-specific sciences) to solve complex problems. It implies that the challenges ahead cannot be solved within a single discipline. **Relationship & Synthesis:** The three directions are interrelated. Effective "Visual-Text Integration" often requires "Cross-Disciplinary Research" (e.g., combining computer vision with linguistics). Similarly, building systems that "Automate with human feedback" benefits from cross-disciplinary insights into human-computer interaction and ethics. Together, they form a cohesive strategy for advancing towards more capable, reliable, and socially integrated intelligent systems. The diagram serves as a high-level roadmap, prioritizing human-centric automation, multimodal understanding, and collaborative problem-solving. </details> Figure 5. Future directions for improving explainability in AI using LLMs ## 9. Future Directions In this survey paper, we discussed the importance of making AI more understandable, focusing on how large language models (LLMs) can help achieve this as shown in Figure 5. We also explored the challenges and limitations of using LLMs for explainability, such as the complexity of their decision-making processes. This survey paper also discusses the importance of features, such as specific words or phrases, in the explanations generated by large language models using saliency maps. Future directions will focus on 1) automating feedback, 2) linking text with visuals, and 3) fostering collaboration across different fields to make AI explanations clearer and more helpful. Automating human feedback would enable AI to adapt its explanations based on user interactions, thus saving time and effort. For example, a medical AI would fine-tuning the explanations by learning from feedback received from doctors (Aftab et al., [n. d.]). Similarly, including text-based explanations combined with visuals, such as heatmaps (Selvaraju et al., 2019; Kim et al., 2018) would explain AI decisions, such as showing highlighted areas in a medical image alongside a clear written explanation. Cross-disciplinary collaboration allows experts in AI, cognitive science, and domain-specific regions, such as medicine and law to come together with the goal of instinctive, relevant, and actionable explanations. For example, such collaboration can allow medicine to create AI capable of explaining its diagnostic decisions in a understandable way by doctors and patients, so they could act upon those findings. The above-discussed efforts thus make AI even more practical and trustworthy for a variety of real-world applications. ### 9.1. Automation with Human Feedback for XAI Applications Explainable AI (XAI) involves methods that help users understand and trust the decisions made by AI models, as discussed earlier in section 1. Previous section 3.3 introduces human feedback through surveys and interviews to refine AI-generated explanations, making them easier to understand (Do et al., 2024). However, relying solely on manual feedback collection can be time-consuming and impractical, especially for large-scale AI applications. For example, in a customer support chatbot used by an e-commerce site with millions of users daily. Manually collecting feedback and analyzing all the responses would take too much time and effort. The future direction is to automate human feedback with AI processes to make the system faster and more efficient. For example, AI could track patterns in user interactions with explanations, such as noticing which explanations are useful and where they often ask for clarification. The automated system can use the patterns to quickly adjust the explanations and enable the explanations to match users’ needs without requiring continuous manual assistance from the people. One practical example of an automatic response system is a movie recommendation system (Reddy et al., 2019). Suppose a platform that explains the suggestions of the recommendation system by saying: ’You liked action movies starring Actor X’ If users find this explanation unclear or unhelpful, the system could adapt by providing simpler language or additional details, such as mentioning similar movies. Automation would save time and make AI interactions smoother and more practical. The approach of automating human feedback to XAI systems can be applied across various sectors, such as healthcare and finance, to enhance the practicality and efficiency of AI systems. In healthcare, suppose that an AI tool assists doctors in disease diagnosis (Aftab et al., [n. d.]). The tool explains its decision, such as ”This diagnosis is based on symptoms X, Y, and Z observed in the patient’s history.” If doctors find the explanation lacking information or unclear, they might manually flag it. Over time, an automated feedback system could analyze common feedback patterns and adjust its explanations, such as including references to recent medical research or visualizing patient data trends for clarity. For the finance sector, in a financial advising app, the system might explain investment recommendations, such as ”This stock is suggested due to its high growth rate and low market volatility (market uncertainty)” (Bianchi and Briere, 2021). If users frequently request clarification on terms, such as ”market volatility,” the AI could adapt by adding definitions or visual aids, such as charts showing the stock’s historical performance. In the education sector (Manna and Sett, 2024), an AI tutor helping students with math problems could explain solutions in steps, such as ”Step 1: Combine alike terms. Step 2: Simplify the equation by performing basic operations to make it easier to solve. Step 3: Solve for the unknown by separating the variable.” If many students indicate confusion, the AI could refine its explanation by adding examples, such as ”For instance, 3x + 2x becomes 5x because you add the coefficients.” In this way, AI systems can meet users’ needs in each field. ### 9.2. Integrating Visual and Text Explanations To enhance the explainability of AI models, future efforts could focus on combining textual and visual explanations. For example, when a model predicts a specific outcome, a corresponding text-based explanation can be presented alongside visual aids, such as heatmaps (Lee et al., 2022), to show the parts of the input that influence the decision. This dual modality (textual response with visual explanation) can make it easier for users to understand the model’s reasoning by combining textual explanations with visual aids. The text provides clarity, while visual aids complement it by highlighting key input features that influence the output, making the connection between inputs and outputs more accessible. One approach involves integrating text explanations from LLMs with visualization techniques, such as Grad-CAM (Gradient-weighted Class Activation Mapping) (Selvaraju et al., 2019). For instance, suppose that an AI model diagnoses medical images: Grad-CAM can highlight the regions on an X-ray that contributed to the diagnosis (Lamba and Rani, 2024), while an LLM provides a text explanation, such as, ”The highlighted region shows abnormalities consistent with pneumonia.” This combination helps users understand the diagnosis both visually and contextually. A promising future direction is using knowledge graphs to connect text-based explanations from LLMs with visual representations. Knowledge graphs help organize and display relationships between important pieces of information to make complex AI outputs easier. For example, in an LLM chatbot, a knowledge graph could show how a user’s question is linked to the information retrieval system and the response it generates. ### 9.3. Cross-Disciplinary Research Explainable AI (XAI) is inherently a multidisciplinary effort (LON, 2024), for example collaboration between researchers from fields, such as computer science, cognitive science (which focuses on understanding how humans perceive and interpret information), human-computer interaction (HCI) (which studies the design and use of computer technologies about user experience), and domain-specific areas (e.g., healthcare or finance) can be envisioned for the effective use of AI (Langer et al., 2021). Each field offers unique perspectives, making it difficult to agree on shared design principles or explainability evaluation standards (Speith, 2022). The main reason for this is that the number of publications in XAI is growing so quickly that researchers struggle to keep up with studies in their field. This leaves them with little time to explore work from other fields, which also has a large and overwhelming amount of research. To address this, collaboration between AI researchers, cognitive scientists, and domain experts is essential to advance the use of large language models (LLMs) in explainable AI (XAI) (Cabour et al., 2023; LON, 2024). Cognitive scientists, who study how the human brain processes information, can help design LLM-generated explanations that align with how people process information, ensuring they are clear and intuitive for users. Domain experts, such as doctors or lawyers, can guide the development of explanations that are tailored to the professional requirement, making them both relevant and actionable. For example, collaboration between AI researchers and healthcare professionals could lead to LLM-based explainability techniques designed for medical decision-making. LLMs could generate textual explanations that summarize patient data or medical imaging results in ways that are understandable to clinicians (Leeftink et al., 2020). Cognitive scientists could further enhance explanations by studying how doctors interpret visual aids, such as Grad-CAM heatmaps as discussed in section 9.2, helping to integrate textual and visual explanations into a seamless and user-friendly format (Lamba and Rani, 2024). This approach could pave the way for more effective and interpretable AI systems in specialized fields. ## References - (1) - LON (2024) 2024. Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions. Information Fusion 106 (2024), 102301. https://doi.org/10.1016/j.inffus.2024.102301 - Abu-Rasheed et al. (2023) Hasan Abu-Rasheed, Mareike Dornhöfer, Christian Weber, Gábor Kismihók, Ulrike Buchmann, and Madjid Fathi. 2023. Building contextual knowledge graphs for personalized learning recommendations using text mining and semantic graph completion. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT). IEEE, 36–40. - Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, et al. 2023. GPT-4 Technical Report. Technical Report arXiv:2303.08774. OpenAI. https://doi.org/10.48550/arXiv.2303.08774 Updated authors list; fixed author names and added citation. - Aftab et al. ([n. d.]) Tuba Aftab, Mansoor Hussain, M Ahsan Saeed, Asma Yousaf, Naseer Ali Shah, and Haroon Ahmed. [n. d.]. XAI and Disease Diagnosis. In Explainable Artificial Intelligence (XAI) in Healthcare. CRC Press, 100–140. - Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. Explanations for commonsenseqa: New dataset and models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 3050–3065. - Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: An Open Large Language Model with State-of-the-Art Performance. (2023). - Amershi et al. (2019) Saleema Amershi, Daniel Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Patrick Collisson, Justin Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-ai interaction. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13. - Amin et al. (2023) Mostafa M. Amin, Erik Cambria, and Björn W. Schuller. 2023. Can ChatGPT’s Responses Boost Traditional Natural Language Processing? arXiv preprint arXiv:2307.04648 (2023), 9. https://arxiv.org/abs/2307.04648 - Anik and Bunt (2024) Ariful Islam Anik and Andrea Bunt. 2024. Diversity Challenges in Recruiting for Human-Centered Explainable AI Studies. In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 429–431. - Attai et al. (2024) Kingsley Attai, Moses Ekpenyong, Constance Amannah, Daniel Asuquo, Peterben Ajuga, Okure Obot, Ekemini Johnson, Anietie John, Omosivie Maduka, Christie Akwaowo, and Faith-Michael Uzoka. 2024. Enhancing the Interpretability of Malaria and Typhoid Diagnosis with Explainable AI and Large Language Models. Tropical Medicine and Infectious Disease 9, 9 (2024). https://doi.org/10.3390/tropicalmed9090216 - Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report. arXiv preprint arXiv:2309.16609 (2023). https://doi.org/10.48550/arXiv.2309.16609 Comments: 59 pages, 5 figures. - Barredo Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58 (2020), 82–115. - Benary et al. (2023) Manuela Benary, Xing David Wang, Max Schmidt, Dominik Soll, Georg Hilfenhaus, Mani Nassir, Christian Sigler, Maren Knödler, Ulrich Keller, Dieter Beule, et al. 2023. Leveraging large language models for decision support in personalized oncology. JAMA Network Open 6, 11 (2023), e2343689–e2343689. - Bhat and Raychowdhury (2023) Ashwin Bhat and Arijit Raychowdhury. 2023. Non-Uniform Interpolation in Integrated Gradients for Low-Latency Explainable-AI. arXiv:2302.11107 [cs.LG] https://arxiv.org/abs/2302.11107 - Bhatt et al. (2020) Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. 2020. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 648–657. https://doi.org/10.1145/3351095.3375624 - Bhattacharjee et al. (2024) Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. 2024. Towards LLM-guided Causal Explainability for Black-box Text Classifiers. Preprint (January 2024). - Bhattacharya et al. (2025) Sagnik Bhattacharya, Muhammad Ahmed Mohsin, Ahsan Bilal, and John M Cioffi. 2025. Task and Perception-aware Distributed Source Coding for Correlated Speech under Bandwidth-constrained Channels. arXiv preprint arXiv:2501.17879 (2025). - Bianchi and Briere (2021) Milo Bianchi and Marie Briere. 2021. Robo-Advising: Less AI and More XAI? Available at SSRN 3825110 (2021). - Billi et al. (2023) Marco Billi, Alessandro Parenti, Giuseppe Pisano, and Marco Sanchi. 2023. Large Language Models and Explainable Law: a Hybrid Methodology. arXiv preprint arXiv:2311.11811 (2023). - Blanchard et al. (2022) Andrew E Blanchard, John Gounley, Debsindhu Bhowmik, Mayanka Chandra Shekar, Isaac Lyngaas, Shang Gao, Junqi Yin, Aristeidis Tsaris, Feiyi Wang, and Jens Glaser. 2022. Language models for the prediction of SARS-CoV-2 inhibitors. The International Journal of High Performance Computing Applications 36, 5-6 (2022), 587–602. - Bostrom and Yudkowsky (2018) Nick Bostrom and Eliezer Yudkowsky. 2018. The Ethics of Artificial Intelligence. In Artificial Intelligence Safety and Security. Chapman and Hall, New York, 57–69. - Bouke et al. (2024) Mohamed Aly Bouke, Omar Imhemed Alramli, and Azizol Abdullah. 2024. XAIRF-WFP: a novel XAI-based random forest classifier for advanced email spam detection. International Journal of Information Security 24, 1 (Oct 2024), 5. https://doi.org/10.1007/s10207-024-00920-1 - Burton et al. (2023) James Burton, Noura Al Moubayed, and Amir Enshaei. 2023. Natural Language Explanations for Machine Learning Classification Decisions. In 2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–9. - Cabour et al. (2023) Garrick Cabour, Andrés Morales-Forero, Élise Ledoux, and Samuel Bassetto. 2023. An explanation space to align user studies with the technical development of Explainable AI. AI & SOCIETY 38, 2 (2023), 869–887. https://doi.org/10.1007/s00146-022-01536-6 - Cai et al. (2019) Carrie J. Cai, Jonas Jongejan, and Jess Holbrook. 2019. The effects of example-based explanations in a machine learning interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19). Association for Computing Machinery, New York, NY, USA, 258–262. https://doi.org/10.1145/3301275.3302289 - Caliskan (2023) Aylin Caliskan. 2023. Artificial Intelligence, Bias, and Ethics. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23). International Joint Conferences on Artificial Intelligence Organization, 7007–7013. https://doi.org/10.24963/ijcai.2023/799 - Camburu et al. (2018a) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018a. e-SNLI: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, Vol. 31. - Camburu et al. (2018b) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018b. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems 31 (2018). - Cantini et al. (2024) Riccardo Cantini, Alessio Orsino, and Domenico Talia. 2024. XAI-driven knowledge distillation of large language models for efficient deployment on low-resource devices. Journal of Big Data 11, 63 (2024). https://doi.org/10.1186/s40537-024-00928-3 - Cao (2020) Longbing Cao. 2020. AI in Finance: A Review. SSRN Electronic Journal (July 10 2020). Available at SSRN: https://ssrn.com/abstract=3647625 or http://dx.doi.org/10.2139/ssrn.3647625. - Cao (2023) Lang Cao. 2023. Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism. arXiv preprint arXiv:2311.01041 (2023). - Cardoso et al. (2004) Horacio Cardoso, Beatriz Crispino, Adriana Mimbacas, and Manuel Enrique Cardoso. 2004. A low prevalence of cystic fibrosis in Uruguayans of mainly European descent. Genet Mol Res 3, 2 (2004), 258–63. - Cerekci et al. (2024) Esma Cerekci, Deniz Alis, Nurper Denizoglu, Ozden Camurdan, Mustafa Ege Seker, Caner Ozer, Muhammed Yusuf Hansu, Toygar Tanyel, Ilkay Oksuz, and Ercan Karaarslan. 2024. Quantitative evaluation of Saliency-Based Explainable artificial intelligence (XAI) methods in Deep Learning-Based mammogram analysis. European Journal of Radiology 173 (2024), 111356. https://doi.org/10.1016/j.ejrad.2024.111356 - Chaves et al. (2024) Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu, Eeshit Dhaval Vaishnav, Byron Lee, S Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan, and Shekoofeh Azizi. 2024. Tx-llm: A large language model for therapeutics. arXiv preprint arXiv:2406.06316 (2024). - Chen et al. (2024) Zichen Chen, Jianda Chen, Ambuj Singh, and Misha Sra. 2024. XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 7578–7596. - Chen et al. (2021) Zhongxia Chen, Xiting Wang, Xing Xie, Mehul Parsana, Akshay Soni, Xiang Ao, and Enhong Chen. 2021. Towards explainable conversational recommendation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2994–3000. - Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2, 3 (2023), 6. - Chu et al. (2023) Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402 (2023). https://arxiv.org/abs/2309.15402 - Dave et al. (2024) Akshay J. Dave, Tat Nghia Nguyen, and Richard B. Vilim. 2024. Integrating LLMs for Explainable Fault Diagnosis in Complex Systems. arXiv:2402.06695 [cs.AI] https://arxiv.org/abs/2402.06695 - Day (2022) Connor Day. 2022. The Relationship Between Twitter Mentions & Stock Volatility During Trading Hours. (2022). - de Bruijn et al. (2022) Hans de Bruijn, Martijn Warnier, and Marijn Janssen. 2022. The perils and pitfalls of explainable AI: Strategies for explaining algorithmic decision-making. Government Information Quarterly 39, 2 (2022), 101666. https://doi.org/10.1016/j.giq.2021.101666 - Do et al. (2024) Hyo Jin Do, Rachel Ostrand, Justin D. Weisz, Casey Dugan, Prasanna Sattigeri, Dennis Wei, Keerthiram Murugesan, and Werner Geyer. 2024. Facilitating Human-LLM Collaboration through Factuality Scores and Source Attributions. arXiv:2405.20434 [cs.HC] https://arxiv.org/abs/2405.20434 - Dong et al. (2017) Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang. 2017. Improving interpretability of deep neural networks with semantic information. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4306–4314. - Ehsan and Riedl (2020) Upol Ehsan and Mark O Riedl. 2020. Human-centered explainable AI: Towards a reflective sociotechnical approach. arXiv preprint arXiv:2002.01092 (2020). - Eiband et al. (2018) Malin Eiband, Hanna Schneider, Marko Bilandzic, Julia Fazekas-Con, Michael Haug, and Heinrich Hussmann. 2018. Bringing transparency design into practice. In Proceedings of the 23rd International Conference on Intelligent User Interfaces. 211–223. - Elouardi et al. (2024) Saida Elouardi, Anas Motii, Mohammed Jouhari, Abdoul Nasser Hassane Amadou, and Mustapha Hedabou. 2024. A survey on Hybrid-CNN and LLMs for intrusion detection systems: Recent IoT datasets. IEEE Access (2024). - Erak et al. (2024) Omar Erak, Nouf Alabbasi, Omar Alhussein, Ismail Lotfi, Amr Hussein, Sami Muhaidat, and Merouane Debbah. 2024. Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards. arXiv:2408.11775 [cs.CL] arXiv preprint, perpetual non-exclusive license. - Fang et al. (2023) Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao. 2023. Bias of AI-Generated Content: An Examination of News Produced by Large Language Models. https://doi.org/10.21203/rs.3.rs-3499674/v1 - Farayola et al. (2024) Oluwatoyin Ajoke Farayola, Oluwabukunmi Latifat Olorunfemi, and Philip Olaseni Shoetan. 2024. DATA PRIVACY AND SECURITY IN IT: A REVIEW OF TECHNIQUES AND CHALLENGES. Computer Science & IT Research Journal 5, 3 (Mar. 2024), 606–615. https://doi.org/10.51594/csitrj.v5i3.909 - Fei et al. (2023) Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. arXiv preprint arXiv:2305.19148 (2023). - Feng et al. (2023a) Duanyu Feng, Yongfu Dai, Jimin Huang, Yifang Zhang, Qianqian Xie, Weiguang Han, Zhengyu Chen, Alejandro Lopez-Lira, and Hao Wang. 2023a. Empowering many, biasing a few: Generalist credit scoring through large language models. arXiv preprint arXiv:2310.00566 (2023). - Feng et al. (2023b) Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023b. Knowledge Card: Filling LLMs’ Knowledge Gaps with Plug-in Specialized Language Models. In Proceedings of the Twelfth International Conference on Learning Representations. - Friel et al. (2024) Robert Friel, Masha Belyi, and Atindriyo Sanyal. 2024. Ragbench: Explainable benchmark for retrieval-augmented generation systems. arXiv preprint arXiv:2407.11005 (2024). - Garcia et al. (2024) Cristian I Garcia, Marcus A DiBattista, Tomás A Letelier, Hunter D Halloran, and Jaime A Camelio. 2024. Framework for LLM applications in manufacturing. Manufacturing Letters 41 (2024), 253–263. - Gerston (2010) Larry N. Gerston. 2010. Public Policy Making: Process and Principles (3rd edition ed.). Routledge, New York. https://doi.org/10.4324/9781315701387 - Gil et al. (2021) Yolanda Gil, Daniel Garijo, Deborah Khider, Craig A. Knoblock, Varun Ratnakar, Maximiliano Osorio, Hernán Vargas, Minh Pham, Jay Pujara, Basel Shbita, Binh Vu, Yao-Yi Chiang, Dan Feldman, Yijun Lin, Hayley Song, Vipin Kumar, Ankush Khandelwal, Michael Steinbach, Kshitij Tayal, Shaoming Xu, Suzanne A. Pierce, Lissa Pearson, Daniel Hardesty-Lewis, Ewa Deelman, Rafael Ferreira Da Silva, Rajiv Mayani, Armen R. Kemanian, Yuning Shi, Lorne Leonard, Scott Peckham, Maria Stoica, Kelly Cobourn, Zeya Zhang, Christopher Duffy, and Lele Shu. 2021. Artificial Intelligence for Modeling Complex Systems: Taming the Complexity of Expert Models to Improve Decision Making. 11, 2 (2021). https://doi.org/10.1145/3453172 - Golovneva et al. (2022) Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations (ICLR). - Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838 (2024). - Hosain et al. (2024) Md Tanzib Hosain, Jamin Rahman Jim, MF Mridha, and Md Mohsin Kabir. 2024. Explainable AI approaches in deep learning: Advancements, applications and challenges. Computers and Electrical Engineering 117 (2024), 109246. - Hua et al. (2024) Wenyue Hua, Yingqiang Ge, Shuyuan Xu, Jianchao Ji, and Yongfeng Zhang. 2024. UP5: Unbiased Foundation Model for Fairness-aware Recommendation. arXiv:2305.12090 [cs.IR] https://arxiv.org/abs/2305.12090 - Huang et al. (2023) Dongxu Huang, Qiang Bu, Jing Zhang, Xiaofei Xie, Junjie Chen, and Heng Cui. 2023. Bias Assessment and Mitigation in LLM-based Code Generation. arXiv preprint arXiv:2309.14345 (2023). https://doi.org/10.48550/arXiv.2309.14345 - IBM (2024) IBM. 2024. What is Edge AI? https://www.ibm.com/topics/edge-ai Accessed: 2024-10-20. - Ichmoukhamedov et al. (2024) Timour Ichmoukhamedov, James Hinns, and David Martens. 2024. How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives. arXiv preprint arXiv:2412.10220 (2024). - Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685 (2020). - Jaff et al. (2024) Evin Jaff, Yuhao Wu, Ning Zhang, and Umar Iqbal. 2024. Data Exposure from LLM Apps: An In-depth Investigation of OpenAI’s GPTs. arXiv:2408.13247 [cs.CR] https://arxiv.org/abs/2408.13247 - Jansen et al. (2018) Peter A Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton T Morrison. 2018. Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. arXiv preprint arXiv:1802.03052 (2018). - Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825 [cs.CL] https://arxiv.org/abs/2310.06825 - Jiang et al. (2024b) Tingting Jiang, Zhumo Sun, Shiting Fu, and Yan Lv. 2024b. Human-AI interaction research agenda: A user-centered perspective. Data and Information Management (2024), 100078. https://doi.org/10.1016/j.dim.2024.100078 - Jiang et al. (2024a) Yue Jiang, Qin Chao, Yile Chen, Xiucheng Li, Shuai Liu, and Gao Cong. 2024a. UrbanLLM: Autonomous Urban Activity Planning and Management with Large Language Models. arXiv preprint arXiv:2406.12360 (2024). - Kadir et al. (2023) Md Abdul Kadir, Amir Mosavi, and Daniel Sonntag. 2023. Evaluation Metrics for XAI: A Review, Taxonomy, and Practical Applications. In 2023 IEEE 27th International Conference on Intelligent Engineering Systems (INES). IEEE, 000111–000124. - Kamruzzaman and Kim (2024) Mahammed Kamruzzaman and Gene Louis Kim. 2024. Prompting techniques for reducing social bias in llms through system 1 and system 2 cognitive processes. arXiv preprint arXiv:2404.17218 (2024). - Khan et al. (2024) Naeem Khan, Muhammad Asim Shahid, and Saad Rasool. 2024. Leveraging AI in Accounting and Finance: Transforming Business Operations and Enhancing Healthcare Decision-Making through Brain-Inspired Analytics. International Journal of Advanced Engineering, Technology, and Innovation 10, 2 (2024). - Khediri et al. (2024) Abderrazak Khediri, Hamda Slimi, Ayoub Yahiaoui, Makhlouf Derdour, Hakim Bendjenna, and Charaf Eddine Ghenai. 2024. Enhancing Machine Learning Model Interpretability in Intrusion Detection Systems through SHAP Explanations and LLM-Generated Descriptions. In 2024 6th International Conference on Pattern Analysis and Intelligent Systems (PAIS). 1–6. https://doi.org/10.1109/PAIS62114.2024.10541168 - Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). arXiv:1711.11279 [stat.ML] https://arxiv.org/abs/1711.11279 - Klein (2020) A. Klein. 2020. Reducing Bias in AI-Based Financial Services. Journal of Financial Technology 15, 3 (2020), 45–58. - Koo et al. (2024) Ryan Koo, Minhwa Lee, Vipul Raheja, Jonginn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking Cognitive Biases in Large Language Models as Evaluators. In Proceedings of the Conference on Cognitive Biases in Large Language Models. - Kroeger et al. (2023) Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. 2023. Are Large Language Models Post Hoc Explainers? arXiv preprint arXiv:2310.05797 (2023). - Kuźniacki et al. (2022) Błażej Kuźniacki, Marco Almada, Kamil Tyliński, Łukasz Górski, Beata Winogradska, and Reza Zeldenrust. 2022. Towards eXplainable Artificial Intelligence (XAI) in tax law: the need for a minimum legal standard. World Tax Journal 14, 4 (2022), 573–616. https://hdl.handle.net/1814/74855 - Laak and Aru (2024) Kristjan-Julius Laak and Jaan Aru. 2024. AI and personalized learning: bridging the gap with modern educational goals. arXiv preprint arXiv:2404.02798 (2024). - Lamba and Rani (2024) Kamini Lamba and Shalli Rani. 2024. A novel approach of brain-computer interfacing (BCI) and Grad-CAM based explainable artificial intelligence: Use case scenario for smart healthcare. Journal of Neuroscience Methods 408 (2024), 110159. https://doi.org/10.1016/j.jneumeth.2024.110159 - Langer et al. (2021) M. Langer, K. Baum, K. Hartmann, S. Hessel, T. Speith, and J. Wahl. 2021. Explainability Auditing for Intelligent Systems: A Rationale for Multi-Disciplinary Perspectives. In 29th IEEE International Requirements Engineering Conference Workshops (REW 2021), T. Yue and M. Mirakhorli (Eds.). IEEE, Piscataway, NJ, USA, 164–168. https://doi.org/10.1109/REW53955.2021.00030 - Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, and et al. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702 (2023). https://arxiv.org/abs/2307.13702 - Lee et al. (2022) Junhee Lee, Hyeonseong Cho, Yun Jang Pyun, Suk-Ju Kang, and Hyoungsik Nam. 2022. Heatmap assisted accuracy score evaluation method for machine-centric explainable deep neural networks. IEEE Access 10 (2022), 64832–64849. - Leeftink et al. (2020) AG Leeftink, IA Bikker, IMH Vliegen, and RJ Boucherie. 2020. Multi-disciplinary planning in health care: a review. Health Systems 9, 2 (2020), 95–118. - Lim and Dey (2009) B.Y. Lim and A.K. Dey. 2009. Assessing Demand for Intelligibility in Context-Aware Applications. In Proceedings of the 11th International Conference on Ubiquitous Computing. ACM, New York, NY, USA, 195–204. - Lim et al. (2019) Brian Y. Lim, Qian Yang, Ashraf Abdul, and Danding Wang. 2019. Why these Explanations? Selecting Intelligibility Types for Explanation Goals. In IUI Workshops. https://api.semanticscholar.org/CorpusID:77393694 - Limonad et al. (2024) Lior Limonad, Fabiana Fournier, Juan Manuel Vera Díaz, Inna Skarbovsky, Shlomit Gur, and Raquel Lazcano. 2024. Monetizing Currency Pair Sentiments through LLM Explainability. arXiv preprint arXiv:2407.19922 (2024). - Liu et al. (2024b) Xiao-huan Liu, Zhen-hua Lu, Tao Wang, and Fei Liu. 2024b. Large language models facilitating modern molecular biology and novel drug development. Frontiers in Pharmacology 15 (2024), 1458739. - Liu et al. (2024a) Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. 2024a. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18041 (2024). - Liu et al. (2023) Zhengliang Liu, Tianyang Zhong, Yiwei Li, Yutong Zhang, Yi Pan, Zihao Zhao, Peixin Dong, Chao Cao, Yuxiao Liu, Peng Shu, et al. 2023. Evaluating large language models for radiology natural language processing. arXiv preprint arXiv:2307.13693 (2023). - Lundberg and Lee (2017) Scott Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. arXiv preprint arXiv:1705.07874 (2017). https://arxiv.org/abs/1705.07874 To appear in NIPS 2017. - Luz (2024) Ayuns Luz. 2024. Enhancing the Interpretability and Explainability of AI-Driven Risk Models Using LLM Capabilities. Technical Report. EasyChair. - Manna and Sett (2024) Supriya Manna and Niladri Sett. 2024. Need of AI in Modern Education: in the Eyes of Explainable AI (xAI). arXiv preprint arXiv:2408.00025 (2024). - Martens et al. (2023) David Martens, James Hinns, Camille Dams, Mark Vergouwen, and Theodoros Evgeniou. 2023. Tell Me a Story! Narrative-Driven XAI with Large Language Models. arXiv preprint arXiv:2309.17057 (2023). https://doi.org/10.48550/arXiv.2309.17057 - Masyutin (2015) Alexey A. Masyutin. 2015. Credit Scoring Based on Social Network Data. Business Informatics 3 (33) (2015), 15–23. - Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 14867–14875. - Memon et al. (2024a) Zeeshan Memon, Muhammad Arham, Adnan Ul-Hasan, and Faisal Shafait. 2024a. LLM-Informed Discrete Prompt Optimization. In ICML 2024 Workshop on LLMs and Cognition. https://openreview.net/forum?id=d0jQuZe6k0 - Memon et al. (2024b) Zeeshan Memon, Muhammad Arham, Adnan Ul-Hasan, and Faisal Shafait. 2024b. LLM-Informed Discrete Prompt Optimization. In ICML 2024 Workshop on LLMs and Cognition. - Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 (2018). - Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251 (2023). - Mohsin (2023) Muhammad Ahmed Mohsin. 2023. Zero-Shot Learning via GANs and SAGANs: A Performance Analysis. In 2023 International Conference on Frontiers of Information Technology (FIT). IEEE, 37–42. - Mohsin et al. (2025a) Muhammad Ahmed Mohsin, Ahsan Bilal, Sagnik Bhattacharya, and John M Cioffi. 2025a. Retrieval Augmented Generation with Multi-Modal LLM Framework for Wireless Environments. arXiv preprint arXiv:2503.07670 (2025). - Mohsin et al. (2025b) Muhammad Ahmed Mohsin, Muhammad Jazib, Zeeshan Alam, Muhmmad Farhan Khan, Muhammad Saad, and Muhammad Ali Jamshed. 2025b. Vision Transformer Based Semantic Communications for Next Generation Wireless Networks. arXiv preprint arXiv:2503.17275 (2025). - Mohsin et al. (2025c) Muhammad Ahmed Mohsin, Hassan Rizwan, Muhammad Umer, Sagnik Bhattacharya, Ahsan Bilal, and John M Cioffi. 2025c. Hierarchical Deep Reinforcement Learning for Adaptive Resource Management in Integrated Terrestrial and Non-Terrestrial Networks. arXiv preprint arXiv:2501.09212 (2025). - Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice Van Keulen, and Christin Seifert. 2023. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. Comput. Surveys 55, 13s (2023), 1–42. - Neerincx et al. (2019) Mark A Neerincx, Willem van Vught, Olivier BA Henkemans, Erika Oleari, Joost Broekens, Rob Peters, Frank Kaptein, Yiannis Demiris, Bernd Kiefer, Daniele Fumagalli, et al. 2019. Socio-cognitive engineering of a robotic partner for child’s diabetes self-management. Frontiers in Robotics and AI 6 (2019), 1–16. - Ng and Fung (2024) Chee Ng and Yuen Fung. 2024. Educational Personalized Learning Path Planning with Large Language Models. arXiv preprint arXiv:2407.11773 (2024). - Nguyen et al. (2024) Thu Nguyen, Alessandro Canossa, and Jichen Zhu. 2024. How Human-Centered Explainable AI Interfaces Are Designed and Evaluated: A Systematic Survey. arXiv preprint arXiv:2403.14496 (March 2024). https://arxiv.org/abs/2403.14496v1 arXiv:2403.14496v1 [cs.HC]. - Nguyen et al. (2023) Van Bach Nguyen, Jörg Schlötterer, and Christin Seifert. 2023. From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent. In World Conference on Explainable Artificial Intelligence. Springer, 71–96. - OpenAI et al. (2023) OpenAI, J. Achiam, S. Adler, S. Agarwal, et al. 2023. GPT-4 Technical Report. Technical Report. OpenAI. - Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Daniel Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katie Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35. 27730–27744. - Pahde et al. (2023) Frederik Pahde, Maximilian Dreyer, Wojciech Samek, and Sebastian Lapuschkin. 2023. Reveal to revise: An explainable ai life cycle for iterative bias correction of deep models. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 596–606. - Park et al. (2021) Hyunseok Park, Aamir Ali, Rahul Mall, Tian Liu, and Andrew S. Barnard. 2021. Fast Derivation of Shapley Based Feature Importances through Feature Extraction Methods for Nanoinformatics. Machine Learning: Science and Technology 2, 3 (2021), 035034. https://doi.org/10.1088/2632-2153/AC0167 - Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamdan Alobeidli, Boris Pannier, Ebrahim Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116 (2023). - Peters and Carman (2024) Uwe Peters and Mary Carman. 2024. Cultural Bias in Explainable AI Research: A Systematic Analysis. arXiv:2403.05579 [cs.HC] https://arxiv.org/abs/2403.05579 - Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361 (2019). - Räuker et al. (2023) Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. 2023. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 ieee conference on secure and trustworthy machine learning (satml). IEEE, 464–483. - Reddy et al. (2019) SRS Reddy, Sravani Nalluri, Subramanyam Kunisetti, S Ashok, and B Venkatesh. 2019. Content-based movie recommendation system using genre correlation. In Smart Intelligent Computing and Applications: Proceedings of the Second International Conference on SCI 2018, Volume 2. Springer, 391–397. - Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should I trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1135–1144. - Schoonderwoerd et al. (2021) Tjeerd A.J. Schoonderwoerd, Wiard Jorritsma, Mark A. Neerincx, and Karel van den Bosch. 2021. Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. International Journal of Human-Computer Studies 154 (2021), 102684. https://doi.org/10.1016/j.ijhcs.2021.102684 - Selvaraju et al. (2019) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2019. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. International Journal of Computer Vision 128, 2 (Oct. 2019), 336–359. https://doi.org/10.1007/s11263-019-01228-7 - Setzu et al. (2021) Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2021. GLocalX - From Local to Global Explanations of Black Box AI Models. Artificial Intelligence 294 (2021), 103457. https://doi.org/10.1016/j.artint.2021.103457 - Shen et al. (2023) Hua Shen, Chieh-Yang Huang, Tongshuang Wu, and Ting-Hao Kenneth Huang. 2023. ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing. In Computer Supported Cooperative Work and Social Computing. 384–387. https://doi.org/10.1145/3584931.3607492 - Singer et al. (2016) M. Singer, C. S. Deutschman, C. W. Seymour, M. Shankar-Hari, D. Annane, M. Bauer, R. Bellomo, G. R. Bernard, J. D. Chiche, C. M. Coopersmith, R. S. Hotchkiss, M. M. Levy, J. C. Marshall, G. S. Martin, S. M. Opal, G. D. Rubenfeld, T. van der Poll, J. L. Vincent, and D. C. Angus. 2016. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 315, 8 (2016), 801–810. https://doi.org/10.1001/jama.2016.0287 - Slack et al. (2023) Dylan Slack, Satyapriya Krishna, Himabindu Lakkaraju, and Sameer Singh. 2023. Explaining Machine Learning Models with Interactive Natural Language Conversations using TalkToModel. Nature Machine Intelligence 5, 8 (2023), 873–883. - Sokol and Vogt (2024) Kacper Sokol and Julia E Vogt. 2024. What does evaluation of explainable artificial intelligence actually tell us? A case for compositional and contextual validation of XAI building blocks. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1–8. - Speith (2022) T. Speith. 2022. A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods. In Proceedings of the 5th ACM Conference on Fairness, Accountability, and Transparency, C. Isbell, S. Lazar, A. Oh, and A. Xiang (Eds.). Association for Computing Machinery, New York, NY, USA, 2239–2250. https://doi.org/10.1145/3531146.3534639 - Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 (2018). - Thanh and Satoh (2024) Nguyen Ha Thanh and Ken Satoh. 2024. KRAG Framework for Enhancing LLMs in the Legal Domain. arXiv preprint arXiv:2410.07551 (2024). - Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature Medicine (2023), 1–11. - Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Pierre Albert, Amjad Almahairi, Yassine Babaei, Nikita Bashlykov, Subhojit Batra, Pranav Bhargava, Sharan Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). - Trivedi et al. (2024) Chandan Trivedi, Pronaya Bhattacharya, Vivek Kumar Prasad, Viraj Patel, Arunendra Singh, Sudeep Tanwar, Ravi Sharma, Srinivas Aluvala, Giovanni Pau, and Gulshan Sharma. 2024. Explainable AI for Industry 5.0: Vision, Architecture, and Potential Directions. IEEE Open Journal of Industry Applications (2024). - Tull et al. (2024) Sean Tull, Robin Lorenz, Stephen Clark, Ilyas Khan, and Bob Coecke. 2024. Towards Compositional Interpretability for XAI. arXiv:2406.17583 [cs.AI] https://arxiv.org/abs/2406.17583 - Umer et al. (2023) Muhammad Umer, Muhammad Ahmed Mohsin, Adnan Ul-Hasan, and Faisal Shafait. 2023. PyramidTabNet: transformer-based table recognition in image-based documents. In International Conference on Document Analysis and Recognition. Springer, 420–437. - Umerenkov et al. (2023a) D. Umerenkov, G. Zubkova, and A. Nesterov. 2023a. Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making. arXiv:2310.01708 [cs.CL] https://arxiv.org/abs/2310.01708 - Umerenkov et al. (2023b) Dmitriy Umerenkov, Galina Zubkova, and Aleksandr Nesterov. 2023b. Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making. arXiv preprint arXiv:2310.01708 (2023). - Wang et al. (2023) Rui Wang, Pengyu Cheng, and Ricardo Henao. 2023. Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Sampling. In 26th International Conference on Artificial Intelligence and Statistics, AISTATS 2023. ML Research Press, 4473–4485. KAUST Repository Item: Exported on 2023-07-28. - Wang et al. (2024) Ruochen Wang, Si Si, Felix Yu, Dorothea Wiesmann, Cho-Jui Hsieh, and Inderjit Dhillon. 2024. Large Language Models are Interpretable Learners. arXiv preprint arXiv:2406.17224 (Jun 2024). https://doi.org/10.48550/arXiv.2406.17224 Work completed during internship at Google. - Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https://arxiv.org/abs/2201.11903 - Weld and Bansal (2019) Daniel S. Weld and Gunjan Bansal. 2019. The Challenge of Crafting Intelligible Intelligence. Commun. ACM 62, 6 (2019), 70–79. - Williams (2016) Thomas N Williams. 2016. Sickle cell disease in sub-Saharan Africa. Hematology/Oncology Clinics 30, 2 (2016), 343–358. - Wolf (2019) Christopher T Wolf. 2019. Explainability scenarios: Towards scenario-based XAI design. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 252–257. - Wu et al. (2023) Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT: A Large Language Model for Finance. arXiv preprint arXiv:2303.17564 (2023). - Xu et al. (2021) Shoujin Xu, Yan Li, Shuai Liu, Zhixuan Fu, Yong Ge, Xiangliang Chen, and Yong Zhang. 2021. Learning causal explanations for recommendation. In The 1st International Workshop on Causality in Search and Recommendation. - Yang et al. (2023a) Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023a. Towards interpretable mental health analysis with large language models. arXiv preprint arXiv:2304.03347 (2023). - Yang et al. (2023b) Wenli Yang, Yuchen Wei, Hanyu Wei, Yanyu Chen, Guan Huang, Xiang Li, Renjie Li, Naimeng Yao, Xinyi Wang, Xiaotong Gu, et al. 2023b. Survey on explainable AI: From approaches, limitations and applications aspects. Human-Centric Intelligent Systems 3, 3 (2023), 161–188. - Yao et al. (2024) Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly. High-Confidence Computing 4, 2 (2024), 100211. https://doi.org/10.1016/j.hcc.2024.100211 - Yeo et al. (2023) Wei Jie Yeo, Ranjan Satapathy, Rick Siow Mong Goh, and Erik Cambria. 2023. How Interpretable are Reasoning Explanations from Prompting Large Language Models? (2023). - Zhang et al. (2024) Bohui Zhang, Albert Meroño-Peñuela, and Elena Simperl. 2024. Towards Explainable Automated Knowledge Engineering with Human-in-the-loop. Department of Informatics, King’s College London (2024). E-mails: bohui.zhang@kcl.ac.uk, albert.merono@kcl.ac.uk, elena.simperl@kcl.ac.uk. - Zhang et al. (2022) Yuhao Zhang, Kevin McAreavey, and Weiru Liu. 2022. Developing and experimenting on approaches to explainability in AI systems. In ICAART 2022: 14th International Conference on Agents and Artificial Intelligence. 518–527. - Zhao et al. (2024) Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology 15, 2 (2024), 1–38. - Zhavoronkov et al. (2019) Alex Zhavoronkov, Yan A Ivanenkov, Alex Aliper, Mark S Veselov, Vladimir A Aladinskiy, Anastasiya V Aladinskaya, Victor A Terentiev, Daniil A Polykovskiy, Maksim D Kuznetsov, Arip Asadulaev, et al. 2019. Deep learning enables rapid identification of potent DDR1 kinase inhibitors. Nature biotechnology 37, 9 (2019), 1038–1040. - Zheng et al. (2024) Yizhen Zheng, Huan Yee Koh, Maddie Yang, Li Li, Lauren T May, Geoffrey I Webb, Shirui Pan, and George Church. 2024. Large language models in drug discovery and development: From disease mechanisms to clinical trials. arXiv preprint arXiv:2409.04481 (2024). - Zhou et al. (2021) Jianlong Zhou, Amir H. Gandomi, Fang Chen, and Andreas Holzinger. 2021. Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics. Electronics 10, 5 (Jan 2021), 593. https://doi.org/10.3390/electronics10050593 - Zou et al. (2024) Hang Zou, Qiyang Zhao, Lina Bariah, Yu Tian, Mehdi Bennis, Samson Lasaulce, Mérouane Debbah, and Faouzi Bader. 2024. GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoning. arXiv:2402.16631v2 [cs.AI] arXiv preprint, CC BY 4.0 License. - Zytek et al. (2024a) Alexandra Zytek, Sara Pido, Sarah Alnegheimish, Laure Berti-Equille, and Kalyan Veeramachaneni. 2024a. Explingo: Explaining AI Predictions using Large Language Models. In 2024 IEEE International Conference on Big Data (BigData). IEEE, 1197–1208. - Zytek et al. (2024b) Alexandra Zytek, Sara Pidò, and Kalyan Veeramachaneni. 2024b. LLMs for XAI: Future Directions for Explaining Explanations. arXiv preprint arXiv:2405.06064 (2024). https://doi.org/10.48550/arXiv.2405.06064