2410.17139v2

# Trustworthy XAI and Its Applications **Authors**: A.S.M AnasFerdous, AbdurRashid, Fatema Tuj JohuraSoshi, ParagBiswas, AngonaBiswas, KishorDatta Gupta [1] MD Abdullah Al Nasim 1,6] Research and Development Department, Pioneer Alpha, Dhaka, Bangladesh 2] Department of Biomedical Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh 4] Msc in Data Science and Analytics, University of Hertfordshire, Hatfield, UK 3, 5] MSEM Department, Westcliff university, California, United States 7] Department of Computer and Information Science, Clark Atlanta University, Georgia, USA ## Abstract Artificial Intelligence (AI) is an important part of our everyday lives. We use it in self-driving cars and smartphone assistants. People often call it a ”black box” because its complex systems, especially deep neural networks, are hard to understand. This complexity raises concerns about accountability, bias, and fairness, even though AI can be quite accurate. Explainable Artificial Intelligence (XAI) is important for building trust. It helps ensure that AI systems work reliably and ethically. This article looks at XAI and its three main parts: transparency, explainability, and trustworthiness. We will discuss why these components matter in real-life situations. We will also review recent studies that show how XAI is used in different fields. Ultimately, gaining trust in AI systems is crucial for their successful use in society. keywords: Artificial Intelligence(AI), XAI, Explainable Artificial Intelligence (XAI), Healthcare, Autonomous Vehicles ## 1 Introduction The foundations of modern artificial intelligence were laid by philosophers who attempted to define human thought as the mechanical manipulation of symbols, which led to the development of the programmable digital computer [1] in the 1940s. Alan Turing may have written the first article on the topic of AI in 1941, though it is now lost, suggesting that he was at least considering the idea at that time.In his groundbreaking essay ”Computing Machinery and Intelligence” from 1950, Turing first presented the idea of the Turing test to the general public [2]. Turing questioned the feasibility of creating thinking robots in it. John McCarthy first used the term artificial intelligence (AI) in 1956 at the Dartmouth Conference [3], but the first models’ numerous flaws have prevented AI from being widely adopted and used in healthcare. Many of these limitations were removed with the advent of deep learning in the early 2000s, and we are now entering a new era of technology where AI can be used in clinical practice through risk assessment models that increase diagnostic accuracy and workflow efficiency. Performance of AI systems has improved significantly in recent years, and these new models expand on their capabilities to include text-image synthesis based on almost any prompt, whereas previous systems primarily focused on generating facial images. <details> <summary>extracted/6367585/image/a1.png Details</summary>  ### Visual Description ## Circular Diagram: Application of AI ### Overview The diagram is a circular flowchart centered around the concept of "Application of AI." It branches out into concentric rings representing different industries, each with specific AI applications. The central node is labeled "Application of AI," and the outer rings are divided into color-coded sectors (e.g., Logistics, Healthcare, Retail) with bullet-pointed subcategories. The diagram emphasizes the integration of AI across diverse sectors. ### Components/Axes - **Central Node**: "Application of AI" (gray). - **Outer Rings**: Divided into 10+ sectors, each labeled with an industry and color-coded: - **Logistics** (green): UAV delivery, Automatic pilot, Intelligent logistics, Planning, Intelligent scheduling algorithm. - **Home Furnishing Industrial** (yellow): No subcategories listed. - **Retail Industry** (orange): Unmanned convenience store, Unmanned warehouse, Intelligent supply chain, Intelligent customer flow statistics and analysis. - **Security Industry** (blue): Human body analysis, Vehicle analysis, Behavior analysis, Image analysis. - **Healthcare** (green): Medical robot, Intelligent drug research, Intelligent diagnosis, Intelligent image recognition, Intelligent health management. - **Electronic Commerce** (yellow): Intelligent customer service robot, Recommendation engine, Image search, Sales and inventory, Commodity pricing. - **Other Industries** (gray): Education (Unmanned examination and marking), Agricultural (Real-time monitoring of crop status), Environmental protection (Intelligent energy consumption monitoring and analysis), Urban management (Traffic route optimization). - **Financial Industry** (blue): Big data analysis, Stock data analysis, Industry trend analysis, Investment risk analysis, Forecast. - **Manufacturing Industry** (orange): Equipment fault prediction, Product defect prediction, Machine vision detection, Man-machine cooperation. - **Color Legend**: Implicitly defined by sector colors (e.g., green for Logistics, yellow for Home Furnishing Industrial). ### Detailed Analysis - **Logistics**: Focuses on automation (UAVs, pilots) and optimization (scheduling algorithms). - **Retail**: Highlights unmanned operations and customer behavior analytics. - **Security**: Emphasizes surveillance and image analysis. - **Healthcare**: Centers on medical robots, diagnostics, and health management. - **Electronic Commerce**: Targets customer service, recommendations, and inventory management. - **Other Industries**: Broad applications in education, agriculture, environmental monitoring, and urban planning. - **Financial Industry**: Data-driven decision-making (risk analysis, forecasting). - **Manufacturing**: Predictive maintenance and human-machine collaboration. ### Key Observations 1. **Sector Diversity**: AI applications span from logistics to healthcare, demonstrating its cross-industry relevance. 2. **Automation Focus**: Many sectors prioritize automation (e.g., unmanned stores, robots). 3. **Data-Driven Insights**: Financial and retail sectors rely heavily on data analysis for decision-making. 4. **Optimization**: Logistics and urban management emphasize efficiency (scheduling, traffic routes). 5. **Healthcare Innovation**: Medical robots and diagnostics highlight AI's role in advancing healthcare. ### Interpretation The diagram illustrates the transformative impact of AI across industries, showcasing its ability to automate processes, enhance decision-making, and optimize operations. Key trends include: - **Automation Dominance**: Unmanned systems and robots are prevalent in retail, healthcare, and logistics. - **Data-Centric Applications**: Financial and retail sectors leverage AI for predictive analytics and customer insights. - **Sector-Specific Solutions**: Healthcare and manufacturing applications address niche challenges (e.g., diagnostics, predictive maintenance). - **Urban and Environmental Focus**: AI's role in traffic management and energy monitoring underscores its potential for sustainability. The diagram suggests that AI is not a one-size-fits-all technology but is tailored to address unique challenges in each industry, driving innovation and efficiency. The absence of numerical data implies a qualitative emphasis on application diversity rather than quantitative metrics. </details> Figure 1: Applications of AI across various domains [4] The diverse range of fields in which AI is being used already demonstrates its applicability and promise to revolutionize business: AI facilitates activities like sentiment analysis, machine translation, and spam filtering by making it easier for computers to comprehend and produce human language in the discipline of natural language processing (NLP) [5]. Additionally, computer vision [6] makes it possible for computers to understand visual data, which advances areas like facial recognition, object identification, and self-driving cars. Machine learning (ML), which has uses in fraud detection, recommendation systems, predictive analytics, and other domains, has made it possible for computers to learn from data. The design, development, and application of machines are the focus of the AI field of robotics [7]. Many industries, including manufacturing, healthcare, and space exploration, use various machines [8] [9]. Combining artificial intelligence with business intelligence (BI) [10] improves how businesses collect, process, and visualize data. This leads to better decision-making and increased productivity. In healthcare, AI helps diagnose diseases, develop treatments, and provide personalized care, which improves patient outcomes [11], [12], [13]. AI also plays a significant role in education by engaging students, customizing lessons, and automating administrative tasks, resulting in more personalized learning experiences. AI in agriculture increases agricultural output, reduces costs, and ensures environmental sustainability through data-driven strategies. In a similar vein, AI in manufacturing boosts output, efficiency, and quality through work automation and process optimization. AI is changing operations, enhancing services, and changing global industry landscapes in a number of sectors, including banking, retail, energy, transportation [14], handwriting detection [15], and government. AI is widely used in a wide number of industries, as seen in Figure 1. Retail, security, healthcare, e-commerce, manufacturing, banking, logistics and transportation, and home furnishings are some of these sectors. These applications rely on moderately advanced AI technology, such as computer vision, natural language processing, and machine learning. The contributions of this research can be stated below: 1. Providing an overview of XAI that understands the significance of black box models. For fair and ethical purposes, XAI fosters trust among humans and AI. 1. Discussing Deep Learning based systems that will be consistent and trustworthy. Moreover, recent studies from the literature have been reviewed properly. 1. Providing guidelines regarding XAI that will be helpful for detecting problems from numerous domains. 1. The paper identifies and analyzes three key components of XAI: transparency, explainability, and trustworthiness. It details how these elements are essential for understanding and improving AI systems. ### 1.1 Third Wave of Artificial Intelligence (3AI) Most current commercial AI technology is called ”narrow AI.” This means these systems are highly specialized and can only perform a few specific tasks. For example, even the best self-driving cars rely on limited AI systems. Another drawback of today’s AI is its reliance on large training data sets. A typical machine learning program needs tens of thousands of cat photos to recognize cats accurately, while a three-year-old child can do this with just a few examples. The idea of ”Third Wave AI” comes from the need for AI to become more humanlike in various ways to overcome these limitations and achieve its full potential. <details> <summary>x1.jpg Details</summary>  ### Visual Description ## Timeline Diagram: Evolution of AI Waves (Pre-2010 to 2030-) ### Overview The diagram illustrates the progression of AI development across four distinct waves, spanning from Pre-2010 to 2030-. Each wave is represented by a color-coded box with descriptive bullet points, connected by curved arrows indicating chronological advancement. The timeline is positioned at the top, with wave labels and descriptions aligned vertically below. ### Components/Axes - **Timeline**: Horizontal axis at the top, labeled with time periods: - Pre-2010 (First Wave) - 2010-2020 (Second Wave) - 2020-2030 (Third Wave) - 2030- (Fourth Wave) - **Wave Boxes**: Four vertically stacked boxes with rounded corners, each containing bullet points: - **First Wave** (Gray): Pre-2010 - **Second Wave** (Blue): 2010-2020 - **Third Wave** (Orange): 2020-2030 - **Fourth Wave** (Green): 2030- - **Arrows**: Curved gray arrows connect the end of one wave to the start of the next, emphasizing continuity. ### Detailed Analysis #### First Wave (Pre-2010) - **Labels**: "First Wave" (gray box) - **Text**: - Handcrafted/Human programmed - Traditional Programming - No learning capability - Poor handling of uncertainty #### Second Wave (2010-2020) - **Labels**: "Second Wave" (blue box) - **Text**: - Statistical Models trained on BIG Data - Neural Networks - Deep Learning - Individually unreliable #### Third Wave (2020-2030) - **Labels**: "Third Wave" (orange box) - **Text**: - Models to drive decisions - Models to explain decisions #### Fourth Wave (2030-) - **Labels**: "Fourth Wave" (green box) - **Text**: - More human-like learning - Learn from descriptive, contextual models instead of enormous sets of labeled training data - Learn interactively ### Key Observations 1. **Color Coding**: Each wave is distinctly color-coded (gray, blue, orange, green) to differentiate eras. 2. **Progression**: Arrows show a linear flow from First to Fourth Wave, with no branching or regression. 3. **Technical Evolution**: - Early focus on rule-based systems (First Wave). - Shift to data-driven models (Second Wave). - Emphasis on explainability and decision-making (Third Wave). - Transition to human-like, interactive learning (Fourth Wave). 4. **Temporal Scope**: The timeline spans 70+ years, with the Fourth Wave extending beyond 2030. ### Interpretation The diagram traces AI's trajectory from rigid, human-programmed systems to adaptive, context-aware models. Key trends include: - **From Rules to Data**: Early AI relied on explicit programming, while later waves leverage large datasets (BIG Data) and neural networks. - **Reliability Challenges**: Second Wave systems are noted as "individually unreliable," highlighting early limitations in consistency. - **Explainability Focus**: The Third Wave addresses the "black box" problem by prioritizing decision transparency. - **Human-Centric Design**: The Fourth Wave emphasizes learning from descriptive/contextual models and interactive processes, moving beyond labeled data dependency. This progression reflects advancements in computational power, data availability, and algorithmic sophistication, culminating in AI systems that mimic human-like reasoning and adaptability. </details> Figure 2: Past, Present, and Future of AI waves. [16] According to the Defense Advanced Research Projects Agency (DARPA) [17], third-wave AI systems will understand the context of situations, use common sense, and adapt to changes. This will create more natural and intuitive connections between AI systems and people [17]. One of DARPA’s active projects, called XAI, aims to develop these third-wave AI systems. These computers will learn about their environments and the contexts in which they work. They will also build explanations needed to clarify real-world events. - First Wave AI focused on rules, logic, and built knowledge. - Second Wave AI introduced big data, statistical learning, and probabilistic techniques. - The goal of third-wave AI is to develop common sense and the ability to adapt to different contexts. Tractica [16] predicts that the global market for AI software will grow from about 9.5 billion US dollars in 2018 to 118.6 billion by 2025. This data aims to develop AI systems that can perform tasks accurately while providing explanations that people can understand. The term ”third wave” refers to the advancement of AI technologies beyond traditional machine learning. This new phase focuses on creating more advanced systems that can understand context, reason, and think similarly to humans. It draws inspiration from cognitive science and neuroscience, aiming to build AI that can engage with the world in more complex and detailed ways. XAI, or Explainable Artificial Intelligence, focuses on making AI systems, especially machine learning models, easier to understand. The goal is to help people trust the decisions these systems make. XAI techniques work to explain how AI makes predictions and why it behaves the way it does. This helps users, such as developers, regulators, and everyday users, grasp the key factors that affect AI results. Developing AI systems that can make accurate predictions or decisions and explain why they did so is a key goal of both the third wave of AI and explainable AI. By using XAI methods in their design, developers can ensure that these advanced AI systems are not only effective but also easy to understand. This will help build user trust and acceptance. ### 1.2 Concept of Explainable AI AI often struggles with what is called the ”black box” problem because users do not understand how it works. This can lead to issues like lack of trust, confusion, unfair treatment, and violations of privacy. AI systems can also have hidden biases. Explainable AI (XAI) aims to make AI systems easier to understand, helping users know how they make decisions. The goal of XAI is to make AI safer and more user-friendly. Therefore, we need to look at each part of AI individually and discuss its different aspects [18]. There are two main types of machine learning (ML) techniques: white box and black box models [19]. Experts can easily understand the results of a white box model. However, even specialists may find it hard to grasp the results of a black box model [20]. The XAI algorithm [21] follows three key principles: interpretability, explainability, and transparency. A model is transparent when it clearly explains how it gets its results from training data and how it generates labels from test data. Interpretability means being able to explain findings in a way that others understand [22]. While there’s no single accepted definition of explainability, its value is recognized. One definition describes it as a set of clear features that help make decisions, like classifying or predicting outcomes for specific cases. An algorithm that meets these standards helps document and verify decisions and improves itself based on new data. <details> <summary>x2.jpg Details</summary>  ### Visual Description ## Diagram: Comparison of Generic vs. eXplainable AI Approaches ### Overview The diagram contrasts two AI workflows: a **Generic Approach** and an **eXplainable AI (XAI) Approach**. Both start with a **Training Dataset (DB)** and **Machine Learning Processes**, but diverge in their handling of outcomes and user interaction. The Generic Approach produces a **Learned Function** leading to an **Outcome/Decision**, while the XAI Approach introduces an **Explainable Model** and **Explainable Interface** to enhance transparency. ### Components/Axes 1. **Left Side (Generic Approach)**: - **Training Dataset (DB)**: Input for Machine Learning Processes. - **Machine Learning Processes**: Processes data to generate a **Learned Function**. - **Learned Function**: Outputs an **Outcome/Decision**. - **Users**: Interact with the system, raising questions about trust, errors, and decisions. 2. **Right Side (eXplainable AI Approach)**: - **Training Dataset (DB)**: Same input as the Generic Approach. - **Machine Learning Processes**: Processes data to generate an **Explainable Model**. - **Explainable Model**: Feeds into an **Explainable Interface**. - **Explainable Interface**: Provides transparency to **Users**, who now express confidence in understanding decisions and trust. 3. **Thought Bubbles**: - **Generic Approach Users**: Ask questions like *"Why did you do that?"* and *"How can I trust you?"*. - **XAI Approach Users**: State confidence: *"I understand why,"* *"I know when you succeed/fail,"* and *"I know how to trust you."* ### Detailed Analysis - **Training Dataset (DB)**: Positioned at the top of both workflows, serving as the foundational input for Machine Learning Processes. - **Machine Learning Processes**: Central to both approaches, but the Generic Approach lacks mechanisms for transparency. - **Learned Function (Generic)**: A black-box output leading to user skepticism (e.g., *"Why not something else?"*). - **Explainable Model/Interface (XAI)**: Adds layers of interpretability, addressing user concerns through explicit explanations. - **User Interaction**: - Generic users focus on distrust and error correction. - XAI users emphasize understanding, trust, and accountability. ### Key Observations 1. **Transparency Gap**: The Generic Approach lacks mechanisms to explain decisions, leading to user uncertainty. 2. **Trust Building**: The XAI Approach directly addresses user concerns via the Explainable Interface, fostering confidence. 3. **Workflow Complexity**: The XAI Approach introduces additional steps (Explainable Model → Interface) but improves user-system interaction. ### Interpretation The diagram underscores the importance of **explainability** in AI systems. While the Generic Approach prioritizes efficiency, it risks alienating users due to opacity. The XAI Approach, though more complex, aligns with ethical AI principles by enabling users to: - Understand decision logic (*"I understand why"*). - Assess reliability (*"I know when you succeed/fail"*). - Build trust through accountability (*"I know how to trust you"*). This highlights a trade-off between simplicity and transparency, with XAI offering a pathway to responsible AI adoption in user-centric applications. </details> Figure 3: Explainable Artificial Intelligence (XAI): A look at AI now and tomorrow. [18] Researchers are studying intelligent systems to understand them better. This is an important topic. Sometimes, a system needs to understand its own workings to comply with rules. Many complex algorithms, shown in Figure 4, balance achieving high accuracy with being explainable. <details> <summary>x3.jpg Details</summary>  ### Visual Description ## Diagram: Machine Learning Model Trade-offs (Performance vs. Explainability) ### Overview The diagram illustrates a conceptual map of machine learning models positioned along two axes: **Learning Performance** (vertical) and **Explainability** (horizontal). It categorizes models into four groups—Neural Nets/Deep Learning, Statistical Models (SVMs, AOGs), Graphical Models (Bayesian Belief Nets, SRL, CRFs, MLNS, Markov Models), and Ensemble Methods (Random Forest, Decision Tree)—and shows their relationships via directional arrows. --- ### Components/Axes - **Axes**: - **Vertical (Y-axis)**: Learning Performance (low to high, bottom to top). - **Horizontal (X-axis)**: Explainability (low to high, left to right). - **Key Labels**: - **Neural Nets / Deep Learning** (green circle, top-left quadrant). - **Statistical Models** (light blue circle, bottom-center). - Subcategories: SVMs, AOGs. - **Graphical Models** (dark blue oval, center-right). - Subcategories: Bayesian Belief Nets, SRL, CRFs, MLNS, Markov Models. - **Ensemble Methods** (dark blue rectangle, top-right). - Subcategories: Random Forest, Decision Tree. - **Arrows**: - Indicate relationships (e.g., Neural Nets → Graphical Models → Ensemble Methods). - Dashed lines suggest weaker or indirect connections. --- ### Detailed Analysis 1. **Neural Nets / Deep Learning**: - Positioned at the top-left, indicating **high performance** but **low explainability**. - Connected to Graphical Models via a direct arrow. 2. **Statistical Models (SVMs, AOGs)**: - Located at the bottom-center, reflecting **moderate performance** and **high explainability**. - Dashed lines link to Graphical Models, suggesting limited integration. 3. **Graphical Models**: - Central position (middle-right), balancing **moderate performance** and **moderate explainability**. - Subcategories (e.g., Bayesian Belief Nets, CRFs) are listed within the oval. 4. **Ensemble Methods (Random Forest, Decision Tree)**: - Top-right quadrant, showing **high performance** and **high explainability**. - Connected to Graphical Models via a direct arrow, implying synergy. --- ### Key Observations - **Trade-off**: Models with higher performance (e.g., Neural Nets) sacrifice explainability, while simpler models (e.g., SVMs) prioritize transparency over performance. - **Ensemble Methods** occupy the "sweet spot," balancing both axes. - **Graphical Models** act as a bridge between complex and interpretable models. - **Directional Arrows** suggest a progression from complex (Neural Nets) to hybrid (Graphical Models) to balanced (Ensemble Methods) approaches. --- ### Interpretation The diagram highlights the **performance-explainability trade-off** in machine learning. Neural Nets and Deep Learning dominate in performance but are "black boxes," while Statistical Models (e.g., SVMs) are interpretable but less powerful. Graphical Models (e.g., Bayesian Networks) offer a middle ground, and Ensemble Methods (e.g., Random Forests) combine performance with transparency. The arrows imply that combining techniques (e.g., Neural Nets → Graphical Models → Ensemble Methods) could optimize both axes. Notably, no model achieves peak performance **and** peak explainability simultaneously, underscoring the need for context-specific model selection. </details> Figure 4: Trade-off between AI model accuracy and explainability, highlighting the challenge of balancing performance with interpretability. [23] ### 1.3 Classification Tree of XAI XAI techniques are divided into two categories: transparent and post-hoc methods. A transparent approach is one that represents the model’s capabilities and decision-making process in an easy-to-understand way [24]. Transparent models include Bayesian approaches, decision trees, linear regression, and fuzzy inference systems. Transparent approaches can be useful when the internal feature correlations are highly complex or linear. A comprehensive classification of different XAI methods and approaches related to different types of data is shown in Figure 5. <details> <summary>extracted/6367585/image/classification.jpg Details</summary>  ### Visual Description ## Flowchart Diagram: XAI Techniques ### Overview The diagram categorizes Explainable AI (XAI) techniques into two primary branches: **Post hoc methods** and **Transparent methods**. Each branch further subdivides into specific methodologies tailored to different data types (Text, Image, Audio, Video) and model-specific approaches. ### Components/Axes - **Main Title**: "XAI Techniques" (central top node). - **Primary Branches**: - **Post hoc methods** (left branch from title). - **Transparent methods** (right branch from title). - **Subcategories**: - **Post hoc methods**: - **Model agnostic** (left sub-branch). - **Model specific** (right sub-branch). - **Transparent methods**: List of techniques under "All type of Data." ### Detailed Analysis #### Post hoc methods 1. **Model agnostic**: - **Text**: - Feature Relevance - Condition-based Local explanation - **Image**: - Rule-based learning - Feature-based saliency map - **Audio**: - Feature Relevance - Local explanation - **Video**: - Saliency map 2. **Model specific**: - **Text**: - LIME - Perturbation - LRP - Provenance - Taxonomy induc. - **Image**: - SHAP Values - HEAT map - LIME - Counterfactual explanation - Perturbation - **Audio**: - LIME - Perturbation - LRP - **Video**: - SHAP Values - Counterfactual explanation - Perturbation #### Transparent methods - **All type of Data**: - Logistic regression - Decision Trees - K-Nearest neighbors - Rule-based classifier - Bayesian Model ### Key Observations - **Hierarchical Structure**: The diagram emphasizes a top-down flow, starting with broad XAI categories and narrowing into specific techniques. - **Data-Type Specificity**: Both Post hoc and Transparent methods are further divided by data type (Text, Image, Audio, Video), indicating tailored approaches for different modalities. - **Technique Overlap**: Some methods (e.g., LIME, Perturbation) appear in both Model agnostic and Model specific categories, suggesting versatility across contexts. - **Transparency Focus**: Transparent methods are presented as standalone techniques applicable to all data types, contrasting with the Post hoc approach's dependency on model type. ### Interpretation The diagram illustrates the dual paradigms of XAI: 1. **Post hoc methods** prioritize model-agnostic or model-specific explanations after the fact, often relying on data-type-specific tools (e.g., saliency maps for images, LIME for text). 2. **Transparent methods** represent inherently interpretable models (e.g., Decision Trees, Logistic Regression) that do not require post-hoc explanation. This structure highlights the trade-off between model complexity and interpretability, with Post hoc methods offering flexibility for complex models at the cost of added explanation layers, while Transparent methods provide simplicity but may lack the sophistication of advanced models. The inclusion of techniques like SHAP and LIME underscores the importance of feature attribution in modern XAI workflows. </details> Figure 5: Categorization of Explainable AI (XAI) techniques based on data type, illustrating differences between transparent and posthoc approaches [24] Posterior approaches are useful for interpreting the complexity of a model, especially when there are nonlinear relationships or high data complexity. When a model does not follow a direct relationship between data and features, posterior techniques can be an effective tool to explain what the model has learned [24]. Inference using local feature weights is provided by transparent methods such as Bayesian classifiers, support vector machines, logistic regression, and K-nearest neighbors. This model category meets three properties: simulability, decomposability, and algorithmic transparency [24]. ### 1.4 Definition of Transparency in Artificial Intelligence Transparency in XAI is the capacity of an AI system to make its decisions and actions understandable through explanations [25]. Transparency is one of the most important aspects of XAI, rendering AI decisions interpretable and justified. Transparency allows decision-making processes to be closely examined by stakeholders, mitigating risks in applications with societal impact, such as healthcare and finance [26]. The explainability provided by XAI techniques also increases the overall transparency of AI systems. The users are able to examine the decision-making procedure, identify any bias, and analyze the reliability and fairness of the output of the model [27]. Transparent solutions are necessary in areas like the medical field, banking, and autonomous vehicles, where AI-decisions can have significant effects. By providing meaningful information about the internal processes of AI models, XAI methods assist users in recognizing patterns, comprehending relationships, and revealing biases or errors [27]. Due to the heightened transparency, stakeholders can more effectively develop opinions, ensure the predictions made by the model are accurate, and take the necessary action. The examination of ethical criteria showed a link between explainability, transparency, and other quality needs. Figure 6 displays nine quality standards related to explainability and openness. The key standards for AI transparency are marked with “O.” For example, O2 focuses on how to interpret models, O15 highlights the importance of traceability, and O5 and O12 ensure that users understand the information. By following these standards, AI models can provide clear and convincing explanations for their outcomes. Keeping these standards improves accountability and helps reduce bias in AI systems. <details> <summary>extracted/6367585/image/eai3.jpg Details</summary>  ### Visual Description ## Diagram: Transparency and Explainability Framework ### Overview The diagram illustrates a central concept of "Transparency and Explainability" (highlighted in green) interconnected with six surrounding components: Trustworthiness, Privacy, Auditability, Fairness, Traceability, and Understandability. Arrows indicate directional relationships, with "+" and "-" signs denoting positive or negative influences. ### Components/Axes 1. **Central Node**: - **Label**: Transparency and Explainability - **Connections**: - **+** to Trustworthiness (O1, O4, O5, O11) - **+/-** to Privacy (O7, O13) - **+** to Auditability (O4) - **+** to Fairness (O5) 2. **Surrounding Nodes**: - **Traceability** (O2, O9, O12) → **+** to central node - **Understandability** (O2, O5, O12, O15) → **+** to central node - **Privacy** (O7, O13) ← **+/-** from central node - **Auditability** (O4) ← **+** from central node - **Fairness** (O5) ← **+** from central node ### Detailed Analysis - **Trustworthiness** (O1, O4, O5, O11): Strongly linked to the central concept via a unidirectional positive influence. - **Privacy** (O7, O13): Bidirectional relationship with the central node, suggesting both positive and negative interdependencies. - **Auditability** (O4) and **Fairness** (O5): Directly influenced by the central concept with no feedback loops. - **Traceability** (O2, O9, O12) and **Understandability** (O2, O5, O12, O15): Feed into the central node, emphasizing their foundational role in enabling transparency and explainability. ### Key Observations 1. **Bidirectional Privacy Relationship**: The "+/-" arrow between Privacy and the central node suggests a complex interplay, where transparency/explainability may both enhance and conflict with privacy objectives. 2. **Overlap in Objectives**: - O2 (Traceability/Understandability), O4 (Auditability/Trustworthiness), and O5 (Fairness/Understandability) appear in multiple components, indicating shared goals. 3. **Centrality of Transparency/Explainability**: The green node acts as a hub, integrating inputs from Traceability and Understandability while influencing Trustworthiness, Privacy, Auditability, and Fairness. ### Interpretation The diagram positions **Transparency and Explainability** as a core enabler of ethical AI systems, with: - **Positive Feedback Loops**: Traceability and Understandability reinforce the central concept, creating a self-reinforcing cycle for accountability. - **Privacy Trade-offs**: The mixed influence on Privacy highlights potential conflicts between explainability (e.g., revealing decision logic) and data protection (e.g., anonymization). - **Interconnected Standards**: Shared objectives (O2, O4, O5) across components suggest that frameworks for transparency must align with broader ethical principles like fairness and auditability. This structure implies that achieving robust transparency requires balancing technical traceability with human-centric understandability, while navigating privacy constraints. </details> Figure 6: Key qualitative standards (O1-O15) related to explainability and transparency in AI systems, addressing user comprehension, interpretability, and traceability. [28] The growth of AI systems’ explainability and transparency is facilitated by their understandability. When discussing the significance of understandability, the transparency guidelines addressed three points: 1) ensuring that people comprehend the AI system’s behavior and the methods for using it (O5, O12); 2) communicating in an intelligible manner the locations, purposes, and methods of AI use (O15); and 3) making sure people comprehend the distinction between real AI decisions and those that AI merely assists in making (O2) [28]. Thus, by guaranteeing that people are informed about the use of AI in a straightforward and comprehensive manner, understandability promotes explainability and transparency. The necessity of tracking the decisions made by AI systems is highlighted by traceability in transparency requirements (O2, O12) [28]. In order to ensure openness, Organization O12 also noted how crucial it is to track the data utilized in AI decision-making. ### 1.5 Transparency Vs Explainability in AI Explainability and transparency are similar concepts [29]. According to McLarney et al. [30], a transparent AI necessitates that ”Basic elements of data and decisions must be available for inspection during and after AI use.” Transparency is achieved when users have access to their data or can understand how decisions are made. On the other hand, explainability seeks to reveal the reasons for AI’s successes or failures and demonstrate how it utilizes the knowledge and judgment of those it affects. It provides a rational justification for the actions of the AI. Users must clearly know what data is collected, how the AI interprets this data, and how it produces reliable outcomes for each affected individual. This straightforward explanation overlooks the challenges we face when trying to clarify ”black box” algorithms, the context that may be omitted, and the accuracy needed when offering understandable explanations to customers. Therefore, the question arises: is having minimal explainability preferable to having none at all? [30]. Additionally, the belief that explanations can adequately address the dynamic nature of the rich information ecosystem and the appropriateness of managing anomalies are also vital factors to consider. <details> <summary>x4.jpg Details</summary>  ### Visual Description ## Screenshot: Apology Message for Abrupt Conversation Termination ### Overview The image is a screenshot of a text-based message with a black background and white text. The message apologizes for an abrupt conversation ending, explains potential causes (technical issues, user disengagement), and lists three bullet points indicating signs of user disinterest. ### Content Details **Main Text**: "I’m sorry to hear that our conversation ended abruptly. That was not my intention. Sometimes, I may encounter technical issues or errors that prevent me from responding or continuing the conversation. Other times, I may detect some signals or cues that indicate that the user is not interested or satisfied with the conversation, such as:" **Bullet Points**: 1. The user does not reply for a long time or leaves the chat box. 2. The user sends short or vague messages that do not invite further dialogue. 3. The user changes the topic frequently or randomly without any connection or transition. ### Key Observations - The message is structured as a formal apology with explanatory reasoning. - The bullet points are presented as objective criteria for detecting user disengagement. - No numerical data, charts, or diagrams are present. ### Interpretation This message serves as a standardized response to users when a conversation terminates unexpectedly. It aims to: 1. **Acknowledge limitations**: Attribute abrupt endings to technical constraints or AI detection of disengagement. 2. **Educate users**: Provide transparency about how the AI interprets user behavior (e.g., silence, short replies, topic shifts). 3. **Mitigate frustration**: By framing the termination as a mutual or system-driven outcome rather than a user error. The bullet points imply a behavioral analysis framework, suggesting the AI uses heuristics to infer user satisfaction. However, the lack of quantitative data (e.g., thresholds for "long time" or "frequent" topic changes) leaves room for ambiguity in how these criteria are operationalized. </details> Figure 7: Output from the Bing search engine’s conversation feature explaining a failure. a partial screenshot taken using an Android smartphone on March 2, 2023. [17] It’s interesting to note that although certain AI algorithms evaluate data automatically, more and more AI systems are made to explain how their algorithms operate and the logic behind specific choices [17]. For instance, the conversation mode of the Bing search engine provides succinct explanations of its operation (Fig. 7). Sometimes, end users might find these explanations sufficient, but other times, they would be perplexed as to how an AI came to a particular conclusion or acted in a particular manner. When individuals are more confused by the explanation given, it is unrealistic to expect them to become more computer-literate [17]. Instead, we must improve the justification of the AI system. ### 1.6 Definition of Trustworthiness in Artificial Intelligence Creating trustworthy AI systems requires a careful strategy that looks at organizational, ethical, and technical factors. The first step is to set clear standards for trustworthiness. These standards should include accountability, security, privacy, transparency, fairness, and ethical behavior. Using high-quality, unbiased data and clear algorithms that explain AI decisions is essential. Strong security measures and privacy practices protect sensitive information from cyberattacks. It’s important to create accountability frameworks and follow ethical guidelines to ensure responsible AI use. By focusing on user needs and constantly monitoring and updating the systems, AI can stay reliable over time. Applying these principles across all stages of the AI process allows organizations to develop systems that are explainable, equitable, ethical, and robust, which fosters stakeholder and user trust. <details> <summary>extracted/6367585/image/eai4.png Details</summary>  ### Visual Description ## Flowchart: Trustworthy AI Framework ### Overview The image depicts a hierarchical flowchart centered on "Trustworthy AI" as the core concept. Three primary branches radiate from the central node, each representing a distinct ethical dimension: **Ethics of Algorithms**, **Ethics of Data**, and **Ethics of Practice**. Each branch contains subpoints detailing specific ethical principles. ### Components/Axes - **Central Node**: "Trustworthy AI" (green oval, top-center). - **Branches**: 1. **Ethics of Algorithms** (left rectangle, peach-colored). - Subpoints: - Respect for Human Autonomy - Prevention of Harm - Fairness - Explicability 2. **Ethics of Data** (center rectangle, peach-colored). - Subpoints: - Human-centred - Individual Data Control - Transparency - Accountability - Equality 3. **Ethics of Practice** (right rectangle, peach-colored). - Subpoints: - Responsibility - Liability - Codes and Regulations ### Detailed Analysis - **Ethics of Algorithms**: Focuses on algorithmic behavior, emphasizing human-centric design (autonomy, harm prevention, fairness, and transparency in decision-making). - **Ethics of Data**: Addresses data governance, prioritizing individual rights (control, transparency, accountability) and societal equity (equality). - **Ethics of Practice**: Highlights operational accountability (responsibility, liability) and adherence to regulatory frameworks (codes and regulations). ### Key Observations - The flowchart uses a **top-down structure**, with the central concept ("Trustworthy AI") as the root node. - **Ethics of Data** has the most subpoints (5), suggesting greater complexity or emphasis on data-related principles. - **Ethics of Practice** has the fewest subpoints (3), potentially indicating a narrower scope compared to the other branches. - All subpoints are listed in **vertical alignment** under their respective branch titles, with no numerical or quantitative data present. ### Interpretation This flowchart outlines a **comprehensive ethical framework** for AI systems, structured around three pillars: 1. **Algorithmic Integrity**: Ensuring AI systems respect human values (autonomy, fairness) and avoid harm. 2. **Data Ethics**: Prioritizing individual rights and societal equity in data handling. 3. **Practical Accountability**: Enforcing responsibility and compliance with regulations. The absence of numerical data or visual trends suggests the diagram is **conceptual**, designed to guide policy or design rather than measure outcomes. The hierarchical layout implies that "Trustworthy AI" is achieved through the integration of these three ethical dimensions, with no single branch being subordinate to another. The emphasis on "Human-centred" and "Equality" in the data ethics branch highlights a societal focus, while "Codes and Regulations" in practice underscores legal compliance as a foundational requirement. </details> Figure 8: The three key components of XAI: Algorithmic Ethics, Data Ethics, and Practice Ethics. [31] The three elements illustrated in Figure 8 —algorithmic ethics, data ethics, and practice ethics—intersect to create responsible AI. These elements define a data-centered way to handle ethical issues [31]. However, several open challenges still remain with respect to dealing with ethical issues in AI systems. In the work [31], authors give the vision of Trustworthy AI, which mentions that: 1. Human agency and oversight: AI systems have to enable human freedom. They need to facilitate user choice, safeguard fundamental rights, and enable human control. This will assist in developing an equitable and just society. 1. Security and technical robustness: Security and technical proficiency are crucial to prevent damage. In order to enable an AI system to operate efficiently and reduce risks, its creators ought to consider potential risks when designing it. They range from environmental alterations where the system will operate to attacks by malicious individuals. 1. Data protection and data governance: Privacy is a fundamental right that has been highly compromised with the vast amounts of data that artificial intelligence systems gather. It is necessary to protect individual privacy to prevent potential harm. For this purpose, robust data governance must be in place. This includes making sure that the information being utilized is precise and applicable. Furthermore, there is a necessity to establish definite rules for data access and how data must be treated while maintaining the integrity of privacy. 1. Transparency: Explainability and transparency are pretty much dependent on each other. The key objective is to make data, technology, and business models clear. In today’s age, which is the age of pervasive technology, transparency has become a must. It aids customers in comprehending the huge volumes of data collected and the ensuing benefits. 1. Fairness, diversity, and nondiscrimination: Including several voices in AI systems is vital to achieve XAI. We need to involve all individuals who may be affected to ensure equal treatment and access. Fairness and this requirement come hand in hand. 1. Social and environmental welfare: We have to think about the environment and community in seeking justice and doing no harm. We should finance research on AI solutions to global issues. This will make AI systems environmentally friendly and sustainable. AI is supposed to be for the good of all people, including future generations.. 1. Accountability: Accountability and fairness are essential in the context of AI. We need to have systems of holding AI systems accountable for their actions and generated results. Accountability needs to be a constituent part of AI development, deployment, and use, both during and following the activities. ### 1.7 Impact of XAI on Zero Trust Architecture (ZTA) Zero Trust Architecture (ZTA) is a security system that always checks every request, no matter where it comes from, and does not assume trust automatically. Explainable AI (XAI) helps ZTA by ensuring that decisions made by AI in security are clear and reasonable. XAI is especially useful in identity verification, access control, and spotting unusual behavior. AI models analyze how users behave and identify any suspicious activities. By adding explainability, security analysts can better understand and confirm AI-driven security rules. This reduces false alarms and speeds up response times. For example, AI-driven network monitoring systems that use ZTA principles can explain why a specific access attempt looks suspicious. This explanation builds trust in automated cybersecurity decisions [32]. ### 1.8 An Overview of Necessities for Reliable AI Despite heated societal discussions, the requirements for trustworthy AI remain ambiguous and are handled inconsistently by numerous organizations and organizations. Globally, accountability, explainability, verifiability, and fairness are all part of the Fairness, Responsibility, Accuracy, Verifiability, and Accountability in Machine Learning (FAT-ML) principles [33]. Explainability, fairness, privacy, and robustness are just a few of the many needs that will be examined in this study (Table 1). Table 1: Conditions necessary for trustworthy artificial intelligence (AI) | Explainability | To help consumers comprehend, the method by which the AI model generates its output might be demonstrated. | | --- | --- | | Fairness | Regardless of certain protected variables, the AI model’s output can be shown. | | Privacy | It is feasible to prevent issues with personal data that might arise while the AI is being developed. | | Robustness | The AI model can fend against outside threats while continuing to operate correctly. | ## 2 XAI Vs AI XAI improves AI systems by focusing on transparency, clear explanations, and accountability. It offers understandable reasons for decisions, which helps users trust the system and makes it easier to assess fairness compared to traditional ”black box” methods. The main difference between reliable XAI and traditional AI is how they make decisions. While AI can give accurate forecasts or suggestions, reliable XAI emphasizes the need to explain the steps that lead to these results. Clear explanations from XAI systems allow users to judge the fairness and reliability of AI-generated outcomes. To improve security and maintain transparency in AI-driven cybersecurity, we need to integrate XAI into Zero Trust Architecture (ZTA). When explainability methods clarify why certain decisions are made, people can better understand and trust the AI-driven access control and behavioral analytics in ZTA. As we face compliance and operational challenges, future cybersecurity frameworks will rely more on AI automation. It will be essential to ensure that these AI systems can be easily explained [34]. XAI focuses on more than just providing explanations; it also considers ethical issues. AI development processes that follow the principles of Fairness, Accountability, and Transparency (FAT) help ensure that AI systems meet ethical and legal standards. By prioritizing ethical standards, XAI aims to reduce biases, discrimination, and other harmful effects of AI technology. Trustworthy AI is an approach that emphasizes user safety and transparency. Responsible AI developers clearly explain to clients and the public how the technology works, what it is meant for, and its limitations, since no model is perfect. Table 2: Seven Requirements to Meet in Order to Develop Reliable AI | Human Authority and Supervision | Artificial intelligence technology ought to uphold human agency and basic rights, instead of limiting or impeding human autonomy. | The right to get human assistance | Recital 71, Art 22 | | --- | --- | --- | --- | | Robustness and Safety | Systems must be dependable, safe, robust enough to tolerate mistakes or inconsistencies, and capable of deviating from a totally automated decision | Art 22 | | | Data Governance and Privacy | Individuals should be in total control of the information that is about them, and information about them should not be used against them | Notification and information access rights regarding the logic used in automated processes | Art 13, 14, and 15 | | Transparency | Systems using AI ought to be transparent and traceable | The right to get clarification | Recital 71 | | Diversity and Fairness | AI systems have to provide accessibility and take into account the whole spectrum of human capacities, requirements, and standards | Right to not have decisions made only by machines | Art 22 | | Environmental and Social Well-Being | AI should be utilized to promote social change, accountability, and environmental sustainability | Accurate knowledge regarding the importance and possible consequences of making decisions exclusively through automation | Art 13, 14, and 15 | | Accountability | Establishing procedures to guarantee that AI systems and their outcomes are held accountable is essential | Right to be informed when decisions are made only by machines | Art 13, 14 | ## 3 Applications of XAI Authentic XAI has numerous uses in sectors where accountability, interpretability, and transparency are essential. XAI can provide an explanation for a diagnosis or therapy recommendation in medical diagnosis and recommendation systems. Financial institutions can employ XAI for risk assessment, fraud detection, and credit scoring. XAI can help attorneys with contract analysis, lawsuit prediction, and legal research. In autonomous vehicles, XAI plays a significant role in providing context for the decisions made by the AI systems, particularly in high-stakes scenarios such as accidents or unanticipated roadside incidents. XAI can be applied to process optimization, predictive maintenance, and quality control in manufacturing settings. By offering justifications for automated responses or suggestions in chatbots and virtual assistants, XAI can improve customer service. By providing an explanation for the recommendations and assessments made by adaptive learning systems, XAI can help with individualized learning. By providing an explanation for the recommendations and assessments made by adaptive learning systems, XAI can help with individualized learning. We shall concentrate on a few particular applications in this section and go into detail about them. ### 3.1 Application of XAI in Medical Science The field of artificial intelligence (AI) is rapidly growing on a global scale, particularly in healthcare, which is a hot topic for research [35]. There are numerous opportunities to utilize AI technology in the healthcare sector, where the well-being of individuals is at stake, due to its significant relevance and the vast amounts of digital medical data that have been collected [36]. AI has enabled us to perform tasks quickly that were previously unfeasible with traditional technologies. The trustworthiness and openness of AI systems are becoming increasingly important, especially in areas like healthcare. As AI is used more in medical decision-making, people are worried about how reliable and understandable its results are. These worries highlight the need to evaluate AI models carefully to make sure their predictions are based on important and verifiable factors. In critical situations like medicine, proving that AI systems are credible is vital for their safe and effective use. In the medical field, clinical decision support systems (CDSS) utilize AI technology to assist healthcare professionals with critical tasks such as diagnosis and treatment planning [37]. While these systems aim to support healthcare practitioners, misuse can have severe consequences in situations where lives are at risk. For example, false alarms, which are common in scenarios involving urgent patients, can lead to exhaustion among medical personnel. The study [38] significantly contributes to medical skin lesion diagnostics in several ways. First, it modifies an existing explainable AI (XAI) technique to boost user confidence and trust in AI systems. This change involves developing an AI model that can distinguish between different types of skin lesions. The study uses synthetic examples and counter-examples to create explanations that highlight the key features influencing classification decisions. The research [38] trains a deep learning classifier with the ISIC 2019 dataset using the ResNet architecture. This allows professionals to use the explanations to reason effectively. Overall, the study’s main contributions lie in its refinement and evaluation of the XAI technique in a real-world medical setting, its analysis of the latent space, and its thorough user study to assess how effective the explanations are, particularly among experts in the field. This research paper [39] discusses how to recognize brain tumors in MRI images using two effective algorithms: fuzzy C-means (FCM) and Artificial Neural Network (ANN). The authors aim to make the tumor segmentation process more understandable and improve accuracy in identifying tumors. Their main goal is to enhance tools that help doctors diagnose brain tumors more accurately. This research offers two key benefits. First, it helps identify brain cancers in medical images more precisely, which is crucial for early diagnosis and treatment. Second, by incorporating XAI principles into the segmentation process, the researchers make their models’ decisions clearer and easier to understand for patients and medical experts. In summary, this increased clarity boosts the overall trust and acceptance of AI-driven systems in medical image analysis within clinical settings. This study [40] discusses how AI and machine learning can help diagnose whole slide images (WSIs) in pathology. While AI can improve accuracy and efficiency, concerns exist about its reliability because it can be hard to understand. To address these issues, the article suggests using explainable AI methods, which help clarify how AI makes decisions. By adding XAI, pathology systems become more transparent and trustworthy, especially for critical tasks like diagnosing diseases. The study also introduces HistoMapr-Breast, a software tool that uses XAI to assist with breast core biopsies. A recent study examines the importance of making sure AI systems in healthcare are accurate and strong, especially regarding how easy they are to understand and how well they can resist attacks [41]. As AI becomes more common in medical settings, it’s crucial to verify that the predictions these systems make rely on trustworthy features. To tackle this challenge, researchers have proposed various methods to improve model interpretability and explainability. The study shows that adversarial attacks can affect a model’s explainability, even when the model has strong training. Additionally, the authors introduce two types of attack classifiers: one that tells apart harmless and harmful inputs, and another that determines the nature of the attack. This research paper [42] looks at explainable machine learning in cardiology. It discusses the challenges of understanding complex prediction models and how these models affect important healthcare decisions. The study explains the main ideas and methods of explainable machine learning, helping cardiologists understand the benefits and limitations of this approach. The goal is to improve decision-making in clinical settings by offering guidance on when to use easy-to-understand models versus complex ones. This can help improve patient outcomes while ensuring accountability and transparency in predictions. Figure 9 shows a decision tree created from the predictions of a random forest model. This global tree diagram illustrates how the random forest works overall. By following a patient’s path through the tree, individual predictions can be examined. This type of explanation is beneficial because it clarifies both the general functioning of the model and the reasoning behind specific predictions. Decision trees are suitable for fields like cardiology because they use rule-based reasoning similar to clinical decision guidelines. <details> <summary>extracted/6367585/image/eai5.png Details</summary>  ### Visual Description ## Flowchart: Heart Disease Risk Assessment Model ### Overview The image presents a bifurcated decision tree model comparing **Global** and **Local** heart disease risk assessment pathways. Both sections use color-coded nodes (green for "Normal" pathways, pink for "Abnormal") to represent conditional probabilities of heart disease based on clinical parameters. The Local section emphasizes a specific high-risk scenario through a circled probability value. ### Components/Axes 1. **Decision Nodes** (Gray/Blue Rectangles): - Thallium Stress Test (Normal/Abnormal) - ST Depression (≤0.7/≥0.7) - Age (≤54.5/≥54.5) - Cholesterol (≤228/≥228) - Chest Pain Type (Typical/Non-Typical Angina) - Max Heart Rate (≤120/≥120) - ST Depression (≤2.1/≥2.1) 2. **Probability Nodes** (Circular): - Green: Low-risk probabilities (0.069–0.318) - Pink: High-risk probabilities (0.692–1.000) - Special Case: Circled 0.069 in Local section 3. **Flow Structure**: - Top-to-bottom hierarchy with conditional branching - Divergent paths based on binary thresholds - Convergent endpoints at probability values ### Detailed Analysis #### Global Section (Top) - **Thallium Stress Test**: - Normal: Leads to Age assessment - Abnormal: Triggers ST Depression evaluation - **Age**: - ≤54.5: Max Heart Rate assessment - ≥54.5: ST Depression evaluation - **Key Paths**: - Cholesterol >228 + Non-Typical Angina: 0.916 probability - Max Heart Rate >120: 0.069 probability - ST Depression >2.1: 0.875 probability #### Local Section (Bottom) - **Modified Thallium Stress Test**: - Normal: Directly branches to Age - Abnormal: Maintains ST Depression evaluation - **Critical Path**: - Max Heart Rate >120: Circled 0.069 probability (highlighted anomaly) - **Divergent Values**: - Chest Pain Type: 0.286 (Global) vs 1.000 (Local) - ST Depression >2.1: 0.875 (both sections) ### Key Observations 1. **Probability Extremes**: - 1.000 probability in Global section (Cholesterol >228 + Non-Typical Angina) - 0.069 circled value in Local section (Max Heart Rate >120) 2. **Threshold Sensitivity**: - ST Depression thresholds differ between sections (0.7 vs 2.1) - Age cutoff at 54.5 years creates bifurcation point 3. **Color Coding Consistency**: - Green nodes consistently represent lower-risk pathways - Pink nodes indicate higher-risk outcomes ### Interpretation This flowchart demonstrates a **risk stratification model** where clinical parameters interact multiplicatively to predict heart disease probability. The **Local section's circled 0.069 value** suggests a critical threshold for Max Heart Rate >120, potentially indicating an outlier or high-priority clinical scenario. The **Global model** shows higher probabilities (up to 1.000) for combinations involving Non-Typical Angina and elevated Cholesterol, while the **Local model** emphasizes Age-related risks. The divergence in Chest Pain Type probabilities (0.286 vs 1.000) between sections implies different risk weighting schemes for this parameter. The model's binary threshold approach (e.g., ST Depression ≤0.7 vs ≥0.7) simplifies clinical decision-making but may overlook nuanced risk gradients. The consistent use of green/pink color coding across both sections reinforces the model's visual interpretability for clinicians. </details> Figure 9: A model using random forests predicts heart disease by analyzing both local and global decision trees. The global diagram starts by examining whether a patient’s thallium stress test results are normal. If the test shows a problem, the model looks at the patient’s ST depression next. The local graphic shows the specific pathway a patient took in the model, explaining the reasons for their individual prediction. For example, a patient under 54.5 years old, with a maximum heart rate that is high and normal thallium stress test results, has a very low chance of having heart disease [42]. Figure 10 shows the LIME explanations for our heart failure model’s two local predictions. The authors explain how these predictions serve as a clinical decision support tool in Epic, which is an electronic health record designed for doctors (Epic Systems Corporation, Verona, Wisconsin, USA). This kind of explanation helps to clarify the clinical factors that affect each prediction. Importantly, this type of explanation can be added to an EHR, which may improve the practical use of a complex model by making forecasts and clear explanations easy to integrate into clinical work. <details> <summary>x5.jpg Details</summary>  ### Visual Description ## Medical Risk Assessment Report: Heart Disease Probability Analysis ### Overview The image presents a clinical decision support system interface displaying heart disease risk probabilities for six patients. It combines a tabular patient list with detailed risk factor analyses for two representative cases. The interface uses color-coded probability indicators (red-high, yellow-medium, green-low) and provides explanatory factors for each prediction. ### Components/Axes 1. **Left Table Structure** - **Headers**: - "Patient Name" (left column) - "Probability of Heart Disease" (right column) - **Data Points**: - Six redacted patient entries with probabilities: 94%, 79%, 55%, 49%, 31%, 26% - Color-coded ovals: Red (94%, 79%), Yellow (55%, 49%), Green (31%, 26%) 2. **Right Panels** - **Top Panel** (79% High Risk): - Header: "Probability of Heart Disease" (purple header) - Timestamp: "7/21/2021 20:26" - Risk Level: "High" (black text on red background) - Contributing Factors: - Thallium Stress Test: Normal - Number of Major Vessels: 1 - Exercise Induced Angina: No - Chest Pain Type: Typical Angina - Max Heart Rate Achieved: 174 - Sex: Male - **Bottom Panel** (26% Low Risk): - Header: "Probability of Heart Disease" (purple header) - Timestamp: "7/21/2021 20:26" - Risk Level: "Low" (black text on green background) - Contributing Factors: - Thallium Stress Test: Normal - Max Heart Rate Achieved: 131 - Number of Major Vessels: 1 - Sex: Male - ST Depression: 0.10 - Age: 69 ### Detailed Analysis 1. **Probability Distribution** - High Risk (≥70%): 2 patients (94%, 79%) - Medium Risk (50-69%): 2 patients (55%, 49%) - Low Risk (<50%): 2 patients (31%, 26%) 2. **Risk Factor Correlations** - **High Risk Case**: - Elevated max heart rate (174 bpm) - Presence of typical angina - Male sex - Single major vessel involvement - **Low Risk Case**: - Moderate max heart rate (131 bpm) - Minimal ST depression (0.10 mm) - Older age (69 years) - Single major vessel involvement 3. **Temporal Consistency** - Both detailed assessments share identical timestamps (7/21/2021 20:26), suggesting synchronized system calculations. ### Key Observations 1. **Color-Coding Accuracy** - Red ovals consistently represent probabilities ≥70% - Yellow ovals represent 50-69% probabilities - Green ovals represent <50% probabilities 2. **Factor Weighting Patterns** - Max heart rate >150 bpm appears in high-risk cases - ST depression >0.05 mm correlates with increased risk - Male sex appears in both high and low-risk cases but with different factor combinations 3. **Anomalies** - Patient with 55% probability (yellow) shows no explicit risk factors in the detailed panels - 31% probability case includes ST depression as a mitigating factor ### Interpretation The system demonstrates a multi-factorial risk assessment model that integrates: 1. **Physiological Parameters**: Max heart rate, ST depression, and vessel count 2. **Clinical Findings**: Angina presence and chest pain type 3. **Demographic Data**: Age and sex The high-risk case (79%) exemplifies classic coronary artery disease presentation with typical angina and elevated heart rate, while the low-risk case (26%) shows age-related changes without significant ischemia. The model appears to use Bayesian probability calculations, updating risk estimates based on new test results (Thallium Stress Test) and clinical findings. The identical timestamps suggest real-time updating capability, though the lack of patient identifiers limits longitudinal analysis. The system's color-coded interface enables rapid clinical decision-making, with red alerts requiring immediate intervention protocols. </details> Figure 10: Heart disease prediction explanation produced using Local Interpretable Model-Agnostic Explanations (LIME). This illustration shows how clinical decision assistance can be integrated into an Epic electronic health record by means of a local explanation utilizing the LIME algorithm. To help clinicians identify patients who are likely to be at a high risk of heart disease, probabilities are color-coded. To improve the predictability and actionability of the results for doctors, the clinical factors that are most significant to the prediction are shown on the right [42]. For medical AI to work reliably and be widely used, we need to do a lot of research and reach an agreement on important features like explainability, fairness, privacy, and reliability [33]. We must meet clear requirements and standards in any healthcare setting that uses AI, and we need to update these regularly. Additionally, we should establish laws that clarify who is responsible if something goes wrong with a medical AI whether that’s the designers, researchers, healthcare workers, or patients [43]. ### 3.2 Explainability and Interpretability of Autonomous Systems Explainability and interpretability are crucial concepts in the context of autonomous systems, referring to the ability to understand the decisions and behaviors of these systems. Explainability involves an autonomous system’s capacity to provide clear justifications for its actions and choices [44]. This clarity is essential for fostering acceptance and confidence in AI systems, especially in critical fields such as banking, healthcare, and autonomous vehicles. While explainability and interpretability are closely related, interpretability focuses more on understanding the internal mechanisms and processes of the autonomous system [45]. An interpretable system offers users insight into the factors and criteria that influence its decision-making, enabling them to grasp how the system arrived at its conclusions. The research paper The research article [18] focuses on trust and dependability in autonomous systems. Autonomous systems have the potential for system operation, rapid information dissemination, massive data processing, working in hazardous environments, operating with greater resilience and tenacity than humans, and even astronomical examination [46], [47]. Following years of research and development, today’s automated technologies represent the peak of progress in computer recognition, responsive systems, user-friendly interface design, and sensing automation. According to [44], the global market for automotive intelligent hardware, operations, and innovation is projected to grow significantly, increasing from $1.25billionin2017to$ 28.5 billion by 2025. Intel’s research on the expected benefits of autonomous vehicles indicates that implementing these technologies on public roads could reduce annual commute times by 250 million hours and save over 500,000 lives in the United States between 2035 and 2045 [44]. Modern cars utilize artificial intelligence for various functions, including intelligent cruise control, automatic driving and parking, and blind-spot detection (Figure 11). Authors [18] describe the challenges of autonomous systems, like, people sometimes tend to be overly excited about the potential of new ideas and ignore, or at least appear to be unaware of, the potential drawbacks of cutting-edge developments. Even in the early stages of robotics and autonomous system implementation, humanity preferred to put up with faulty goods and services, but they have gradually come to understand the importance of trustworthy and dependable autonomous systems. Numerous examples have demonstrated how operators’ use of automation is greatly impacted by trustworthiness. <details> <summary>x6.jpg Details</summary>  ### Visual Description ## Dashboard Display with Urban Scene Annotations ### Overview The image shows a vehicle's dashboard display with a digital interface and a view through the windshield. The dashboard includes German text, numerical data, and graphical elements. The windshield view depicts an urban street scene with annotated objects (cyclist and pedestrian) using colored bounding boxes and directional arrows. ### Components/Axes 1. **Dashboard Display**: - **Speedometer**: Shows "30" (gray) and "32" (orange) km/h. - **Digital Text**: - "Stadtsverkehr" (urban traffic) - "Bremsvorgang aktiv" (braking active) - "Achtung!" (attention) - "Energieleistung" (energy performance) - **Graph**: A red wavy line labeled "Energieleistung" (energy performance). - **Time**: "15:34 Uhr" (3:34 PM). 2. **Windshield View**: - **Colored Bounding Boxes**: - Red box around a cyclist. - Blue box around a pedestrian. - **Directional Arrows**: - Blue arrows indicating cyclist movement. - Blue arrows indicating pedestrian movement. - **Scene Elements**: - Buildings, parked cars, and a clear sky. ### Detailed Analysis - **Speed**: The vehicle is traveling at 32 km/h (orange value), with a maximum speed of 30 km/h (gray value). - **Alerts**: "Bremsvorgang aktiv" (braking active) and "Achtung!" (attention) suggest the vehicle's systems are engaged in collision avoidance or hazard response. - **Energy Performance Graph**: The red wavy line indicates fluctuating energy levels, possibly related to regenerative braking or power consumption. - **Annotations**: - The cyclist (red box) is positioned ahead of the vehicle, with arrows showing forward motion. - The pedestrian (blue box) is on the sidewalk, with arrows indicating lateral movement. ### Key Observations - The cyclist is closer to the vehicle than the pedestrian, with both annotated for tracking. - The dashboard emphasizes energy efficiency ("Energieleistung") and active braking, suggesting an electric or hybrid vehicle. - The time "15:34 Uhr" aligns with typical urban traffic hours. ### Interpretation The image demonstrates an advanced driver-assistance system (ADAS) in operation. The colored bounding boxes and arrows indicate real-time object detection and trajectory prediction, critical for autonomous driving or semi-autonomous features. The dashboard alerts ("Bremsvorgang aktiv") and energy performance graph highlight the system's focus on safety and efficiency. The urban setting ("Stadtsverkehr") implies frequent stops and interactions with vulnerable road users (cyclists, pedestrians). The absence of a visible legend for the colored boxes suggests the system uses standardized color coding (red for immediate hazards, blue for secondary objects). The energy performance graph's wavy line may correlate with the cyclist's proximity, indicating adaptive energy management based on traffic conditions. </details> Figure 11: An automated vehicle that provides a clear and understandable rationale for its decisions at that moment serves as a prime example of explainable AI in automated driving [18]. When AI has become prevalent in autonomous vehicle (AV) operations, user trust has been identified as a major issue that is essential to the success of these operations. XAI, which calls for the AI system to give the user explanations for every decision it makes, is a viable approach to fostering user trust for such integrated AI-based driving systems [48]. This work develops explainable Deep Learning (DL) models to improve trustworthiness in autonomous driving systems, driven by the need to improve user trust and the potential of innovative XAI technology in addressing such requirements. The main concept of this [48] research is to frame the decision-making process of autonomous vehicles (AVs) as an image-captioning task, generating textual descriptions of driving scenarios to serve as understandable explanations for humans. The proposed multi-modal deep learning architecture, shown in Figure 12, utilizes Transformers to model the relationship between images and language, generating meaningful descriptions and driving actions. Key contributions include improving the AV decision-making process for better explainability, developing a fully Transformer-based model, and outperforming baseline models. This results in enhanced user trust, valuable insights for AV developers, and improved interpretability through attention mechanisms and goal induction. <details> <summary>extracted/6367585/image/eai8.png Details</summary>  ### Visual Description ## Diagram: Multimodal Image-Language Processing Pipeline ### Overview The diagram illustrates a technical architecture for processing visual and linguistic data. It consists of three interconnected modules: an **Image Feature Encoder**, an **Attention Network**, and a **Language Decoder**. The pipeline transforms an input image into a sequence of obstacle predictions using feature extraction, attention mechanisms, and recurrent neural networks. --- ### Components/Axes #### Image Feature Encoder - **Input Image**: A street scene with buildings, vehicles, and a crosswalk. - **Resized Image**: Processed by a **Resnet Mobile net** feature extractor. - **Image Features**: Output as a 3D tensor (`s x w x h`), representing spatial and channel dimensions. #### Attention Network - **Attention Score**: Computed via a weighted sum of image features (`s x f`). - **Attended Features**: Weighted combination of original features using attention scores (colors: orange, yellow, light yellow). - **Flow**: Image features → Attention Score → Attended Features. #### Language Decoder - **LSTM Layers**: Two sequential LSTM units (`h₀`, `h₁`) process attended features. - **Output Sequence**: Generated obstacles (`y₁`, `y₂`, ...) prefixed with `<SOS>` (Start of Sentence). - **Flow**: Attended Features → LSTM → Obstacles. --- ### Detailed Analysis 1. **Image Feature Encoder**: - Input image is resized and fed into a **Resnet Mobile net** to extract spatial features (`s x w x h`). - Features are flattened into a 2D matrix for further processing. 2. **Attention Network**: - Attention scores are computed (orange/yellow gradients) to prioritize relevant image regions. - Attended features are generated by element-wise multiplication of original features and attention scores. 3. **Language Decoder**: - LSTM layers (`h₀`, `h₁`) process attended features sequentially. - Output sequence starts with `<SOS>` and predicts obstacles (`y₁`, `y₂`, ...) autoregressively. --- ### Key Observations - **Attention Mechanism**: Highlights critical image regions (e.g., vehicles, pedestrians) for obstacle detection. - **LSTM Hierarchy**: Two LSTM layers suggest hierarchical temporal modeling of obstacle sequences. - **Color Coding**: Attention scores use a gradient (orange > yellow > light yellow) to denote importance. --- ### Interpretation This architecture integrates **computer vision** and **natural language processing** for scene understanding. The attention network acts as a bridge, focusing the language model on salient image features. The use of Resnet Mobile net ensures lightweight feature extraction for mobile deployment, while LSTMs handle sequential obstacle prediction. The diagram emphasizes **cross-modal alignment**, where visual and linguistic data are jointly optimized for tasks like autonomous navigation or image captioning. </details> Figure 12: The Transformers-based multi-modal deep learning architecture that is being suggested [48] This research [49] aims to investigate the integration of XAI into autonomous vehicular systems to improve transparency and human trust. It delves into the functioning of multiple inner vehicle modules, emphasizing the importance of understanding the vehicle’s decision-making processes for user credibility and reliability. The main contribution lies in introducing XAI to the domain of autonomous vehicles, showcasing its role in fostering trust, and highlighting advancements through comparative analysis. The output comprises the creation of visual explanatory techniques and an intrusion detection classifier, which show considerable advances over previous work in terms of transparency and safety in autonomous transportation systems. ### 3.3 Applications of XAI for Operations in the Industry The process industry is a subset of businesses that manufacture items from raw materials (not components) using formulae or recipes. Given the magnitude and dynamic nature of operations in the process sector, it becomes evident that the next great step ahead will be the capacity for people and AI systems to collaborate to ensure production stability and dependability [50]. AI systems must successfully inform the individuals who share the ecosystem about their objectives, intentions, and findings as the first step toward collaboration. We can hope people to work ”with” automation rather than ”around” it, thanks in part to the systematic approach to XAI. This research [51] focuses on XAI applications in the process industry. The research argues that current AI models are not transparent enough for process industry applications and highlights the need for XAI models that can be understood by human experts. The main contribution is outlining the challenges and research needs for XAI in the process industry. The outcome is to develop XAI models that are safe, reliable, and meet the needs of human users in the process industry. Table 3: Examples of AI applications in process industry operations, including pertinent data, users, and procedures. (RNN = Recurrent Neural Network; KNN = K-Nearest Neighbor; ANN = Artificial Neural Network; SVM = Support Vector Machine; SVR = Support Vector Regression; RF = Random Forest; IF = Isolation Forest) [51] | Reference | Relevant Data | End Users | Application | AI Methods | | --- | --- | --- | --- | --- | | [52], [53], [54] | Process signals | Operator, Process Engineer, Automation engineer | Process monitoring | RNN, KNN | | [55], [56], [57] | Process signals, Alarms, Vibration | Process engineer, Automation engineer, Operator, Maintenance engineer | Fault diagnosis | ANN, SVM, Bayes Classifier | | [58], [59], [60] | Process signals, Acoustic signals | Operator | Event prediction | ANN | | [61], [62], [63] | Process signals | Operator | Soft sensors | SVR, ANN, RF | | [64], [65], [66] | Vibration, Process signals | Operator, Maintenance engineer, Scheduler | Predictive maintenance | RNN, IF | Table 3 shows examples of AI applied to operational activities in the process industry. This table should give an idea of the breadth of use cases, users, relevant data sources, and applicable AI methodologies; however, it is not intended to be a full or systematic examination. ## 4 Future of Trustworthy (XAI) <details> <summary>x7.jpg Details</summary>  ### Visual Description ## Diagram: Human-AI Interaction with Explainable AI Components ### Overview This diagram illustrates a human-AI interaction framework emphasizing explainability and user feedback. It depicts bidirectional communication between an AI system, explainable AI components, and a human user through tangible interfaces. The flow includes decision outputs, user probing capabilities, and iterative feedback loops. ### Components/Axes 1. **AI System** (Top-left box) - Contains "AI System" label - Outputs "Decision output" via rightward arrow - Receives "Human in the loop - feedback" via leftward arrow 2. **Explainable AI** (Bottom-left box) - Contains two stacked gray boxes: - **Decision Explanation**: "User probes the model" - **Decision Explanation**: "Convey a single explanation" 3. **Data Physicalizing & Tangible User Interfaces** (Central vertical box) - Contains bidirectional arrows: - Left arrow labeled "Explanation Interface" - Right arrow labeled "Human in the loop - feedback" 4. **Human Figure** (Far right) - Simple black silhouette representing the user ### Flow Direction - Primary flow: AI System → Decision output → Human - Secondary flows: - Human feedback → AI System - Explainable AI components ↔ Data Physicalizing interfaces ### Key Observations 1. **Bidirectional Communication**: The system emphasizes continuous feedback between AI and human users 2. **Explainability Layers**: Two distinct explanation mechanisms are shown: - Proactive probing capability ("User probes the model") - Simplified explanation delivery ("Convey a single explanation") 3. **Tangible Interface Role**: Acts as a bridge between abstract AI decisions and physical user interaction 4. **Cyclical Improvement**: Feedback loops suggest iterative refinement of AI decisions ### Interpretation This diagram demonstrates a human-centered AI design philosophy where: - **Transparency** is maintained through explainable components - **User Agency** is preserved via probing capabilities - **System Adaptability** is enabled through feedback loops - **Tangibility** transforms abstract AI outputs into actionable user interfaces The separation of explanation mechanisms suggests a dual approach to AI interpretability - allowing both deep technical investigation and simplified understanding. The central role of "Data Physicalizing & Tangible User Interfaces" implies a focus on making AI decisions perceptible through physical/digital interfaces, while maintaining technical transparency for expert users. </details> Figure 13: Assessing the user’s interaction with XAI [27]. Figure 13 illustrates the precise location of each XAI domain and its relationship with the human user. According to [67], many explanations of AI systems tend to be static and convey only a single message. However, explanations alone do not facilitate true understanding [68]. To enhance comprehension, users should have the ability to explore the system through interactive explanations, as most existing XAI libraries currently lack options for user engagement and explanation customization. This represents a promising avenue for advancing the field of XAI [68] and [67]. Additionally, various efforts have been made to improve human-machine collaboration by moving beyond static explanations. Explainable AI offers a way to improve how people interact with AI systems. As AI technology grows, it is crucial to ensure that these systems are accountable and transparent. XAI helps by clarifying how AI models work and building trust among users. We can expect many new developments in XAI. These include making AI models more open, focusing on human-centered designs, ensuring compliance with regulations, and creating hybrid AI systems. XAI will prioritize designs that are easy for users to understand and provide clear explanations. This clarity will help build trust and encourage more people to use AI systems. Regulatory frameworks are likely to require the use of XAI in important areas to ensure accountability and transparency. Future XAI systems will need to be sensitive to context and provide interactive explanations. This will allow people to engage with AI decisions in real time and adapt to different situations. We must also work to improve digital literacy and tackle ethical issues to ensure that AI systems follow moral principles and society’s values, making XAI technologies accessible to everyone. The success of XAI depends on its ability to bridge the communication gap between AI systems and human users, which encourages cooperation, mutual respect, and trust in an increasingly AI-driven world. This study [69] offers a thorough analysis of XAI, focusing on two primary areas of inquiry: general XAI difficulties and research directions, as well as ML life cycle phase-based challenges and research directions. In order to shed light on the significance of formalism, customization of explanations, encouraging reliable AI, interdisciplinary partnerships, interpretability-performance trade-offs, and other topics, the study synthesizes important points from the body of existing literature. The primary contribution is the methodical synthesis and analysis of the body of literature to identify important problems and future directions for XAI research [69]. The research offers a thorough review of the current state of XAI research and provides insightful information for future studies and breakthroughs in the area by structuring the debate around general issues and ML life cycle phases. The primary finding of the study is the identification and clarification of 39 important points that cover a range of issues and potential avenues for future XAI research. The importance of conveying data quality, utilizing human expertise in model development, applying rule extraction for interpretability, addressing security concerns, investigating XAI for reinforcement learning and safety, and taking into account the implications of privacy rights in explanation are just a few of the many topics covered by these points. Furthermore, the paper indicates directions for further research and application by highlighting the potential contributions that XAI may make to a number of fields, including digital forensics, IoT, and 5G. <details> <summary>x8.jpg Details</summary>  ### Visual Description ## Hierarchical Diagram: Challenges and Research Directions of XAI in the Deployment Phase ### Overview The image presents a hierarchical structure outlining key challenges and research directions for Explainable Artificial Intelligence (XAI) during the deployment phase. A central box labeled "Challenges and Research Directions of XAI in the Deployment Phase" connects via lines to 11 sub-topic boxes on the right, organized vertically. ### Components/Axes - **Left Box**: Contains the main title: "Challenges and Research Directions of XAI in the Deployment Phase." - **Right-Side Sub-Topics**: 11 vertically stacked boxes connected to the main title by lines. Sub-topics include: 1. Human-machine teaming 2. XAI and security 3. XAI and reinforcement learning 4. XAI and safety 5. Machine-to-machine explanation 6. XAI and privacy 7. Explainable AI planning (XAIP) 8. Explainable recommendation 9. Explainable agency and explainable embodied agents 10. XAI as a service 11. Improving explanations with ontologies ### Detailed Analysis - **Structure**: The diagram uses a simple hierarchical layout with no numerical data, colors, or legends. All sub-topics are text-based and directly linked to the central theme. - **Textual Content**: All sub-topics are explicitly labeled, with no embedded diagrams or additional annotations. The connections imply a causal or thematic relationship between the main title and each sub-topic. ### Key Observations - The diagram emphasizes **11 distinct research areas** under XAI deployment challenges. - Sub-topics span technical (e.g., "XAI and privacy"), application-specific (e.g., "XAI as a service"), and methodological (e.g., "Improving explanations with ontologies") domains. - No sub-topic is visually prioritized (e.g., no larger font or bold text), suggesting equal emphasis. ### Interpretation This diagram maps the multidisciplinary scope of XAI research in deployment, highlighting technical, ethical, and practical challenges. The absence of quantitative data suggests it is a conceptual framework rather than an empirical study. The interconnected sub-topics imply that addressing XAI deployment requires interdisciplinary collaboration across AI, security, human-computer interaction, and domain-specific applications. The inclusion of "XAI as a service" and "Improving explanations with ontologies" reflects emerging trends toward scalable, interpretable AI systems. The diagram underscores the complexity of deploying XAI systems, where technical robustness (e.g., safety, privacy) must align with user-centric design (e.g., human-machine teaming, recommendations). </details> Figure 14: Issues and Future Research Paths for XAI throughout its Deployment Stage [69]. Deploying machine learning solutions begins the deployment process and continues until we cease utilizing them, possibly even after that. Figure 14 illustrates the XAI research directions and challenges that were explored for this phase. ## 5 Conclusions XAI, or Explainable Artificial Intelligence, is becoming important in many industries because it helps solve key challenges with using AI. As AI becomes more common in our daily lives, understanding how it works is essential. XAI provides tools that help people see and understand how AI models make decisions. The main goal of XAI is to make these models easier to understand. It allows people to look inside the ”black box” of AI and see what affects its decisions. The paper gives a clear overview of the key parts of XAI. It also discusses three main areas where XAI can be applied. Finally, the authors talk about the challenges of using XAI and suggest possible future directions. Acknowledgements The authors would like to express their sincere gratitude to everyone who encourages and appreciates their scientific work. ## Declarations Not applicable ## References - Stephens [2023] Stephens, E.: The mechanical turk: A short history of ‘artificial artificial intelligence’. Cultural Studies 37 (1), 65–87 (2023) - Kaul et al. [2020] Kaul, V., Enslin, S., Gross, S.A.: History of artificial intelligence in medicine. Gastrointest Endosc 92 (4), 807–812 (2020) https://doi.org/10.1016/j.gie.2020.06.040 . Epub 2020 Jun 18 - Buchanan [2005] Buchanan, B.G.: A (very) brief history of artificial intelligence. Ai Magazine 26 (4), 53–53 (2005) - Wang et al. [2021] Wang, L., Liu, Z., Liu, A., Tao, F.: Artificial intelligence in product lifecycle management. The International Journal of Advanced Manufacturing Technology 114, 771–796 (2021) - Shamshiri et al. [2024] Shamshiri, A., Ryu, K.R., Park, J.Y.: Text mining and natural language processing in construction. Automation in Construction 158, 105200 (2024) - Khang et al. [2024] Khang, A., Abdullayev, V., Litvinova, E., Chumachenko, S., Alyar, A.V., Anh, P.: Application of computer vision (cv) in the healthcare ecosystem. In: Computer Vision and AI-Integrated IoT Technologies in the Medical Ecosystem, pp. 1–16. CRC Press, ??? (2024) - Vallès-Peris and Domènech [2023] Vallès-Peris, N., Domènech, M.: Caring in the in-between: a proposal to introduce responsible ai and robotics to healthcare. AI & SOCIETY 38 (4), 1685–1695 (2023) - Biswas et al. [2023] Biswas, A., Abdullah Al, N.M., Ali, M.S., Hossain, I., Ullah, M.A., Talukder, S.: Active learning on medical image. In: Data Driven Approaches on Medical Imaging, pp. 51–67. Springer, ??? (2023) - Biswas and Islam [2022] Biswas, A., Islam, M.S.: Mri brain tumor classification technique using fuzzy c-means clustering and artificial neural network. In: International Conference on Artificial Intelligence for Smart Community: AISC 2020, 17–18 December, Universiti Teknologi Petronas, Malaysia, pp. 1005–1012 (2022). Springer - Zohuri and Moghaddam [2020] Zohuri, B., Moghaddam, M.: From business intelligence to artificial intelligence. Journal of Material Sciences & Manufacturing Research. SRC/JMSMR/102 Page 3 (2020) - Biswas and Islam [2023] Biswas, A., Islam, M.S.: A hybrid deep cnn-svm approach for brain tumor classification. Journal of Information Systems Engineering & Business Intelligence 9 (1) (2023) - Biswas and Islam [2021] Biswas, A., Islam, M.: Ann-based brain tumor classification: Performance analysis using k-means and fcm clustering with various training functions. In: Explainable Artificial Intelligence for Smart Cities, pp. 83–102. CRC Press, ??? (2021) - Biswas et al. [2023] Biswas, A., Md Abdullah Al, N., Imran, A., Sejuty, A.T., Fairooz, F., Puppala, S., Talukder, S.: Generative adversarial networks for data augmentation. In: Data Driven Approaches on Medical Imaging, pp. 159–177. Springer, ??? (2023) - Gong et al. [2023] Gong, T., Zhu, L., Yu, F.R., Tang, T.: Edge intelligence in intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems (2023) - Biswas and Islam [2021] Biswas, A., Islam, M.S.: An efficient cnn model for automated digital handwritten digit classification. Journal of Information Systems Engineering and Business Intelligence 7 (1), 42–55 (2021) - Malik [2019] Malik, A.: Explainable Intelligence Part 1 - XAI, the Third Wave Of AI. https://www.linkedin.com/pulse/explainable-intelligence-part-1-xai-third-wave-ai-ajay-malik/ - Schoenherr et al. [2023] Schoenherr, J.R., Abbas, R., Michael, K., Rivas, P., Anderson, T.D.: Designing ai using a human-centered approach: Explainability and accuracy toward trustworthiness. IEEE Transactions on Technology and Society 4 (1), 9–23 (2023) - Chamola et al. [2023] Chamola, V., Hassija, V., Sulthana, A.R., Ghosh, D., Dhingra, D., Sikdar, B.: A review of trustworthy and explainable artificial intelligence (xai). IEEE Access (2023) - Guleria and Sood [2023] Guleria, P., Sood, M.: Explainable ai and machine learning: performance evaluation and explainability of classifiers on educational data mining inspired career counseling. Education and Information Technologies 28 (1), 1081–1116 (2023) - Mirzaei et al. [2023] Mirzaei, S., Mao, H., Al-Nima, R.R.O., Woo, W.L.: Explainable ai evaluation: A top-down approach for selecting optimal explanations for black box models. Information 15 (1), 4 (2023) - Vyas [2023] Vyas, B.: Explainable ai: Assessing methods to make ai systems more transparent and interpretable. International Journal of New Media Studies: International Peer Reviewed Scholarly Indexed Journal 10 (1), 236–242 (2023) - Wang et al. [2024] Wang, A.Q., Karaman, B.K., Kim, H., Rosenthal, J., Saluja, R., Young, S.I., Sabuncu, M.R.: A framework for interpretability in machine learning for medical imaging. IEEE Access (2024) - Ghnemat et al. [2023] Ghnemat, R., Alodibat, S., Abu Al-Haija, Q.: Explainable artificial intelligence (xai) for deep learning based medical imaging classification. Journal of Imaging 9 (9), 177 (2023) - Gohel et al. [2021] Gohel, P., Singh, P., Mohanty, M.: Explainable ai: current status and future directions. arXiv preprint arXiv:2107.07045 (2021) - Wang and Ding [2024] Wang, P., Ding, H.: The rationality of explanation or human capacity? understanding the impact of explainable artificial intelligence on human-ai trust and decision performance. Information Processing & Management 61 (4), 103732 (2024) - Herm [2023] Herm, L.-V.: Algorithmic decision-making facilities: Perception and design of explainable ai-based decision support systems. PhD thesis, Universität Würzburg (2023) - Thalpage [2023] Thalpage, N.: Unlocking the black box: Explainable artificial intelligence (xai) for trust and transparency in ai systems. Journal of Digital Art & Humanities 4 (1), 31–36 (2023) - Balasubramaniam et al. [2023] Balasubramaniam, N., Kauppinen, M., Rannisto, A., Hiekkanen, K., Kujala, S.: Transparency and explainability of ai systems: From ethical guidelines to requirements. Information and Software Technology 159, 107197 (2023) - Arrieta and et al. [2020] Arrieta, A.B., al.: Explainable artificial intelligence (XAI): Concepts taxonomies opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020) - McLarney and et al. [2021] McLarney, E., al.: NASA framework for the ethical use of artificial intelligence (AI) (2021) - Kumar et al. [2020] Kumar, A., Braud, T., Tarkoma, S., Hui, P.: Trustworthy ai in the age of pervasive computing and big data. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–6 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156127 - Guembe et al. [2022] Guembe, B., Azeta, A., Osamor, V., Ekpo, R.: Explainable artificial intelligence, the fourth pillar of zero trust security. Available at SSRN 4331547 (2022) - Kim et al. [2023] Kim, M., Sohn, H., Choi, S., Kim, S.: Requirements for trustworthy artificial intelligence and its application in healthcare. Healthcare Informatics Research 29 (4), 315 (2023) - Charmet et al. [2022] Charmet, F., Tanuwidjaja, H.C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., Zhang, Z.: Explainable artificial intelligence for cybersecurity: a literature survey. Annals of Telecommunications 77 (11), 789–812 (2022) - Jiang et al. [2017] Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., Wang, Y.: Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology 2 (4) (2017) - Davenport and Kalakota [2019] Davenport, T., Kalakota, R.: The potential for artificial intelligence in healthcare. Future healthcare journal 6 (2), 94 (2019) - Jaspers et al. [2011] Jaspers, M.W., Smeulers, M., Vermeulen, H., Peute, L.W.: Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. Journal of the American Medical Informatics Association 18 (3), 327–334 (2011) - Metta et al. [2023] Metta, C., Beretta, A., Guidotti, R., Yin, Y., Gallinari, P., Rinzivillo, S., Giannotti, F.: Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning. International Journal of Data Science and Analytics, 1–13 (2023) - Akpan et al. [2022] Akpan, A.G., Nkubli, F.B., Ezeano, V.N., Okwor, A.C., Ugwuja, M.C., Offiong, U.: Xai for medical image segmentation in medical decision support systems. Explainable Artificial Intelligence in Medical Decision Support Systems 50, 137 (2022) - Tosun et al. [2020] Tosun, A.B., Pullara, F., Becich, M.J., Taylor, D.L., Fine, J.L., Chennubhotla, S.C.: Explainable ai (xai) for anatomic pathology. Advances in Anatomic Pathology 27 (4), 241–250 (2020) - Agrawal et al. [2024] Agrawal, N., Pendharkar, I., Shroff, J., Raghuvanshi, J., Neogi, A., Patil, S., Walambe, R., Kotecha, K.: A-xai: adversarial machine learning for trustable explainability. AI and Ethics, 1–32 (2024) - Petch et al. [2022] Petch, J., Di, S., Nelson, W.: Opening the black box: the promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology 38 (2), 204–213 (2022) - Rajpurkar et al. [2022] Rajpurkar, P., Chen, E., Banerjee, O., Topol, E.J.: Ai in health and medicine. Nature medicine 28 (1), 31–38 (2022) - Atakishiyev et al. [2021] Atakishiyev, S., Salameh, M., Yao, H., Goebel, R.: Explainable artificial intelligence for autonomous driving: A comprehensive overview and field guide for future research directions. arXiv preprint arXiv:2112.11561 (2021) - Alexandrov [2017] Alexandrov, N.: Explainable ai decisions for human-autonomy interactions. In: 17th AIAA Aviation Technology, Integration, and Operations Conference, p. 3991 (2017) - Xu et al. [2019] Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., Zhu, J.: Explainable ai: A brief survey on history, research areas, approaches and challenges. In: Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part II 8, pp. 563–574 (2019). Springer - Yazdanpanah et al. [2021] Yazdanpanah, V., Gerding, E., Stein, S., Dastani, M., Jonker, C.M., Norman, T.: Responsibility research for trustworthy autonomous systems (2021) - Dong et al. [2023] Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., Labi, S.: Why did the ai make that decision? towards an explainable artificial intelligence (xai) for autonomous driving systems. Transportation research part C: emerging technologies 156, 104358 (2023) - Madhav and Tyagi [2022] Madhav, A.S., Tyagi, A.K.: Explainable artificial intelligence (xai): connecting artificial decision-making and human trust in autonomous vehicles. In: Proceedings of Third International Conference on Computing, Communications, and Cyber-Security: IC4S 2021, pp. 123–136 (2022). Springer - Hoffmann et al. [2021] Hoffmann, M.W., Drath, R., Ganz, C.: Proposal for requirements on industrial ai solutions. In: Machine Learning for Cyber Physical Systems: Selected Papers from the International Conference ML4CPS 2020, pp. 63–72 (2021). Springer Berlin Heidelberg - Kotriwala et al. [2021] Kotriwala, A., Klöpper, B., Dix, M., Gopalakrishnan, G., Ziobro, D., Potschka, A.: Xai for operations in the process industry-applications, theses, and research directions. In: AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering, pp. 1–12 (2021) - Mamandipoor et al. [2020] Mamandipoor, B., Majd, M., Sheikhalishahi, S., Modena, C., Osmani, V.: Monitoring and detecting faults in wastewater treatment plants using deep learning. Environmental Monitoring and Assessment 192 (3), 148 (2020) - Cecílio et al. [2014] Cecílio, I., Ottewill, J., Pretlove, J., Thornhill, N.: Nearest neighbors method for detecting transient disturbances in process and electromechanical systems. Journal of Process Control 24, 1382–1393 (2014) - Banjanovic-Mehmedovic et al. [2017] Banjanovic-Mehmedovic, L., Hajdarevic, A., Kantardzic, M., Mehmedovic, F., Dzananovic, I.: Neural network-based data-driven modelling of anomaly detection in thermal power plant. Automatika: časopis za automatiku, mjerenje, elektroniku, računarstvo i komunikacije 58, 69–79 (2017) - Ruiz et al. [2001] Ruiz, D., Canton, J., Nougués, J., Espuna, A., Puigjaner, L.: On-line fault diagnosis system support for reactive scheduling in multipurpose batch chemical plants. Computers & Chemical Engineering 25, 829–837 (2001) - Yélamos et al. [2007] Yélamos, I., Graells, M., Puigjaner, L., Escudero, G.: Simultaneous fault diagnosis in chemical plants using a multilabel approach. AIChE Journal 53, 2871–2884 (2007) - Lucke et al. [2020] Lucke, M., Stief, A., Chioua, M., Ottewill, J., Thornhill, N.: Fault detection and identification combining process measurements and statistical alarms. Control Engineering Practice 94, 104195 (2020) - Dorgo et al. [2018] Dorgo, G., Pigler, P., Haragovics, M., Abonyi, J.: Learning operation strategies from alarm management systems by temporal pattern mining and deep learning. Computer Aided Chemical Engineering 43, 1003–1008 (2018) - Giuliani et al. [2019] Giuliani, M., Camarda, G., Montini, M., Cadei, L., Bianco, A., Shokry, A., Baraldi, P., Zio, E., et al.: Flaring events prediction and prevention through advanced big data analytics and machine learning algorithms. In: Offshore Mediterranean Conference and Exhibition (2019). Offshore Mediterranean Conference - Carter and Briens [2018] Carter, A., Briens, L.: An application of deep learning to detect process upset during pharmaceutical manufacturing using passive acoustic emissions. International journal of pharmaceutics 552, 235–240 (2018) - Desai et al. [2006] Desai, K., Badhe, Y., Tambe, S., Kulkarni, B.: Soft-sensor development for fed-batch bioreactors using support vector regression. Biochemical Engineering Journal 27, 225–239 (2006) - Shang et al. [2014] Shang, C., Yang, F., Huang, D., Lyu, W.: Data-driven soft sensor development based on deep learning technique. Journal of Process Control 24, 223–233 (2014) - Napier and Aldrich [2017] Napier, L., Aldrich, C.: An isamill™ soft sensor based on random forests and principal component analysis. IFAC-PapersOnLine 50, 1175–1180 (2017) - Amihai et al. [2018a] Amihai, I., Gitzel, R., Kotriwala, A., Pareschi, D., Subbiah, S., Sosale, G.: An industrial case study using vibration data and machine learning to predict asset health. In: 2018 IEEE 20th Conference on Business Informatics (CBI), vol. 1, pp. 178–185 (2018). IEEE - Amihai et al. [2018b] Amihai, I., Chioua, M., Gitzel, R., Kotriwala, A., Pareschi, D., Sosale, G., Subbiah, S.: Modeling machine health using gated recurrent units with entity embeddings and k-means clustering. In: 2018 IEEE 16th International Conference on Industrial Informatics (INDIN), pp. 212–217 (2018). IEEE - Kolokas et al. [2020] Kolokas, N., Vafeiadis, T., Ioannidis, D., Tzovaras, D.: Fault prognostics in industrial domains using unsupervised machine learning classifiers. Simulation Modelling Practice and Theory, 102109 (2020) - Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.-Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, pp. 1–18 (2018). Association for Computing Machinery - Adadi and Berrada [2018] Adadi, A., Berrada, M.: Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access 6, 52138–52160 (2018) https://doi.org/10.1109/ACCESS.2018.2870052 - Saeed and Omlin [2023] Saeed, W., Omlin, C.: Explainable ai (xai): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263, 110273 (2023)