2410.17139v2

# Trustworthy XAI and Its Applications **Authors**: \fnmA.S.M Anas\surFerdous, \fnmAbdur\surRashid, \fnmFatema Tuj Johura\surSoshi, \fnmParag\surBiswas, \fnmAngona\surBiswas, \fnmKishor\surDatta Gupta > nasim.abdullah@ieee.org > anasferdous001@gmail.com > rabdurrashid091@gmail.com > fatemasoshi@gmail.com > text2parag@gmail.com > angonabiswas28@gmail.com > kgupta@cau.edu[[[[[ [1] \fnm MD Abdullah Al \sur Nasim 1,6] \orgdiv Research and Development Department, \orgname Pioneer Alpha, \orgaddress \city Dhaka, \country Bangladesh 2] \orgdiv Department of Biomedical Engineering, \orgname Bangladesh University of Engineering and Technology, \orgaddress \city Dhaka, \country Bangladesh 4] \orgdiv Msc in Data Science and Analytics, \orgname University of Hertfordshire, \orgaddress \city Hatfield, \country UK 3, 5] \orgdiv MSEM Department, \orgname Westcliff university, \orgaddress \city California, \country United States 7] \orgdiv Department of Computer and Information Science, \orgname Clark Atlanta University, \city Georgia, \country USA Abstract Artificial Intelligence (AI) is an important part of our everyday lives. We use it in self-driving cars and smartphone assistants. People often call it a ”black box” because its complex systems, especially deep neural networks, are hard to understand. This complexity raises concerns about accountability, bias, and fairness, even though AI can be quite accurate. Explainable Artificial Intelligence (XAI) is important for building trust. It helps ensure that AI systems work reliably and ethically. This article looks at XAI and its three main parts: transparency, explainability, and trustworthiness. We will discuss why these components matter in real-life situations. We will also review recent studies that show how XAI is used in different fields. Ultimately, gaining trust in AI systems is crucial for their successful use in society. keywords: Artificial Intelligence(AI), XAI, Explainable Artificial Intelligence (XAI), Healthcare, Autonomous Vehicles 1 Introduction The foundations of modern artificial intelligence were laid by philosophers who attempted to define human thought as the mechanical manipulation of symbols, which led to the development of the programmable digital computer [1] in the 1940s. Alan Turing may have written the first article on the topic of AI in 1941, though it is now lost, suggesting that he was at least considering the idea at that time.In his groundbreaking essay ”Computing Machinery and Intelligence” from 1950, Turing first presented the idea of the Turing test to the general public [2]. Turing questioned the feasibility of creating thinking robots in it. John McCarthy first used the term artificial intelligence (AI) in 1956 at the Dartmouth Conference [3], but the first models’ numerous flaws have prevented AI from being widely adopted and used in healthcare. Many of these limitations were removed with the advent of deep learning in the early 2000s, and we are now entering a new era of technology where AI can be used in clinical practice through risk assessment models that increase diagnostic accuracy and workflow efficiency. Performance of AI systems has improved significantly in recent years, and these new models expand on their capabilities to include text-image synthesis based on almost any prompt, whereas previous systems primarily focused on generating facial images. <details> <summary>extracted/6367585/image/a1.png Details</summary>  ### Visual Description \n ## Circular Diagram: Application of AI ### Overview The image is a circular diagram illustrating the application of Artificial Intelligence (AI) across various industries. The diagram is segmented into eight major industry sectors, each color-coded and containing a list of specific AI applications within that sector. The center of the circle is labeled "Application of AI". ### Components/Axes The diagram consists of eight sectors, arranged clockwise: 1. **Logistics Industry** (Blue) 2. **Financial Industry** (Purple) 3. **Manufacturing Industry** (Orange) 4. **Other Industries** (Green) - includes sub-points related to education, agriculture, environmental protection, and urban management. 5. **Electronic Commerce** (Yellow) 6. **Health Care** (Red) 7. **Security Industry** (Pink) 8. **Retail Industry** (Light Green) The center of the diagram contains the text "Application of AI". Each sector has a list of bullet points detailing specific AI applications. ### Detailed Analysis or Content Details **1. Logistics Industry (Blue):** * UAV delivery * Automatic pilot * Intelligent sorting * Intelligent logistics planning * Intelligent scheduling algorithm **2. Financial Industry (Purple):** * Analysis industry trend forecast * Investment risk analysis * Big data analysis of stock securities **3. Manufacturing Industry (Orange):** * Equipment fault prediction * Product defect detection * Machine vision positioning * Man-machine cooperation **4. Other Industries (Green):** * Education: Unmanned examination and marking * Agricultural: Real-time monitoring of crop status * Environmental protection: Intelligent energy consumption monitoring and analysis * Urban management: Traffic route optimization **5. Electronic Commerce (Yellow):** * Intelligent customer service robot * Recommendation engine * Image search * Sales and inventory forecasts * Commodity pricing **6. Health Care (Red):** * Medical robot * Intelligent drug research and development * Intelligent treatment * Intelligent image recognition * Intelligent health management **7. Security Industry (Pink):** * Human body analysis * Vehicle analysis * Behavior analysis * Image analysis **8. Retail Industry (Light Green):** * Unmanned convenience store * Unmanned warehouse * Intelligent supply chain * Intelligent customer flow statistics and analysis * Anti-theft alarm * Programmable control * Intelligent speakers * Remote control * Environmental monitoring ### Key Observations The diagram demonstrates a broad range of AI applications across diverse industries. The "Other Industries" sector is notably different, listing broader application areas (education, agriculture, etc.) rather than specific AI techniques. The Retail Industry has the most listed applications (8), while the Logistics and Health Care industries have 5 each. The diagram suggests that AI is being integrated into nearly all aspects of modern industry. ### Interpretation The diagram illustrates the pervasive nature of AI and its potential to transform various sectors. The clustering of applications within each industry suggests that AI solutions are often tailored to the specific needs and challenges of that sector. The diagram serves as a high-level overview, highlighting the breadth of AI's impact rather than delving into the technical details of each application. The emphasis on automation (UAV delivery, automatic pilot, intelligent sorting), data analysis (stock securities, customer flow statistics), and intelligent systems (robots, recommendation engines) indicates key areas of AI focus. The inclusion of "Other Industries" suggests that AI's reach extends beyond the traditionally recognized sectors. The diagram is a conceptual representation and does not provide quantitative data on the adoption rate or impact of AI in each industry. </details> Figure 1: Applications of AI across various domains [4] The diverse range of fields in which AI is being used already demonstrates its applicability and promise to revolutionize business: AI facilitates activities like sentiment analysis, machine translation, and spam filtering by making it easier for computers to comprehend and produce human language in the discipline of natural language processing (NLP) [5]. Additionally, computer vision [6] makes it possible for computers to understand visual data, which advances areas like facial recognition, object identification, and self-driving cars. Machine learning (ML), which has uses in fraud detection, recommendation systems, predictive analytics, and other domains, has made it possible for computers to learn from data. The design, development, and application of machines are the focus of the AI field of robotics [7]. Many industries, including manufacturing, healthcare, and space exploration, use various machines [8] [9]. Combining artificial intelligence with business intelligence (BI) [10] improves how businesses collect, process, and visualize data. This leads to better decision-making and increased productivity. In healthcare, AI helps diagnose diseases, develop treatments, and provide personalized care, which improves patient outcomes [11], [12], [13]. AI also plays a significant role in education by engaging students, customizing lessons, and automating administrative tasks, resulting in more personalized learning experiences. AI in agriculture increases agricultural output, reduces costs, and ensures environmental sustainability through data-driven strategies. In a similar vein, AI in manufacturing boosts output, efficiency, and quality through work automation and process optimization. AI is changing operations, enhancing services, and changing global industry landscapes in a number of sectors, including banking, retail, energy, transportation [14], handwriting detection [15], and government. AI is widely used in a wide number of industries, as seen in Figure 1. Retail, security, healthcare, e-commerce, manufacturing, banking, logistics and transportation, and home furnishings are some of these sectors. These applications rely on moderately advanced AI technology, such as computer vision, natural language processing, and machine learning. The contributions of this research can be stated below: 1. Providing an overview of XAI that understands the significance of black box models. For fair and ethical purposes, XAI fosters trust among humans and AI. 1. Discussing Deep Learning based systems that will be consistent and trustworthy. Moreover, recent studies from the literature have been reviewed properly. 1. Providing guidelines regarding XAI that will be helpful for detecting problems from numerous domains. 1. The paper identifies and analyzes three key components of XAI: transparency, explainability, and trustworthiness. It details how these elements are essential for understanding and improving AI systems. 1.1 Third Wave of Artificial Intelligence (3AI) Most current commercial AI technology is called ”narrow AI.” This means these systems are highly specialized and can only perform a few specific tasks. For example, even the best self-driving cars rely on limited AI systems. Another drawback of today’s AI is its reliance on large training data sets. A typical machine learning program needs tens of thousands of cat photos to recognize cats accurately, while a three-year-old child can do this with just a few examples. The idea of ”Third Wave AI” comes from the need for AI to become more humanlike in various ways to overcome these limitations and achieve its full potential. <details> <summary>x1.jpg Details</summary>  ### Visual Description \n ## Diagram: AI/ML Wave Progression ### Overview The image is a cyclical diagram illustrating the progression of Artificial Intelligence and Machine Learning through four waves, spanning from before 2010 to 2030 and beyond. Each wave is represented by a colored box with associated characteristics listed as bullet points. Arrows indicate the flow from one wave to the next, suggesting a continuous evolution. ### Components/Axes The diagram consists of four main components, each representing a "Wave": * **First Wave (Gray):** Pre 2010 * **Second Wave (Blue):** 2010-2020 * **Third Wave (Orange):** 2020-2030 * **Fourth Wave (Green):** 2030- The diagram also includes directional arrows indicating the progression between waves. Time periods are indicated alongside the arrows. ### Detailed Analysis or Content Details **First Wave (Gray): Pre 2010** * Handcrafted/Human programmed * Traditional Programming * No learning capability * Poor handling of uncertainty **Second Wave (Blue): 2010-2020** * Statistical Models trained on BIG Data * Neural Networks -Deep Learning * Individually unreliable **Third Wave (Orange): 2020-2030** * Models to drive decisions * Models to explain decisions **Fourth Wave (Green): 2030-** * More human like learning * Learn from descriptive, contextual models instead of enormous sets of labeled training data * Learn interactively ### Key Observations The diagram presents a clear progression of AI/ML techniques. The waves build upon each other, with each subsequent wave addressing the limitations of the previous one. The shift from handcrafted programming to data-driven models, and then to more explainable and human-like learning, is a central theme. The diagram highlights the increasing sophistication and capability of AI/ML over time. ### Interpretation The diagram illustrates a narrative of AI/ML development. The "waves" represent distinct eras characterized by dominant approaches and capabilities. The first wave represents the foundational era of rule-based systems. The second wave marks the rise of data-driven methods, particularly deep learning. The third wave focuses on applying these models to real-world decision-making and understanding their reasoning. The fourth wave envisions a future where AI/ML systems learn more like humans, leveraging contextual understanding and interactive learning. The cyclical nature of the diagram suggests that this progression is not necessarily linear, and that future developments may revisit or combine elements from earlier waves. The emphasis on "explainability" and "human-like learning" in the later waves reflects a growing concern about the transparency and trustworthiness of AI/ML systems. The diagram implies that the field is moving towards AI that is not only powerful but also understandable and aligned with human values. </details> Figure 2: Past, Present, and Future of AI waves. [16] According to the Defense Advanced Research Projects Agency (DARPA) [17], third-wave AI systems will understand the context of situations, use common sense, and adapt to changes. This will create more natural and intuitive connections between AI systems and people [17]. One of DARPA’s active projects, called XAI, aims to develop these third-wave AI systems. These computers will learn about their environments and the contexts in which they work. They will also build explanations needed to clarify real-world events. - First Wave AI focused on rules, logic, and built knowledge. - Second Wave AI introduced big data, statistical learning, and probabilistic techniques. - The goal of third-wave AI is to develop common sense and the ability to adapt to different contexts. Tractica [16] predicts that the global market for AI software will grow from about 9.5 billion US dollars in 2018 to 118.6 billion by 2025. This data aims to develop AI systems that can perform tasks accurately while providing explanations that people can understand. The term ”third wave” refers to the advancement of AI technologies beyond traditional machine learning. This new phase focuses on creating more advanced systems that can understand context, reason, and think similarly to humans. It draws inspiration from cognitive science and neuroscience, aiming to build AI that can engage with the world in more complex and detailed ways. XAI, or Explainable Artificial Intelligence, focuses on making AI systems, especially machine learning models, easier to understand. The goal is to help people trust the decisions these systems make. XAI techniques work to explain how AI makes predictions and why it behaves the way it does. This helps users, such as developers, regulators, and everyday users, grasp the key factors that affect AI results. Developing AI systems that can make accurate predictions or decisions and explain why they did so is a key goal of both the third wave of AI and explainable AI. By using XAI methods in their design, developers can ensure that these advanced AI systems are not only effective but also easy to understand. This will help build user trust and acceptance. 1.2 Concept of Explainable AI AI often struggles with what is called the ”black box” problem because users do not understand how it works. This can lead to issues like lack of trust, confusion, unfair treatment, and violations of privacy. AI systems can also have hidden biases. Explainable AI (XAI) aims to make AI systems easier to understand, helping users know how they make decisions. The goal of XAI is to make AI safer and more user-friendly. Therefore, we need to look at each part of AI individually and discuss its different aspects [18]. There are two main types of machine learning (ML) techniques: white box and black box models [19]. Experts can easily understand the results of a white box model. However, even specialists may find it hard to grasp the results of a black box model [20]. The XAI algorithm [21] follows three key principles: interpretability, explainability, and transparency. A model is transparent when it clearly explains how it gets its results from training data and how it generates labels from test data. Interpretability means being able to explain findings in a way that others understand [22]. While there’s no single accepted definition of explainability, its value is recognized. One definition describes it as a set of clear features that help make decisions, like classifying or predicting outcomes for specific cases. An algorithm that meets these standards helps document and verify decisions and improves itself based on new data. <details> <summary>x2.jpg Details</summary>  ### Visual Description \n ## Diagram: Generic vs. Explainable AI Approach ### Overview The image presents a comparative diagram illustrating the difference between a "Generic Approach" to Machine Learning and an "eXplainable AI Approach". Both approaches share initial stages of data input and processing, but diverge in their output and interaction with users. The diagram uses visual metaphors (cylinders, gears, lightbulbs, people) to represent components and arrows to indicate flow. ### Components/Axes The diagram is divided into two main sections, labeled "Generic Approach" (left) and "eXplainable AI Approach" (right). Each section contains the following components: * **Training Dataset:** Represented by a purple cylinder with a blue top, labeled "DB". * **Machine Learning Processes:** Represented by a gear icon. * **Learned Function/Explainable Model:** Represented by a lightbulb (Generic) or a cluster of spheres (Explainable). * **Outcome/Decision/Explainable Interface:** Represented by an arrow pointing towards a person icon, labeled "Users". * **Question Bubbles:** Each section has a speech bubble containing questions or statements. ### Detailed Analysis or Content Details **Generic Approach (Left Side):** 1. **Training Dataset (DB):** Input to the system. 2. **Machine Learning Processes:** Processes the training data. 3. **Learned Function:** The output of the machine learning processes, represented as a lightbulb. 4. **Outcome/Decision:** The lightbulb's output is directed towards "Users". 5. **Question Bubble:** Contains the following questions: * "Why did you do that?" * "Why not something else?" * "How/When can I trust you?" * "How do I correct error?" **eXplainable AI Approach (Right Side):** 1. **Training Dataset (DB):** Input to the system, identical to the Generic Approach. 2. **Machine Learning Processes:** Processes the training data, identical to the Generic Approach. 3. **Explainable Model:** The output of the machine learning processes, represented as a cluster of spheres. 4. **Explainable Interface:** Connects the Explainable Model to the "Users". 5. **Question Bubble:** Contains the following statements: * "I understand Why." * "I understand Why Not" * "I know when you succeed/fail." * "I know when/how to trust you." ### Key Observations * Both approaches start with the same input (Training Dataset) and processing (Machine Learning Processes). * The key difference lies in the output and the interaction with users. The Generic Approach provides a "black box" output (Learned Function) without explanation, while the eXplainable AI Approach provides an "Explainable Model" and an "Explainable Interface" to facilitate understanding. * The questions posed in the Generic Approach highlight the lack of transparency and trust in traditional machine learning models. * The statements in the eXplainable AI Approach demonstrate the benefits of transparency and understanding. ### Interpretation The diagram illustrates the growing need for explainability in AI systems. Traditional machine learning models, while often accurate, can be opaque and difficult to understand. This lack of transparency can hinder trust and adoption, particularly in critical applications. The eXplainable AI approach aims to address this issue by providing insights into the model's decision-making process, allowing users to understand *why* a particular outcome was reached. This fosters trust, enables debugging, and facilitates responsible AI development. The diagram effectively conveys the shift from a "black box" approach to a more transparent and user-friendly AI paradigm. The questions and statements within the bubbles are particularly effective in highlighting the core benefits of explainable AI. </details> Figure 3: Explainable Artificial Intelligence (XAI): A look at AI now and tomorrow. [18] Researchers are studying intelligent systems to understand them better. This is an important topic. Sometimes, a system needs to understand its own workings to comply with rules. Many complex algorithms, shown in Figure 4, balance achieving high accuracy with being explainable. <details> <summary>x3.jpg Details</summary>  ### Visual Description \n ## Diagram: Machine Learning Model Landscape ### Overview The image is a diagram illustrating a landscape of machine learning models, positioned based on their relative "Learning Performance" and "Explainability". The diagram uses overlapping circles and rectangles to represent different model categories, with arrows indicating relationships and flow between them. There are also several data points scattered around the diagram, presumably representing specific models or instances. ### Components/Axes * **Axes:** * Y-axis: "Learning Performance" (vertical) * X-axis: "Explainability" (horizontal) * **Model Categories (represented by shapes):** * Neural Nets (Green Circle) * Deep Learning (Text within circle) * Graphical Models (Light Blue Circle) * Bayesian Belief Nets (Text within circle) * Statistical Models (Grey Circle) * SVMs (Text within circle) * AOGs (Text within circle) * Ensemble Methods (Dark Blue Rectangle) * Random Forest (Text within rectangle) * Decision Tree (Text within rectangle) * **Connecting Lines:** Dashed lines connect Statistical Models to Graphical Models and Ensemble Methods. * **Data Points:** Five circular data points of varying sizes and shades of grey are scattered around the diagram. ### Detailed Analysis The diagram positions models along a spectrum of Learning Performance and Explainability. * **Neural Nets:** Located towards the upper-left, suggesting high learning performance but lower explainability. * **Graphical Models:** Positioned slightly to the right of Neural Nets, indicating a balance between learning performance and explainability. * **Statistical Models:** Located towards the lower-left, suggesting lower learning performance but potentially higher explainability. * **Ensemble Methods:** Positioned towards the upper-right, suggesting high learning performance and relatively high explainability. * **Data Points:** * Top-left: Small, dark grey circle. (Approx. Learning Performance: 8, Explainability: 2) * Top-center: Medium, dark grey circle. (Approx. Learning Performance: 9, Explainability: 4) * Center-left: Small, light grey circle. (Approx. Learning Performance: 5, Explainability: 3) * Bottom-center: Medium, light grey circle. (Approx. Learning Performance: 3, Explainability: 6) * Bottom-right: Small, dark grey circle. (Approx. Learning Performance: 2, Explainability: 8) The dashed lines indicate relationships between model types: * Statistical Models connect to Graphical Models. * Statistical Models connect to Ensemble Methods. * Graphical Models connect to Ensemble Methods. ### Key Observations * There's a general trend of increasing learning performance as explainability decreases, and vice-versa. * Ensemble Methods appear to offer a good balance between the two. * The data points are scattered, suggesting that within each category, there's variation in performance and explainability. * The diagram doesn't provide specific numerical values for performance or explainability, only relative positioning. ### Interpretation This diagram presents a conceptual map of machine learning models, highlighting the trade-off between learning performance and explainability. It suggests that choosing a model involves balancing these two factors based on the specific application. The positioning of the models implies a hierarchy or relationship. For example, the connections between Statistical Models and other categories suggest that Statistical Models can serve as a foundation or component for more complex models. The data points likely represent individual models or datasets, demonstrating the variability within each category. The diagram is a simplification of a complex landscape. It doesn't account for other important factors like computational cost, data requirements, or model complexity. However, it provides a useful framework for understanding the relative strengths and weaknesses of different machine learning approaches. The diagram is a qualitative representation, and the exact positioning of the models is subjective and depends on the specific context. </details> Figure 4: Trade-off between AI model accuracy and explainability, highlighting the challenge of balancing performance with interpretability. [23] 1.3 Classification Tree of XAI XAI techniques are divided into two categories: transparent and post-hoc methods. A transparent approach is one that represents the model’s capabilities and decision-making process in an easy-to-understand way [24]. Transparent models include Bayesian approaches, decision trees, linear regression, and fuzzy inference systems. Transparent approaches can be useful when the internal feature correlations are highly complex or linear. A comprehensive classification of different XAI methods and approaches related to different types of data is shown in Figure 5. <details> <summary>extracted/6367585/image/classification.jpg Details</summary>  ### Visual Description ## Diagram: XAI Techniques ### Overview This diagram illustrates a hierarchical classification of Explainable Artificial Intelligence (XAI) techniques. The techniques are categorized based on whether they are "Post hoc methods" or "Transparent methods," and further subdivided by their applicability (model agnostic vs. model specific) and data type (Text, Image, Audio, Video). ### Components/Axes The diagram is structured as a tree diagram, branching from a central title "XAI Techniques". The main branches are "Post hoc methods" and "Transparent methods". "Post hoc methods" further branches into "Model agnostic" and "Model specific". Each of these branches then subdivides based on data type: Text, Image, Audio, and Video. ### Detailed Analysis or Content Details **1. XAI Techniques (Top Level)** - Title: "XAI Techniques" - positioned at the very top center. **2. Post hoc methods** - Branching from the top center. - Sub-branches: "Model agnostic" and "Model specific". **3. Transparent methods** - Branching from the top right. - Sub-branch: "All type of Data" **4. Model agnostic** - Positioned to the left of "Model specific". - Data type sub-branches: - **Text:** - Feature Relevance - Condition based explanation - Local explanation - **Image:** - Rule based learning - Feature based saliency map - **Audio:** - Feature Relevance - Local explanation - **Video:** - Saliency map **5. Model specific** - Positioned to the right of "Model agnostic". - Data type sub-branches: - **Text:** - LIME - Perturbation - LRP - Provenance - Taxonomy induc. - **Image:** - SHAP Values - HEAT map - LIME - Counterfactual explanation - Perturbation - **Audio:** - LIME - Perturbation - LRP - **Video:** - SHAP Values - Counterfactual explanation - Perturbation **6. Transparent methods** - Positioned on the right side of the diagram. - Data type sub-branch: "All type of Data" - Logistic regression - Decision Trees - K-Nearest neighbors - Rule based classifier - Bayesian Model ### Key Observations - The diagram clearly distinguishes between methods applied *after* model training ("Post hoc") and methods that are inherently interpretable due to their structure ("Transparent"). - "Model agnostic" methods can be applied to any model, while "Model specific" methods are tailored to particular model architectures. - The variety of techniques available differs across data types, with Text and Image having the most extensive lists. - LIME, Perturbation, and LRP appear in multiple branches, suggesting their versatility across different data types. ### Interpretation The diagram provides a useful overview of the landscape of XAI techniques. It highlights the trade-offs between model flexibility (agnostic vs. specific) and interpretability (post hoc vs. transparent). The categorization by data type is practical, as the best XAI approach often depends on the nature of the data being analyzed. The repetition of certain techniques (LIME, Perturbation, LRP) suggests they are foundational methods with broad applicability. The diagram suggests that while transparent methods are available, they are limited to certain model types, making post-hoc methods crucial for understanding complex, black-box models. The diagram does not provide any quantitative data or performance metrics; it is purely a structural classification. </details> Figure 5: Categorization of Explainable AI (XAI) techniques based on data type, illustrating differences between transparent and posthoc approaches [24] Posterior approaches are useful for interpreting the complexity of a model, especially when there are nonlinear relationships or high data complexity. When a model does not follow a direct relationship between data and features, posterior techniques can be an effective tool to explain what the model has learned [24]. Inference using local feature weights is provided by transparent methods such as Bayesian classifiers, support vector machines, logistic regression, and K-nearest neighbors. This model category meets three properties: simulability, decomposability, and algorithmic transparency [24]. 1.4 Definition of Transparency in Artificial Intelligence Transparency in XAI is the capacity of an AI system to make its decisions and actions understandable through explanations [25]. Transparency is one of the most important aspects of XAI, rendering AI decisions interpretable and justified. Transparency allows decision-making processes to be closely examined by stakeholders, mitigating risks in applications with societal impact, such as healthcare and finance [26]. The explainability provided by XAI techniques also increases the overall transparency of AI systems. The users are able to examine the decision-making procedure, identify any bias, and analyze the reliability and fairness of the output of the model [27]. Transparent solutions are necessary in areas like the medical field, banking, and autonomous vehicles, where AI-decisions can have significant effects. By providing meaningful information about the internal processes of AI models, XAI methods assist users in recognizing patterns, comprehending relationships, and revealing biases or errors [27]. Due to the heightened transparency, stakeholders can more effectively develop opinions, ensure the predictions made by the model are accurate, and take the necessary action. The examination of ethical criteria showed a link between explainability, transparency, and other quality needs. Figure 6 displays nine quality standards related to explainability and openness. The key standards for AI transparency are marked with “O.” For example, O2 focuses on how to interpret models, O15 highlights the importance of traceability, and O5 and O12 ensure that users understand the information. By following these standards, AI models can provide clear and convincing explanations for their outcomes. Keeping these standards improves accountability and helps reduce bias in AI systems. <details> <summary>extracted/6367585/image/eai3.jpg Details</summary>  ### Visual Description \n ## Diagram: Relationships of AI Ethical Principles ### Overview The image is a diagram illustrating the relationships between different ethical principles in the context of Artificial Intelligence (AI). The central concept is "Transparency and Explainability," which is shown to have positive influences on several other principles. The diagram uses rounded rectangles to represent the principles and arrows to indicate the direction of influence. ### Components/Axes The diagram consists of the following components: * **Central Node:** "Transparency and Explainability" – positioned in the center of the diagram. * **Connected Nodes:** * "Understandability" (O2, O5, O12, O15) – positioned to the left of the central node. * "Traceability" (O2, O9, O12) – positioned below and to the left of the central node. * "Trustworthiness" (O1, O4, O5, O11) – positioned above the central node. * "Privacy" (O7, O13) – positioned to the right and above the central node. * "Auditability" (O4) – positioned to the right of the central node. * "Fairness" (O5) – positioned below the central node. * **Arrows:** Arrows with "+" signs indicate a positive influence. An arrow between "Privacy" and the central node has both "+" and "-" signs, indicating a potentially complex or bidirectional relationship. * **Identifiers:** Each principle is associated with a set of identifiers enclosed in parentheses (e.g., O1, O2, O5). ### Detailed Analysis or Content Details The diagram shows the following relationships: * **Transparency and Explainability → Understandability:** Positive influence. * **Transparency and Explainability → Traceability:** Positive influence. * **Transparency and Explainability → Trustworthiness:** Positive influence. * **Transparency and Explainability → Privacy:** Positive and negative influence. * **Transparency and Explainability → Auditability:** Positive influence. * **Transparency and Explainability → Fairness:** Positive influence. The identifiers associated with each principle are: * Understandability: O2, O5, O12, O15 * Traceability: O2, O9, O12 * Trustworthiness: O1, O4, O5, O11 * Privacy: O7, O13 * Auditability: O4 * Fairness: O5 ### Key Observations The diagram emphasizes the central role of "Transparency and Explainability" in fostering other ethical AI principles. The bidirectional influence on "Privacy" suggests that while transparency can enhance privacy, it can also potentially compromise it depending on the context. The identifiers (O1, O2, etc.) likely refer to specific objectives, requirements, or guidelines within a larger framework. ### Interpretation This diagram represents a causal model of how ethical principles in AI relate to each other. It suggests that investing in transparency and explainability is crucial for building AI systems that are understandable, traceable, trustworthy, private, auditable, and fair. The "+" signs on the arrows indicate that increasing transparency and explainability will likely lead to improvements in these other areas. The "+/-" sign on the arrow to "Privacy" indicates a more nuanced relationship, where transparency could either enhance or detract from privacy depending on how it is implemented. The identifiers (O1-O15) suggest that these principles are derived from a more detailed set of requirements or objectives. This diagram is a high-level conceptual representation and doesn't provide quantitative data or specific implementation details. It serves as a visual aid for understanding the interconnectedness of ethical considerations in AI development. </details> Figure 6: Key qualitative standards (O1-O15) related to explainability and transparency in AI systems, addressing user comprehension, interpretability, and traceability. [28] The growth of AI systems’ explainability and transparency is facilitated by their understandability. When discussing the significance of understandability, the transparency guidelines addressed three points: 1) ensuring that people comprehend the AI system’s behavior and the methods for using it (O5, O12); 2) communicating in an intelligible manner the locations, purposes, and methods of AI use (O15); and 3) making sure people comprehend the distinction between real AI decisions and those that AI merely assists in making (O2) [28]. Thus, by guaranteeing that people are informed about the use of AI in a straightforward and comprehensive manner, understandability promotes explainability and transparency. The necessity of tracking the decisions made by AI systems is highlighted by traceability in transparency requirements (O2, O12) [28]. In order to ensure openness, Organization O12 also noted how crucial it is to track the data utilized in AI decision-making. 1.5 Transparency Vs Explainability in AI Explainability and transparency are similar concepts [29]. According to McLarney et al. [30], a transparent AI necessitates that ”Basic elements of data and decisions must be available for inspection during and after AI use.” Transparency is achieved when users have access to their data or can understand how decisions are made. On the other hand, explainability seeks to reveal the reasons for AI’s successes or failures and demonstrate how it utilizes the knowledge and judgment of those it affects. It provides a rational justification for the actions of the AI. Users must clearly know what data is collected, how the AI interprets this data, and how it produces reliable outcomes for each affected individual. This straightforward explanation overlooks the challenges we face when trying to clarify ”black box” algorithms, the context that may be omitted, and the accuracy needed when offering understandable explanations to customers. Therefore, the question arises: is having minimal explainability preferable to having none at all? [30]. Additionally, the belief that explanations can adequately address the dynamic nature of the rich information ecosystem and the appropriateness of managing anomalies are also vital factors to consider. <details> <summary>x4.jpg Details</summary>  ### Visual Description ## System Overview: Multi-Agent Path Finding (MAPF) Visualization This document details the visualization of a Multi-Agent Path Finding (MAPF) system. The visualization displays agents navigating a grid-based environment, avoiding collisions and reaching their designated goals. ### 1. Environment Representation * **Grid:** The environment is represented as a 2D grid. Each cell in the grid can be either: * **Free Space:** Represents navigable areas. * **Obstacle:** Represents blocked areas that agents cannot traverse. * **Dimensions:** The grid dimensions are configurable (e.g., 20x20, 50x50). * **Visualization:** Free space is typically displayed as white or a light color, while obstacles are displayed as black or a dark color. ### 2. Agent Representation * **Shape:** Agents are visually represented as circles, squares, or other distinct shapes. * **Color:** Each agent is assigned a unique color for easy identification. * **Size:** Agent size is proportional to their radius or dimensions. * **Goal:** Each agent has a designated goal location, marked on the grid with a distinct symbol (e.g., a star, a flag). ### 3. Path Visualization * **Path Lines:** The planned path for each agent is visualized as a series of connected lines. * **Path Color:** Path lines are colored differently from the agent's color to distinguish the planned path from the agent's current position. * **Path Thickness:** Path line thickness can be adjusted for clarity. * **Real-time Updates:** The path visualization updates in real-time as agents move and replan their paths. ### 4. Collision Avoidance Visualization * **Collision Detection:** The system detects potential collisions between agents. * **Collision Warning:** When a potential collision is detected, a visual warning is displayed (e.g., a flashing border around the agents, a highlighted area). * **Resolution:** The visualization shows how the system resolves collisions, such as agents slowing down, stopping, or changing their paths. ### 5. Performance Metrics The visualization can display performance metrics such as: | Metric | Description | | ------------------ | ----------------------------------------- | | **Makespan** | Total time taken for all agents to reach goals | | **Total Cost** | Sum of the distances traveled by all agents | | **Number of Collisions** | Total number of collisions that occurred | | **Computation Time** | Time taken to compute the paths | ### 6. User Interface (UI) Controls * **Start/Pause/Stop:** Buttons to control the simulation. * **Grid Size:** Input field to adjust the grid dimensions. * **Number of Agents:** Input field to adjust the number of agents. * **Agent Speed:** Slider to control the agent's movement speed. * **Algorithm Selection:** Dropdown menu to select different MAPF algorithms (e.g., CBS, ECBS). * **Visualization Options:** Checkboxes or toggles to control the display of paths, collisions, and performance metrics. ### 7. Algorithm Implementations The visualization supports various MAPF algorithms, including: * **Conflict-Based Search (CBS):** A two-level search algorithm that finds optimal solutions. * **Enhanced CBS (ECBS):** An improved version of CBS that reduces search time. * **Prioritized Planning:** An algorithm that prioritizes agents based on their distance to the goal. * **Velocity Obstacles:** An approach that uses velocity obstacles to avoid collisions. </details> Figure 7: Output from the Bing search engine’s conversation feature explaining a failure. a partial screenshot taken using an Android smartphone on March 2, 2023. [17] It’s interesting to note that although certain AI algorithms evaluate data automatically, more and more AI systems are made to explain how their algorithms operate and the logic behind specific choices [17]. For instance, the conversation mode of the Bing search engine provides succinct explanations of its operation (Fig. 7). Sometimes, end users might find these explanations sufficient, but other times, they would be perplexed as to how an AI came to a particular conclusion or acted in a particular manner. When individuals are more confused by the explanation given, it is unrealistic to expect them to become more computer-literate [17]. Instead, we must improve the justification of the AI system. 1.6 Definition of Trustworthiness in Artificial Intelligence Creating trustworthy AI systems requires a careful strategy that looks at organizational, ethical, and technical factors. The first step is to set clear standards for trustworthiness. These standards should include accountability, security, privacy, transparency, fairness, and ethical behavior. Using high-quality, unbiased data and clear algorithms that explain AI decisions is essential. Strong security measures and privacy practices protect sensitive information from cyberattacks. It’s important to create accountability frameworks and follow ethical guidelines to ensure responsible AI use. By focusing on user needs and constantly monitoring and updating the systems, AI can stay reliable over time. Applying these principles across all stages of the AI process allows organizations to develop systems that are explainable, equitable, ethical, and robust, which fosters stakeholder and user trust. <details> <summary>extracted/6367585/image/eai4.png Details</summary>  ### Visual Description \n ## Diagram: Trustworthy AI Framework ### Overview The image is a diagram illustrating the components of Trustworthy AI, broken down into three main ethical considerations: Ethics of Algorithms, Ethics of Data, and Ethics of Practice. Each of these main categories is further subdivided into four specific principles. The diagram uses a hierarchical structure with arrows indicating the flow of influence from the central concept of "Trustworthy AI" to its constituent parts. ### Components/Axes The diagram consists of: * **Central Node:** "Trustworthy AI" (light green oval) * **Main Categories (3):** * "Ethics of Algorithms" (light orange rectangle) * "Ethics of Data" (light orange rectangle) * "Ethics of Practice" (light orange rectangle) * **Sub-Categories (12):** * Under "Ethics of Algorithms": "Respect for Human Autonomy", "Prevention of Harm", "Fairness", "Explicability" * Under "Ethics of Data": "Human-centred", "Individual Data Control", "Transparency", "Accountability", "Equality" * Under "Ethics of Practice": "Responsibility", "Liability", "Codes and Regulations" * **Arrows:** Black arrows connect "Trustworthy AI" to each of the three main categories. ### Detailed Analysis / Content Details The diagram presents a structured breakdown of Trustworthy AI. The central concept, "Trustworthy AI", branches out into three core ethical areas. * **Ethics of Algorithms:** This area focuses on the design and function of the algorithms themselves. * "Respect for Human Autonomy" * "Prevention of Harm" * "Fairness" * "Explicability" * **Ethics of Data:** This area concerns the data used to train and operate AI systems. * "Human-centred" * "Individual Data Control" * "Transparency" * "Accountability" * "Equality" * **Ethics of Practice:** This area addresses the implementation and governance of AI systems. * "Responsibility" * "Liability" * "Codes and Regulations" The diagram is laid out horizontally, with "Ethics of Algorithms" on the left, "Ethics of Data" in the center, and "Ethics of Practice" on the right. Each sub-category is contained within its respective main category rectangle. ### Key Observations The diagram emphasizes a holistic approach to Trustworthy AI, recognizing that ethical considerations span the entire lifecycle of an AI system – from algorithm design to data management and practical implementation. The inclusion of "Equality" under "Ethics of Data" and "Codes and Regulations" under "Ethics of Practice" suggests a focus on both fairness and legal compliance. ### Interpretation This diagram illustrates a framework for developing and deploying AI systems responsibly. It suggests that Trustworthy AI is not simply a technical challenge, but a multifaceted ethical one. The diagram's structure implies that all three ethical areas – Algorithms, Data, and Practice – are equally important and interconnected. A failure in any one area could compromise the overall trustworthiness of the AI system. The diagram serves as a visual reminder of the key principles that should guide the development and use of AI, promoting a human-centered and ethically sound approach. The diagram does not contain any quantitative data, but rather presents a qualitative framework for ethical considerations. </details> Figure 8: The three key components of XAI: Algorithmic Ethics, Data Ethics, and Practice Ethics. [31] The three elements illustrated in Figure 8 —algorithmic ethics, data ethics, and practice ethics—intersect to create responsible AI. These elements define a data-centered way to handle ethical issues [31]. However, several open challenges still remain with respect to dealing with ethical issues in AI systems. In the work [31], authors give the vision of Trustworthy AI, which mentions that: 1. Human agency and oversight: AI systems have to enable human freedom. They need to facilitate user choice, safeguard fundamental rights, and enable human control. This will assist in developing an equitable and just society. 1. Security and technical robustness: Security and technical proficiency are crucial to prevent damage. In order to enable an AI system to operate efficiently and reduce risks, its creators ought to consider potential risks when designing it. They range from environmental alterations where the system will operate to attacks by malicious individuals. 1. Data protection and data governance: Privacy is a fundamental right that has been highly compromised with the vast amounts of data that artificial intelligence systems gather. It is necessary to protect individual privacy to prevent potential harm. For this purpose, robust data governance must be in place. This includes making sure that the information being utilized is precise and applicable. Furthermore, there is a necessity to establish definite rules for data access and how data must be treated while maintaining the integrity of privacy. 1. Transparency: Explainability and transparency are pretty much dependent on each other. The key objective is to make data, technology, and business models clear. In today’s age, which is the age of pervasive technology, transparency has become a must. It aids customers in comprehending the huge volumes of data collected and the ensuing benefits. 1. Fairness, diversity, and nondiscrimination: Including several voices in AI systems is vital to achieve XAI. We need to involve all individuals who may be affected to ensure equal treatment and access. Fairness and this requirement come hand in hand. 1. Social and environmental welfare: We have to think about the environment and community in seeking justice and doing no harm. We should finance research on AI solutions to global issues. This will make AI systems environmentally friendly and sustainable. AI is supposed to be for the good of all people, including future generations.. 1. Accountability: Accountability and fairness are essential in the context of AI. We need to have systems of holding AI systems accountable for their actions and generated results. Accountability needs to be a constituent part of AI development, deployment, and use, both during and following the activities. 1.7 Impact of XAI on Zero Trust Architecture (ZTA) Zero Trust Architecture (ZTA) is a security system that always checks every request, no matter where it comes from, and does not assume trust automatically. Explainable AI (XAI) helps ZTA by ensuring that decisions made by AI in security are clear and reasonable. XAI is especially useful in identity verification, access control, and spotting unusual behavior. AI models analyze how users behave and identify any suspicious activities. By adding explainability, security analysts can better understand and confirm AI-driven security rules. This reduces false alarms and speeds up response times. For example, AI-driven network monitoring systems that use ZTA principles can explain why a specific access attempt looks suspicious. This explanation builds trust in automated cybersecurity decisions [32]. 1.8 An Overview of Necessities for Reliable AI Despite heated societal discussions, the requirements for trustworthy AI remain ambiguous and are handled inconsistently by numerous organizations and organizations. Globally, accountability, explainability, verifiability, and fairness are all part of the Fairness, Responsibility, Accuracy, Verifiability, and Accountability in Machine Learning (FAT-ML) principles [33]. Explainability, fairness, privacy, and robustness are just a few of the many needs that will be examined in this study (Table 1). Table 1: Conditions necessary for trustworthy artificial intelligence (AI) | Explainability | To help consumers comprehend, the method by which the AI model generates its output might be demonstrated. | | --- | --- | | Fairness | Regardless of certain protected variables, the AI model’s output can be shown. | | Privacy | It is feasible to prevent issues with personal data that might arise while the AI is being developed. | | Robustness | The AI model can fend against outside threats while continuing to operate correctly. | 2 XAI Vs AI XAI improves AI systems by focusing on transparency, clear explanations, and accountability. It offers understandable reasons for decisions, which helps users trust the system and makes it easier to assess fairness compared to traditional ”black box” methods. The main difference between reliable XAI and traditional AI is how they make decisions. While AI can give accurate forecasts or suggestions, reliable XAI emphasizes the need to explain the steps that lead to these results. Clear explanations from XAI systems allow users to judge the fairness and reliability of AI-generated outcomes. To improve security and maintain transparency in AI-driven cybersecurity, we need to integrate XAI into Zero Trust Architecture (ZTA). When explainability methods clarify why certain decisions are made, people can better understand and trust the AI-driven access control and behavioral analytics in ZTA. As we face compliance and operational challenges, future cybersecurity frameworks will rely more on AI automation. It will be essential to ensure that these AI systems can be easily explained [34]. XAI focuses on more than just providing explanations; it also considers ethical issues. AI development processes that follow the principles of Fairness, Accountability, and Transparency (FAT) help ensure that AI systems meet ethical and legal standards. By prioritizing ethical standards, XAI aims to reduce biases, discrimination, and other harmful effects of AI technology. Trustworthy AI is an approach that emphasizes user safety and transparency. Responsible AI developers clearly explain to clients and the public how the technology works, what it is meant for, and its limitations, since no model is perfect. Table 2: Seven Requirements to Meet in Order to Develop Reliable AI | Human Authority and Supervision | Artificial intelligence technology ought to uphold human agency and basic rights, instead of limiting or impeding human autonomy. | The right to get human assistance | Recital 71, Art 22 | | --- | --- | --- | --- | | Robustness and Safety | Systems must be dependable, safe, robust enough to tolerate mistakes or inconsistencies, and capable of deviating from a totally automated decision | Art 22 | | | Data Governance and Privacy | Individuals should be in total control of the information that is about them, and information about them should not be used against them | Notification and information access rights regarding the logic used in automated processes | Art 13, 14, and 15 | | Transparency | Systems using AI ought to be transparent and traceable | The right to get clarification | Recital 71 | | Diversity and Fairness | AI systems have to provide accessibility and take into account the whole spectrum of human capacities, requirements, and standards | Right to not have decisions made only by machines | Art 22 | | Environmental and Social Well-Being | AI should be utilized to promote social change, accountability, and environmental sustainability | Accurate knowledge regarding the importance and possible consequences of making decisions exclusively through automation | Art 13, 14, and 15 | | Accountability | Establishing procedures to guarantee that AI systems and their outcomes are held accountable is essential | Right to be informed when decisions are made only by machines | Art 13, 14 | 3 Applications of XAI Authentic XAI has numerous uses in sectors where accountability, interpretability, and transparency are essential. XAI can provide an explanation for a diagnosis or therapy recommendation in medical diagnosis and recommendation systems. Financial institutions can employ XAI for risk assessment, fraud detection, and credit scoring. XAI can help attorneys with contract analysis, lawsuit prediction, and legal research. In autonomous vehicles, XAI plays a significant role in providing context for the decisions made by the AI systems, particularly in high-stakes scenarios such as accidents or unanticipated roadside incidents. XAI can be applied to process optimization, predictive maintenance, and quality control in manufacturing settings. By offering justifications for automated responses or suggestions in chatbots and virtual assistants, XAI can improve customer service. By providing an explanation for the recommendations and assessments made by adaptive learning systems, XAI can help with individualized learning. By providing an explanation for the recommendations and assessments made by adaptive learning systems, XAI can help with individualized learning. We shall concentrate on a few particular applications in this section and go into detail about them. 3.1 Application of XAI in Medical Science The field of artificial intelligence (AI) is rapidly growing on a global scale, particularly in healthcare, which is a hot topic for research [35]. There are numerous opportunities to utilize AI technology in the healthcare sector, where the well-being of individuals is at stake, due to its significant relevance and the vast amounts of digital medical data that have been collected [36]. AI has enabled us to perform tasks quickly that were previously unfeasible with traditional technologies. The trustworthiness and openness of AI systems are becoming increasingly important, especially in areas like healthcare. As AI is used more in medical decision-making, people are worried about how reliable and understandable its results are. These worries highlight the need to evaluate AI models carefully to make sure their predictions are based on important and verifiable factors. In critical situations like medicine, proving that AI systems are credible is vital for their safe and effective use. In the medical field, clinical decision support systems (CDSS) utilize AI technology to assist healthcare professionals with critical tasks such as diagnosis and treatment planning [37]. While these systems aim to support healthcare practitioners, misuse can have severe consequences in situations where lives are at risk. For example, false alarms, which are common in scenarios involving urgent patients, can lead to exhaustion among medical personnel. The study [38] significantly contributes to medical skin lesion diagnostics in several ways. First, it modifies an existing explainable AI (XAI) technique to boost user confidence and trust in AI systems. This change involves developing an AI model that can distinguish between different types of skin lesions. The study uses synthetic examples and counter-examples to create explanations that highlight the key features influencing classification decisions. The research [38] trains a deep learning classifier with the ISIC 2019 dataset using the ResNet architecture. This allows professionals to use the explanations to reason effectively. Overall, the study’s main contributions lie in its refinement and evaluation of the XAI technique in a real-world medical setting, its analysis of the latent space, and its thorough user study to assess how effective the explanations are, particularly among experts in the field. This research paper [39] discusses how to recognize brain tumors in MRI images using two effective algorithms: fuzzy C-means (FCM) and Artificial Neural Network (ANN). The authors aim to make the tumor segmentation process more understandable and improve accuracy in identifying tumors. Their main goal is to enhance tools that help doctors diagnose brain tumors more accurately. This research offers two key benefits. First, it helps identify brain cancers in medical images more precisely, which is crucial for early diagnosis and treatment. Second, by incorporating XAI principles into the segmentation process, the researchers make their models’ decisions clearer and easier to understand for patients and medical experts. In summary, this increased clarity boosts the overall trust and acceptance of AI-driven systems in medical image analysis within clinical settings. This study [40] discusses how AI and machine learning can help diagnose whole slide images (WSIs) in pathology. While AI can improve accuracy and efficiency, concerns exist about its reliability because it can be hard to understand. To address these issues, the article suggests using explainable AI methods, which help clarify how AI makes decisions. By adding XAI, pathology systems become more transparent and trustworthy, especially for critical tasks like diagnosing diseases. The study also introduces HistoMapr-Breast, a software tool that uses XAI to assist with breast core biopsies. A recent study examines the importance of making sure AI systems in healthcare are accurate and strong, especially regarding how easy they are to understand and how well they can resist attacks [41]. As AI becomes more common in medical settings, it’s crucial to verify that the predictions these systems make rely on trustworthy features. To tackle this challenge, researchers have proposed various methods to improve model interpretability and explainability. The study shows that adversarial attacks can affect a model’s explainability, even when the model has strong training. Additionally, the authors introduce two types of attack classifiers: one that tells apart harmless and harmful inputs, and another that determines the nature of the attack. This research paper [42] looks at explainable machine learning in cardiology. It discusses the challenges of understanding complex prediction models and how these models affect important healthcare decisions. The study explains the main ideas and methods of explainable machine learning, helping cardiologists understand the benefits and limitations of this approach. The goal is to improve decision-making in clinical settings by offering guidance on when to use easy-to-understand models versus complex ones. This can help improve patient outcomes while ensuring accountability and transparency in predictions. Figure 9 shows a decision tree created from the predictions of a random forest model. This global tree diagram illustrates how the random forest works overall. By following a patient’s path through the tree, individual predictions can be examined. This type of explanation is beneficial because it clarifies both the general functioning of the model and the reasoning behind specific predictions. Decision trees are suitable for fields like cardiology because they use rule-based reasoning similar to clinical decision guidelines. <details> <summary>extracted/6367585/image/eai5.png Details</summary>  ### Visual Description \n ## Decision Tree: Probability of Heart Disease ### Overview The image presents two decision trees, labeled "Global" and "Local", both aiming to predict the probability of heart disease based on a series of medical tests and patient characteristics. The trees visually represent a series of binary decisions leading to a final probability estimate. Both trees have identical structure and values. ### Components/Axes The decision trees are structured with nodes representing tests or characteristics, and branches representing the outcomes of those tests. The leaf nodes represent the estimated probability of heart disease. The following characteristics are used: * **Thallium Stress Test:** Normal vs. Not Normal * **ST Depression:** ≤ 0.7 vs. > 0.7 * **Age:** ≤ 54.5 vs. > 54.5 * **Cholesterol:** ≤ 228 vs. > 228 * **Chest Pain Type:** Typical Angina vs. Atypical Angina * **Max Heart Rate:** ≤ 120 vs. > 120 * **ST Depression:** ≤ 2.1 vs. > 2.1 The final probabilities are displayed at the leaf nodes. The trees are labeled "Global" and "Local" at the top-left. The phrase "Probability of Heart Disease" is present at the bottom-left of each tree. ### Detailed Analysis or Content Details Both trees are identical, so the analysis will cover one tree. The tree starts with the "Thallium Stress Test". * **Thallium Stress Test = Normal:** The tree branches to "Age". * **Age ≤ 54.5:** Branches to "Cholesterol". * **Cholesterol ≤ 228:** Probability = 0.235 * **Cholesterol > 228:** Probability = 0.692 * **Age > 54.5:** Branches to "ST Depression". * **ST Depression ≤ 2.1:** Probability = 0.318 * **ST Depression > 2.1:** Probability = 0.875 * **Thallium Stress Test ≠ Normal:** The tree branches to "ST Depression". * **ST Depression ≤ 0.7:** Branches to "Chest Pain Type". * **Chest Pain Type = Typical Angina:** Probability = 0.916 * **Chest Pain Type = Atypical Angina:** Probability = 0.286 * **ST Depression > 0.7:** Branches to "Max Heart Rate". * **Max Heart Rate ≤ 120:** Probability = 1.000 * **Max Heart Rate > 120:** Probability = 0.069 The probabilities are represented as numerical values within red-bordered circles. The "Local" tree has a green circle around the probability of 0.069. ### Key Observations The probabilities vary significantly depending on the combination of factors. A "Normal" Thallium Stress Test generally indicates a lower probability of heart disease, while an "Abnormal" test often leads to higher probabilities. The "Max Heart Rate > 120" and "ST Depression > 0.7" combinations result in the highest probabilities (1.000 and 0.875 respectively). The "Local" tree highlights the probability of 0.069. ### Interpretation The decision trees provide a simplified model for assessing the probability of heart disease based on a set of clinical variables. The structure of the tree suggests that the "Thallium Stress Test" is considered the most important initial indicator, followed by age, cholesterol, chest pain type, and max heart rate. The probabilities at the leaf nodes represent the estimated risk level for individuals falling into those specific categories. The highlighting of 0.069 in the "Local" tree suggests this value is of particular interest or represents a specific case being analyzed. The trees demonstrate a clear relationship between medical test results and the likelihood of heart disease, offering a visual representation of a diagnostic process. The identical structure of the "Global" and "Local" trees suggests they represent the same underlying model, potentially applied to different datasets or contexts. </details> Figure 9: A model using random forests predicts heart disease by analyzing both local and global decision trees. The global diagram starts by examining whether a patient’s thallium stress test results are normal. If the test shows a problem, the model looks at the patient’s ST depression next. The local graphic shows the specific pathway a patient took in the model, explaining the reasons for their individual prediction. For example, a patient under 54.5 years old, with a maximum heart rate that is high and normal thallium stress test results, has a very low chance of having heart disease [42]. Figure 10 shows the LIME explanations for our heart failure model’s two local predictions. The authors explain how these predictions serve as a clinical decision support tool in Epic, which is an electronic health record designed for doctors (Epic Systems Corporation, Verona, Wisconsin, USA). This kind of explanation helps to clarify the clinical factors that affect each prediction. Importantly, this type of explanation can be added to an EHR, which may improve the practical use of a complex model by making forecasts and clear explanations easy to integrate into clinical work. <details> <summary>x5.jpg Details</summary>  ### Visual Description ## Diagram: Heart Disease Probability Assessment ### Overview The image presents a visual assessment of the probability of heart disease for multiple patients. The assessment is represented as a series of stacked blocks, each representing a patient, with a color-coded probability percentage. Alongside each probability block are factors contributing to the prediction. The diagram appears to compare patient data to predict heart disease risk. ### Components/Axes The diagram consists of the following components: * **Patient Blocks:** Stacked rectangular blocks, each representing a patient. The blocks are colored based on the probability of heart disease (red for high, yellow for moderate, green for low). * **Probability Percentage:** A large percentage value displayed within each patient block, indicating the probability of heart disease. * **Probability Labels:** A header section labeled "Probability of Heart Disease" appears twice, once for the high probability assessment and once for the low probability assessment. * **Factors Contributing to Prediction:** A list of factors used in the prediction, along with their corresponding values for each patient. * **Date/Time:** "Probability calculated: 7/21/2021 20:26" is displayed twice. * **Legend:** A small legend in the top-left corner labels the columns as "Patient Name" and "Probability of Heart Disease". ### Detailed Analysis or Content Details The diagram presents data for six patients, with the following probabilities: 1. **Patient 1:** 94% (Red Block) - No contributing factors are listed. 2. **Patient 2:** 79% (Red Block) - Factors: * Thallium Stress Test: Normal * Number of Major Vessels: 1 * Exercise Induced Angina: No * Chest Pain Type: Typical Angina * Max Heart Rate Achieved: 174 * Sex: Male 3. **Patient 3:** 55% (Yellow Block) - No contributing factors are listed. 4. **Patient 4:** 49% (Yellow Block) - No contributing factors are listed. 5. **Patient 5:** 31% (Green Block) - No contributing factors are listed. 6. **Patient 6:** 26% (Green Block) - Factors: * Thallium Stress Test: Normal * Max Heart Rate Achieved: 131 * Number of Major Vessels: 1 * Sex: Male * ST Depression: 0.10 * Age: 69 The "Probability calculated" timestamp is consistent across both detailed assessments: 7/21/2021 20:26. ### Key Observations * The probabilities range from 26% to 94%, indicating a wide variation in heart disease risk among the patients. * The patients with the highest probabilities (94% and 79%) have detailed contributing factors listed, while those with moderate and low probabilities often lack this information. * The factors listed for the high-probability patient (79%) suggest a combination of risk factors, including typical angina and a high maximum heart rate. * The low-probability patient (26%) has factors like normal Thallium Stress Test, but also ST Depression and age (69) which are potential risk factors. * The diagram uses a color scheme to visually represent the probability levels: red for high, yellow for moderate, and green for low. ### Interpretation The diagram demonstrates a heart disease risk assessment process. The varying probabilities suggest that the assessment considers multiple factors, and the presence or absence of these factors significantly impacts the predicted risk. The lack of detailed factors for some patients could indicate incomplete data or a simpler assessment model for those cases. The factors listed provide insight into the criteria used for the assessment. For example, the presence of exercise-induced angina and a high maximum heart rate are associated with increased risk, while a normal Thallium Stress Test is associated with lower risk. The inclusion of age and ST depression in the low-probability assessment suggests that even patients with generally favorable indicators may still be at risk. The diagram highlights the complexity of heart disease risk assessment and the importance of considering multiple factors to arrive at an accurate prediction. The visual representation makes it easy to compare the risk levels of different patients and identify potential areas for intervention. The timestamp suggests the assessment is dynamic and can be recalculated based on updated data. </details> Figure 10: Heart disease prediction explanation produced using Local Interpretable Model-Agnostic Explanations (LIME). This illustration shows how clinical decision assistance can be integrated into an Epic electronic health record by means of a local explanation utilizing the LIME algorithm. To help clinicians identify patients who are likely to be at a high risk of heart disease, probabilities are color-coded. To improve the predictability and actionability of the results for doctors, the clinical factors that are most significant to the prediction are shown on the right [42]. For medical AI to work reliably and be widely used, we need to do a lot of research and reach an agreement on important features like explainability, fairness, privacy, and reliability [33]. We must meet clear requirements and standards in any healthcare setting that uses AI, and we need to update these regularly. Additionally, we should establish laws that clarify who is responsible if something goes wrong with a medical AI whether that’s the designers, researchers, healthcare workers, or patients [43]. 3.2 Explainability and Interpretability of Autonomous Systems Explainability and interpretability are crucial concepts in the context of autonomous systems, referring to the ability to understand the decisions and behaviors of these systems. Explainability involves an autonomous system’s capacity to provide clear justifications for its actions and choices [44]. This clarity is essential for fostering acceptance and confidence in AI systems, especially in critical fields such as banking, healthcare, and autonomous vehicles. While explainability and interpretability are closely related, interpretability focuses more on understanding the internal mechanisms and processes of the autonomous system [45]. An interpretable system offers users insight into the factors and criteria that influence its decision-making, enabling them to grasp how the system arrived at its conclusions. The research paper The research article [18] focuses on trust and dependability in autonomous systems. Autonomous systems have the potential for system operation, rapid information dissemination, massive data processing, working in hazardous environments, operating with greater resilience and tenacity than humans, and even astronomical examination [46], [47]. Following years of research and development, today’s automated technologies represent the peak of progress in computer recognition, responsive systems, user-friendly interface design, and sensing automation. According to [44], the global market for automotive intelligent hardware, operations, and innovation is projected to grow significantly, increasing from $1.25billionin2017to$ 28.5 billion by 2025. Intel’s research on the expected benefits of autonomous vehicles indicates that implementing these technologies on public roads could reduce annual commute times by 250 million hours and save over 500,000 lives in the United States between 2035 and 2045 [44]. Modern cars utilize artificial intelligence for various functions, including intelligent cruise control, automatic driving and parking, and blind-spot detection (Figure 11). Authors [18] describe the challenges of autonomous systems, like, people sometimes tend to be overly excited about the potential of new ideas and ignore, or at least appear to be unaware of, the potential drawbacks of cutting-edge developments. Even in the early stages of robotics and autonomous system implementation, humanity preferred to put up with faulty goods and services, but they have gradually come to understand the importance of trustworthy and dependable autonomous systems. Numerous examples have demonstrated how operators’ use of automation is greatly impacted by trustworthiness. <details> <summary>x6.jpg Details</summary>  ### Visual Description \n ## Photograph: Street Scene with Object Detection and Dashboard Display ### Overview The image depicts a street scene, likely in a European city, as viewed from the driver's perspective inside a vehicle. Several objects – pedestrians, cyclists, and cars – are highlighted with bounding boxes. A dashboard display is visible in the foreground, showing vehicle data and a warning message. ### Components/Axes The image is composed of the following elements: * **Street Scene:** Buildings, road, sidewalks, trees, and other vehicles. * **Object Detection Boxes:** Red boxes around pedestrians and cyclists, blue boxes around cars. * **Dashboard Display:** Contains numerical readings and a warning message. * **Directional Arrows:** Blue double arrows on the road surface. ### Detailed Analysis or Content Details The street scene appears to be a relatively busy urban environment. The buildings are multi-story and have a classic European architectural style. **Object Detection:** * **Pedestrians:** At least three pedestrians are identified with red bounding boxes. One is on the left sidewalk, and two are crossing the street. * **Cyclists:** One cyclist is identified with a red bounding box, riding on the right side of the road. * **Cars:** Multiple cars are identified with blue bounding boxes, parked along the sides of the road and moving in both directions. **Dashboard Display:** * **Numerical Readings:** The display shows the numbers "30" and "32" with units unclear. Below these numbers is "Ankunft: 15:34 Uhr". * **Warning Message:** A prominent red text message reads "Bremsvorgang aktiv" (German). Below this is "Stadtverkehr" and "Achtung!". * **Graphical Elements:** A graph with a wave-like pattern is visible on the right side of the display, labeled "Energieleistung". * **Arrow Indicators:** A blue arrow indicator is present between the numerical readings. **Directional Arrows:** * Two blue double arrows are painted on the road surface, indicating the direction of traffic flow. ### Key Observations * The object detection system is actively identifying pedestrians, cyclists, and cars. * The dashboard display is warning the driver about an active braking process ("Bremsvorgang aktiv"). * The time displayed is 3:34 PM ("15:34 Uhr"). * The dashboard display suggests the vehicle is in urban traffic ("Stadtverkehr"). * The warning message "Achtung!" (Attention!) indicates a potentially hazardous situation. ### Interpretation The image demonstrates a scenario where an advanced driver-assistance system (ADAS) is actively monitoring the surroundings and providing warnings to the driver. The object detection system is identifying potential hazards (pedestrians, cyclists, cars), and the dashboard display is alerting the driver to an active braking event. The "Bremsvorgang aktiv" message suggests the vehicle is automatically applying the brakes, likely in response to a detected hazard. The presence of the "Energieleistung" graph suggests the vehicle may be a hybrid or electric vehicle, and the graph is displaying energy usage information. **German Translation and Analysis:** * **"Bremsvorgang aktiv"**: "Braking process active" - Indicates the vehicle's braking system is engaged. * **"Stadtverkehr"**: "City traffic" - Confirms the vehicle is operating in an urban environment. * **"Achtung!"**: "Attention!" - A general warning signal. * **"Ankunft: 15:34 Uhr"**: "Arrival: 3:34 PM" - Indicates the estimated time of arrival. * **"Energieleistung"**: "Energy performance" - Indicates the energy usage of the vehicle. The combination of these elements suggests a sophisticated vehicle system designed to enhance safety and provide information to the driver in a complex urban environment. The system is actively working to prevent collisions and optimize energy usage. The image is likely a demonstration or test case for such a system. </details> Figure 11: An automated vehicle that provides a clear and understandable rationale for its decisions at that moment serves as a prime example of explainable AI in automated driving [18]. When AI has become prevalent in autonomous vehicle (AV) operations, user trust has been identified as a major issue that is essential to the success of these operations. XAI, which calls for the AI system to give the user explanations for every decision it makes, is a viable approach to fostering user trust for such integrated AI-based driving systems [48]. This work develops explainable Deep Learning (DL) models to improve trustworthiness in autonomous driving systems, driven by the need to improve user trust and the potential of innovative XAI technology in addressing such requirements. The main concept of this [48] research is to frame the decision-making process of autonomous vehicles (AVs) as an image-captioning task, generating textual descriptions of driving scenarios to serve as understandable explanations for humans. The proposed multi-modal deep learning architecture, shown in Figure 12, utilizes Transformers to model the relationship between images and language, generating meaningful descriptions and driving actions. Key contributions include improving the AV decision-making process for better explainability, developing a fully Transformer-based model, and outperforming baseline models. This results in enhanced user trust, valuable insights for AV developers, and improved interpretability through attention mechanisms and goal induction. <details> <summary>extracted/6367585/image/eai8.png Details</summary>  ### Visual Description \n ## Diagram: Image Captioning Architecture ### Overview This diagram illustrates the architecture of an image captioning system, detailing the flow of information from an input image through an image feature encoder, attention network, and language decoder. The system appears to utilize a ResNet or MobileNet for feature extraction, an attention mechanism to focus on relevant image regions, and an LSTM-based decoder to generate a textual description. ### Components/Axes The diagram is divided into three main sections: "Image Feature Encoder" (left), "Attention Network" (center), and "Language Decoder" (right). * **Image Feature Encoder:** Includes components labeled "Input Image", "Resized Image", "Feature extractor", "Resnet Mobile net", "Image features", and dimensions "s = w x h". * **Attention Network:** Contains "Attention Score", "Attended features", and an "Attention" operation (represented by a circled multiplication symbol). * **Language Decoder:** Includes "LSTM", "h₀", "h₁", "<SOS>" (Start of Sentence), "y₁", "y₂", and "Obstacles". * **Data Flow:** Arrows indicate the direction of information flow between components. ### Detailed Analysis or Content Details The process begins with an "Input Image" which is then "Resized". The resized image is fed into a "Feature extractor" (ResNet or MobileNet), which outputs "Image features" with dimensions 's' (height) by 'w' (width) by 'h' (depth). These image features are then passed to the "Attention Network". The "Attention Network" calculates an "Attention Score" and produces "Attended features". These attended features, along with the original "Image features", are fed into an LSTM within the "Language Decoder". The first LSTM receives the "<SOS>" token as input and produces a hidden state "h₀" and an output "y₁". This process is repeated with subsequent LSTMs, receiving the previous hidden state (e.g., "h₁") and the word "Obstacles" as input, generating "y₂", and so on. The outputs "y₁" and "y₂" are then fed back into the attention mechanism. The diagram shows a repeating pattern of LSTM layers and attention mechanisms, suggesting a sequence generation process. The dimensions of the attended features are indicated as s x t, where 't' is likely the sequence length. ### Key Observations * The architecture employs an attention mechanism, allowing the decoder to focus on different parts of the image when generating each word of the caption. * The use of LSTM suggests a recurrent neural network approach to language modeling. * The diagram highlights the key components of an image captioning system, providing a high-level overview of the data flow. * The "Obstacles" input to the LSTM suggests the system may be designed to generate captions that specifically mention obstacles detected in the image. ### Interpretation This diagram represents a typical encoder-decoder architecture for image captioning. The encoder (Image Feature Encoder) transforms the image into a fixed-length vector representation (image features). The decoder (Language Decoder) then uses this representation to generate a sequence of words (the caption). The attention mechanism bridges the gap between the encoder and decoder, allowing the decoder to selectively attend to different parts of the image when generating each word. The inclusion of "Obstacles" as an input to the decoder suggests that the system is designed for a specific application, such as autonomous navigation or robotic perception, where identifying and describing obstacles is crucial. The diagram effectively illustrates the interplay between visual and linguistic information in the task of image captioning. The repeating LSTM structure indicates the system is capable of generating captions of variable length. The use of ResNet or MobileNet as the feature extractor suggests a focus on leveraging pre-trained models for efficient and accurate feature extraction. </details> Figure 12: The Transformers-based multi-modal deep learning architecture that is being suggested [48] This research [49] aims to investigate the integration of XAI into autonomous vehicular systems to improve transparency and human trust. It delves into the functioning of multiple inner vehicle modules, emphasizing the importance of understanding the vehicle’s decision-making processes for user credibility and reliability. The main contribution lies in introducing XAI to the domain of autonomous vehicles, showcasing its role in fostering trust, and highlighting advancements through comparative analysis. The output comprises the creation of visual explanatory techniques and an intrusion detection classifier, which show considerable advances over previous work in terms of transparency and safety in autonomous transportation systems. 3.3 Applications of XAI for Operations in the Industry The process industry is a subset of businesses that manufacture items from raw materials (not components) using formulae or recipes. Given the magnitude and dynamic nature of operations in the process sector, it becomes evident that the next great step ahead will be the capacity for people and AI systems to collaborate to ensure production stability and dependability [50]. AI systems must successfully inform the individuals who share the ecosystem about their objectives, intentions, and findings as the first step toward collaboration. We can hope people to work ”with” automation rather than ”around” it, thanks in part to the systematic approach to XAI. This research [51] focuses on XAI applications in the process industry. The research argues that current AI models are not transparent enough for process industry applications and highlights the need for XAI models that can be understood by human experts. The main contribution is outlining the challenges and research needs for XAI in the process industry. The outcome is to develop XAI models that are safe, reliable, and meet the needs of human users in the process industry. Table 3: Examples of AI applications in process industry operations, including pertinent data, users, and procedures. (RNN = Recurrent Neural Network; KNN = K-Nearest Neighbor; ANN = Artificial Neural Network; SVM = Support Vector Machine; SVR = Support Vector Regression; RF = Random Forest; IF = Isolation Forest) [51] | Reference | Relevant Data | End Users | Application | AI Methods | | --- | --- | --- | --- | --- | | [52], [53], [54] | Process signals | Operator, Process Engineer, Automation engineer | Process monitoring | RNN, KNN | | [55], [56], [57] | Process signals, Alarms, Vibration | Process engineer, Automation engineer, Operator, Maintenance engineer | Fault diagnosis | ANN, SVM, Bayes Classifier | | [58], [59], [60] | Process signals, Acoustic signals | Operator | Event prediction | ANN | | [61], [62], [63] | Process signals | Operator | Soft sensors | SVR, ANN, RF | | [64], [65], [66] | Vibration, Process signals | Operator, Maintenance engineer, Scheduler | Predictive maintenance | RNN, IF | Table 3 shows examples of AI applied to operational activities in the process industry. This table should give an idea of the breadth of use cases, users, relevant data sources, and applicable AI methodologies; however, it is not intended to be a full or systematic examination. 4 Future of Trustworthy (XAI) <details> <summary>x7.jpg Details</summary>  ### Visual Description \n ## Diagram: AI System with Explainable AI and Human Interaction ### Overview The image is a diagram illustrating the interaction between an AI System, Explainable AI (XAI), a human user, and a component labeled "Data Physicalizing & Tangible User Interfaces". It depicts a cyclical flow of information and feedback. The diagram uses boxes and arrows to represent components and their relationships. ### Components/Axes The diagram consists of the following components: * **AI System:** A large rectangular box labeled "AI System" positioned on the left side of the diagram. * **Explainable AI:** A smaller rectangular box labeled "Explainable AI" positioned below and slightly to the left of the "Data Physicalizing & Tangible User Interfaces" box. * **Data Physicalizing & Tangible User Interfaces:** A large rectangular box positioned in the center of the diagram. * **Human User:** A stick figure representing a human user, positioned on the right side of the diagram. * **Decision Output:** A rectangular box labeled "Decision output" connected to the "AI System" and pointing towards the "Human User". * **Human in the loop - feedback:** A rectangular box labeled "Human in the loop - feedback" connecting the "Human User" to the "Data Physicalizing & Tangible User Interfaces" box. * **Explanation Interface:** A rectangular box labeled "Explanation Interface" connecting the "Data Physicalizing & Tangible User Interfaces" box to the "Explainable AI" box. * **Decision Explanation User probes the model:** A rectangular box within the "Explainable AI" box, labeled "Decision Explanation User probes the model". * **Decision Explanation Convey a single explanation:** A rectangular box within the "Explainable AI" box, labeled "Decision Explanation Convey a single explanation". ### Detailed Analysis or Content Details The diagram illustrates the following flow: 1. The "AI System" generates a "Decision output" which is directed towards the "Human User". 2. The "Human User" provides "Human in the loop - feedback" to the "Data Physicalizing & Tangible User Interfaces" component. 3. The "Data Physicalizing & Tangible User Interfaces" component communicates via an "Explanation Interface" to the "Explainable AI" component. 4. The "Explainable AI" component has two internal functions: "Decision Explanation User probes the model" and "Decision Explanation Convey a single explanation". 5. The "Explainable AI" component sends information back to the "AI System", completing the cycle. The arrows indicate a bidirectional flow of information between the "AI System" and the "Data Physicalizing & Tangible User Interfaces", and between the "Explainable AI" and the "Data Physicalizing & Tangible User Interfaces". The arrow from the "AI System" to the "Human User" is unidirectional. ### Key Observations The diagram emphasizes the importance of explainability in AI systems. The inclusion of "Explainable AI" and the "Explanation Interface" suggests a focus on making AI decisions transparent and understandable to human users. The "Human in the loop - feedback" component highlights the role of human input in refining and improving the AI system. The "Data Physicalizing & Tangible User Interfaces" component suggests a focus on making data more accessible and understandable through physical or tangible interactions. ### Interpretation This diagram illustrates a human-centered approach to AI development. It suggests that AI systems should not operate as "black boxes" but should be designed to be explainable and responsive to human feedback. The inclusion of "Data Physicalizing & Tangible User Interfaces" indicates an interest in exploring novel ways to interact with and understand AI-generated data. The cyclical flow of information suggests an iterative process of learning and improvement, where human feedback is used to refine the AI system's decision-making process. The diagram highlights the importance of trust and collaboration between humans and AI systems. The diagram does not contain any numerical data or specific measurements; it is a conceptual representation of a system architecture. </details> Figure 13: Assessing the user’s interaction with XAI [27]. Figure 13 illustrates the precise location of each XAI domain and its relationship with the human user. According to [67], many explanations of AI systems tend to be static and convey only a single message. However, explanations alone do not facilitate true understanding [68]. To enhance comprehension, users should have the ability to explore the system through interactive explanations, as most existing XAI libraries currently lack options for user engagement and explanation customization. This represents a promising avenue for advancing the field of XAI [68] and [67]. Additionally, various efforts have been made to improve human-machine collaboration by moving beyond static explanations. Explainable AI offers a way to improve how people interact with AI systems. As AI technology grows, it is crucial to ensure that these systems are accountable and transparent. XAI helps by clarifying how AI models work and building trust among users. We can expect many new developments in XAI. These include making AI models more open, focusing on human-centered designs, ensuring compliance with regulations, and creating hybrid AI systems. XAI will prioritize designs that are easy for users to understand and provide clear explanations. This clarity will help build trust and encourage more people to use AI systems. Regulatory frameworks are likely to require the use of XAI in important areas to ensure accountability and transparency. Future XAI systems will need to be sensitive to context and provide interactive explanations. This will allow people to engage with AI decisions in real time and adapt to different situations. We must also work to improve digital literacy and tackle ethical issues to ensure that AI systems follow moral principles and society’s values, making XAI technologies accessible to everyone. The success of XAI depends on its ability to bridge the communication gap between AI systems and human users, which encourages cooperation, mutual respect, and trust in an increasingly AI-driven world. This study [69] offers a thorough analysis of XAI, focusing on two primary areas of inquiry: general XAI difficulties and research directions, as well as ML life cycle phase-based challenges and research directions. In order to shed light on the significance of formalism, customization of explanations, encouraging reliable AI, interdisciplinary partnerships, interpretability-performance trade-offs, and other topics, the study synthesizes important points from the body of existing literature. The primary contribution is the methodical synthesis and analysis of the body of literature to identify important problems and future directions for XAI research [69]. The research offers a thorough review of the current state of XAI research and provides insightful information for future studies and breakthroughs in the area by structuring the debate around general issues and ML life cycle phases. The primary finding of the study is the identification and clarification of 39 important points that cover a range of issues and potential avenues for future XAI research. The importance of conveying data quality, utilizing human expertise in model development, applying rule extraction for interpretability, addressing security concerns, investigating XAI for reinforcement learning and safety, and taking into account the implications of privacy rights in explanation are just a few of the many topics covered by these points. Furthermore, the paper indicates directions for further research and application by highlighting the potential contributions that XAI may make to a number of fields, including digital forensics, IoT, and 5G. <details> <summary>x8.jpg Details</summary>  ### Visual Description \n ## Diagram: Challenges and Research Directions of XAI in the Deployment Phase ### Overview The image presents a vertically oriented diagram listing challenges and research directions for Explainable Artificial Intelligence (XAI) during the deployment phase. It consists of a left-aligned label and a series of horizontally oriented rectangular blocks, each containing a specific research area. There is no quantitative data or axes; it's a qualitative listing of topics. ### Components/Axes * **Left Label:** "Challenges and Research Directions of XAI in the Deployment Phase" - positioned on the left side of the diagram. * **Rectangular Blocks:** Ten blocks, stacked vertically, each containing a text label. ### Content Details The following research areas are listed, from top to bottom: 1. Human-machine teaming 2. XAI and security 3. XAI and reinforcement learning 4. XAI and safety 5. Machine-to-machine explanation 6. XAI and privacy 7. Explainable AI planning (XAIP) 8. Explainable recommendation 9. Explainable agency and explainable embodied agents 10. XAI as a service 11. Improving explanations with ontologies ### Key Observations The diagram presents a list of research areas without any prioritization or ranking. The topics cover a broad range of concerns related to deploying XAI systems, including human interaction, security, safety, privacy, and technical aspects like planning and recommendation systems. ### Interpretation The diagram suggests that the successful deployment of XAI requires addressing a multifaceted set of challenges. The listed areas represent key research directions needed to ensure XAI systems are not only explainable but also trustworthy, secure, safe, and aligned with human values. The inclusion of topics like "XAI as a service" and "Improving explanations with ontologies" indicates a move towards more standardized and knowledge-based approaches to XAI. The diagram doesn't provide any insights into the relative importance of these challenges or the relationships between them, but it serves as a useful overview of the current research landscape in XAI deployment. It is a high-level conceptual map rather than a data-driven visualization. </details> Figure 14: Issues and Future Research Paths for XAI throughout its Deployment Stage [69]. Deploying machine learning solutions begins the deployment process and continues until we cease utilizing them, possibly even after that. Figure 14 illustrates the XAI research directions and challenges that were explored for this phase. 5 Conclusions XAI, or Explainable Artificial Intelligence, is becoming important in many industries because it helps solve key challenges with using AI. As AI becomes more common in our daily lives, understanding how it works is essential. XAI provides tools that help people see and understand how AI models make decisions. The main goal of XAI is to make these models easier to understand. It allows people to look inside the ”black box” of AI and see what affects its decisions. The paper gives a clear overview of the key parts of XAI. It also discusses three main areas where XAI can be applied. Finally, the authors talk about the challenges of using XAI and suggest possible future directions. \bmhead Acknowledgements The authors would like to express their sincere gratitude to everyone who encourages and appreciates their scientific work. Declarations Not applicable References - \bibcommenthead - Stephens [2023] Stephens, E.: The mechanical turk: A short history of ‘artificial artificial intelligence’. Cultural Studies 37 (1), 65–87 (2023) - Kaul et al. [2020] Kaul, V., Enslin, S., Gross, S.A.: History of artificial intelligence in medicine. Gastrointest Endosc 92 (4), 807–812 (2020) https://doi.org/10.1016/j.gie.2020.06.040 . Epub 2020 Jun 18 - Buchanan [2005] Buchanan, B.G.: A (very) brief history of artificial intelligence. Ai Magazine 26 (4), 53–53 (2005) - Wang et al. [2021] Wang, L., Liu, Z., Liu, A., Tao, F.: Artificial intelligence in product lifecycle management. The International Journal of Advanced Manufacturing Technology 114, 771–796 (2021) - Shamshiri et al. [2024] Shamshiri, A., Ryu, K.R., Park, J.Y.: Text mining and natural language processing in construction. Automation in Construction 158, 105200 (2024) - Khang et al. [2024] Khang, A., Abdullayev, V., Litvinova, E., Chumachenko, S., Alyar, A.V., Anh, P.: Application of computer vision (cv) in the healthcare ecosystem. In: Computer Vision and AI-Integrated IoT Technologies in the Medical Ecosystem, pp. 1–16. CRC Press, ??? (2024) - Vallès-Peris and Domènech [2023] Vallès-Peris, N., Domènech, M.: Caring in the in-between: a proposal to introduce responsible ai and robotics to healthcare. AI & SOCIETY 38 (4), 1685–1695 (2023) - Biswas et al. [2023] Biswas, A., Abdullah Al, N.M., Ali, M.S., Hossain, I., Ullah, M.A., Talukder, S.: Active learning on medical image. In: Data Driven Approaches on Medical Imaging, pp. 51–67. Springer, ??? (2023) - Biswas and Islam [2022] Biswas, A., Islam, M.S.: Mri brain tumor classification technique using fuzzy c-means clustering and artificial neural network. In: International Conference on Artificial Intelligence for Smart Community: AISC 2020, 17–18 December, Universiti Teknologi Petronas, Malaysia, pp. 1005–1012 (2022). Springer - Zohuri and Moghaddam [2020] Zohuri, B., Moghaddam, M.: From business intelligence to artificial intelligence. Journal of Material Sciences & Manufacturing Research. SRC/JMSMR/102 Page 3 (2020) - Biswas and Islam [2023] Biswas, A., Islam, M.S.: A hybrid deep cnn-svm approach for brain tumor classification. Journal of Information Systems Engineering & Business Intelligence 9 (1) (2023) - Biswas and Islam [2021] Biswas, A., Islam, M.: Ann-based brain tumor classification: Performance analysis using k-means and fcm clustering with various training functions. In: Explainable Artificial Intelligence for Smart Cities, pp. 83–102. CRC Press, ??? (2021) - Biswas et al. [2023] Biswas, A., Md Abdullah Al, N., Imran, A., Sejuty, A.T., Fairooz, F., Puppala, S., Talukder, S.: Generative adversarial networks for data augmentation. In: Data Driven Approaches on Medical Imaging, pp. 159–177. Springer, ??? (2023) - Gong et al. [2023] Gong, T., Zhu, L., Yu, F.R., Tang, T.: Edge intelligence in intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems (2023) - Biswas and Islam [2021] Biswas, A., Islam, M.S.: An efficient cnn model for automated digital handwritten digit classification. Journal of Information Systems Engineering and Business Intelligence 7 (1), 42–55 (2021) - Malik [2019] Malik, A.: Explainable Intelligence Part 1 - XAI, the Third Wave Of AI. https://www.linkedin.com/pulse/explainable-intelligence-part-1-xai-third-wave-ai-ajay-malik/ - Schoenherr et al. [2023] Schoenherr, J.R., Abbas, R., Michael, K., Rivas, P., Anderson, T.D.: Designing ai using a human-centered approach: Explainability and accuracy toward trustworthiness. IEEE Transactions on Technology and Society 4 (1), 9–23 (2023) - Chamola et al. [2023] Chamola, V., Hassija, V., Sulthana, A.R., Ghosh, D., Dhingra, D., Sikdar, B.: A review of trustworthy and explainable artificial intelligence (xai). IEEE Access (2023) - Guleria and Sood [2023] Guleria, P., Sood, M.: Explainable ai and machine learning: performance evaluation and explainability of classifiers on educational data mining inspired career counseling. Education and Information Technologies 28 (1), 1081–1116 (2023) - Mirzaei et al. [2023] Mirzaei, S., Mao, H., Al-Nima, R.R.O., Woo, W.L.: Explainable ai evaluation: A top-down approach for selecting optimal explanations for black box models. Information 15 (1), 4 (2023) - Vyas [2023] Vyas, B.: Explainable ai: Assessing methods to make ai systems more transparent and interpretable. International Journal of New Media Studies: International Peer Reviewed Scholarly Indexed Journal 10 (1), 236–242 (2023) - Wang et al. [2024] Wang, A.Q., Karaman, B.K., Kim, H., Rosenthal, J., Saluja, R., Young, S.I., Sabuncu, M.R.: A framework for interpretability in machine learning for medical imaging. IEEE Access (2024) - Ghnemat et al. [2023] Ghnemat, R., Alodibat, S., Abu Al-Haija, Q.: Explainable artificial intelligence (xai) for deep learning based medical imaging classification. Journal of Imaging 9 (9), 177 (2023) - Gohel et al. [2021] Gohel, P., Singh, P., Mohanty, M.: Explainable ai: current status and future directions. arXiv preprint arXiv:2107.07045 (2021) - Wang and Ding [2024] Wang, P., Ding, H.: The rationality of explanation or human capacity? understanding the impact of explainable artificial intelligence on human-ai trust and decision performance. Information Processing & Management 61 (4), 103732 (2024) - Herm [2023] Herm, L.-V.: Algorithmic decision-making facilities: Perception and design of explainable ai-based decision support systems. PhD thesis, Universität Würzburg (2023) - Thalpage [2023] Thalpage, N.: Unlocking the black box: Explainable artificial intelligence (xai) for trust and transparency in ai systems. Journal of Digital Art & Humanities 4 (1), 31–36 (2023) - Balasubramaniam et al. [2023] Balasubramaniam, N., Kauppinen, M., Rannisto, A., Hiekkanen, K., Kujala, S.: Transparency and explainability of ai systems: From ethical guidelines to requirements. Information and Software Technology 159, 107197 (2023) - Arrieta and et al. [2020] Arrieta, A.B., al.: Explainable artificial intelligence (XAI): Concepts taxonomies opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020) - McLarney and et al. [2021] McLarney, E., al.: NASA framework for the ethical use of artificial intelligence (AI) (2021) - Kumar et al. [2020] Kumar, A., Braud, T., Tarkoma, S., Hui, P.: Trustworthy ai in the age of pervasive computing and big data. In: 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 1–6 (2020). https://doi.org/10.1109/PerComWorkshops48775.2020.9156127 - Guembe et al. [2022] Guembe, B., Azeta, A., Osamor, V., Ekpo, R.: Explainable artificial intelligence, the fourth pillar of zero trust security. Available at SSRN 4331547 (2022) - Kim et al. [2023] Kim, M., Sohn, H., Choi, S., Kim, S.: Requirements for trustworthy artificial intelligence and its application in healthcare. Healthcare Informatics Research 29 (4), 315 (2023) - Charmet et al. [2022] Charmet, F., Tanuwidjaja, H.C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., Zhang, Z.: Explainable artificial intelligence for cybersecurity: a literature survey. Annals of Telecommunications 77 (11), 789–812 (2022) - Jiang et al. [2017] Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., Wang, Y.: Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology 2 (4) (2017) - Davenport and Kalakota [2019] Davenport, T., Kalakota, R.: The potential for artificial intelligence in healthcare. Future healthcare journal 6 (2), 94 (2019) - Jaspers et al. [2011] Jaspers, M.W., Smeulers, M., Vermeulen, H., Peute, L.W.: Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. Journal of the American Medical Informatics Association 18 (3), 327–334 (2011) - Metta et al. [2023] Metta, C., Beretta, A., Guidotti, R., Yin, Y., Gallinari, P., Rinzivillo, S., Giannotti, F.: Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning. International Journal of Data Science and Analytics, 1–13 (2023) - Akpan et al. [2022] Akpan, A.G., Nkubli, F.B., Ezeano, V.N., Okwor, A.C., Ugwuja, M.C., Offiong, U.: Xai for medical image segmentation in medical decision support systems. Explainable Artificial Intelligence in Medical Decision Support Systems 50, 137 (2022) - Tosun et al. [2020] Tosun, A.B., Pullara, F., Becich, M.J., Taylor, D.L., Fine, J.L., Chennubhotla, S.C.: Explainable ai (xai) for anatomic pathology. Advances in Anatomic Pathology 27 (4), 241–250 (2020) - Agrawal et al. [2024] Agrawal, N., Pendharkar, I., Shroff, J., Raghuvanshi, J., Neogi, A., Patil, S., Walambe, R., Kotecha, K.: A-xai: adversarial machine learning for trustable explainability. AI and Ethics, 1–32 (2024) - Petch et al. [2022] Petch, J., Di, S., Nelson, W.: Opening the black box: the promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology 38 (2), 204–213 (2022) - Rajpurkar et al. [2022] Rajpurkar, P., Chen, E., Banerjee, O., Topol, E.J.: Ai in health and medicine. Nature medicine 28 (1), 31–38 (2022) - Atakishiyev et al. [2021] Atakishiyev, S., Salameh, M., Yao, H., Goebel, R.: Explainable artificial intelligence for autonomous driving: A comprehensive overview and field guide for future research directions. arXiv preprint arXiv:2112.11561 (2021) - Alexandrov [2017] Alexandrov, N.: Explainable ai decisions for human-autonomy interactions. In: 17th AIAA Aviation Technology, Integration, and Operations Conference, p. 3991 (2017) - Xu et al. [2019] Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., Zhu, J.: Explainable ai: A brief survey on history, research areas, approaches and challenges. In: Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part II 8, pp. 563–574 (2019). Springer - Yazdanpanah et al. [2021] Yazdanpanah, V., Gerding, E., Stein, S., Dastani, M., Jonker, C.M., Norman, T.: Responsibility research for trustworthy autonomous systems (2021) - Dong et al. [2023] Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., Labi, S.: Why did the ai make that decision? towards an explainable artificial intelligence (xai) for autonomous driving systems. Transportation research part C: emerging technologies 156, 104358 (2023) - Madhav and Tyagi [2022] Madhav, A.S., Tyagi, A.K.: Explainable artificial intelligence (xai): connecting artificial decision-making and human trust in autonomous vehicles. In: Proceedings of Third International Conference on Computing, Communications, and Cyber-Security: IC4S 2021, pp. 123–136 (2022). Springer - Hoffmann et al. [2021] Hoffmann, M.W., Drath, R., Ganz, C.: Proposal for requirements on industrial ai solutions. In: Machine Learning for Cyber Physical Systems: Selected Papers from the International Conference ML4CPS 2020, pp. 63–72 (2021). Springer Berlin Heidelberg - Kotriwala et al. [2021] Kotriwala, A., Klöpper, B., Dix, M., Gopalakrishnan, G., Ziobro, D., Potschka, A.: Xai for operations in the process industry-applications, theses, and research directions. In: AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering, pp. 1–12 (2021) - Mamandipoor et al. [2020] Mamandipoor, B., Majd, M., Sheikhalishahi, S., Modena, C., Osmani, V.: Monitoring and detecting faults in wastewater treatment plants using deep learning. Environmental Monitoring and Assessment 192 (3), 148 (2020) - Cecílio et al. [2014] Cecílio, I., Ottewill, J., Pretlove, J., Thornhill, N.: Nearest neighbors method for detecting transient disturbances in process and electromechanical systems. Journal of Process Control 24, 1382–1393 (2014) - Banjanovic-Mehmedovic et al. [2017] Banjanovic-Mehmedovic, L., Hajdarevic, A., Kantardzic, M., Mehmedovic, F., Dzananovic, I.: Neural network-based data-driven modelling of anomaly detection in thermal power plant. Automatika: časopis za automatiku, mjerenje, elektroniku, računarstvo i komunikacije 58, 69–79 (2017) - Ruiz et al. [2001] Ruiz, D., Canton, J., Nougués, J., Espuna, A., Puigjaner, L.: On-line fault diagnosis system support for reactive scheduling in multipurpose batch chemical plants. Computers & Chemical Engineering 25, 829–837 (2001) - Yélamos et al. [2007] Yélamos, I., Graells, M., Puigjaner, L., Escudero, G.: Simultaneous fault diagnosis in chemical plants using a multilabel approach. AIChE Journal 53, 2871–2884 (2007) - Lucke et al. [2020] Lucke, M., Stief, A., Chioua, M., Ottewill, J., Thornhill, N.: Fault detection and identification combining process measurements and statistical alarms. Control Engineering Practice 94, 104195 (2020) - Dorgo et al. [2018] Dorgo, G., Pigler, P., Haragovics, M., Abonyi, J.: Learning operation strategies from alarm management systems by temporal pattern mining and deep learning. Computer Aided Chemical Engineering 43, 1003–1008 (2018) - Giuliani et al. [2019] Giuliani, M., Camarda, G., Montini, M., Cadei, L., Bianco, A., Shokry, A., Baraldi, P., Zio, E., et al.: Flaring events prediction and prevention through advanced big data analytics and machine learning algorithms. In: Offshore Mediterranean Conference and Exhibition (2019). Offshore Mediterranean Conference - Carter and Briens [2018] Carter, A., Briens, L.: An application of deep learning to detect process upset during pharmaceutical manufacturing using passive acoustic emissions. International journal of pharmaceutics 552, 235–240 (2018) - Desai et al. [2006] Desai, K., Badhe, Y., Tambe, S., Kulkarni, B.: Soft-sensor development for fed-batch bioreactors using support vector regression. Biochemical Engineering Journal 27, 225–239 (2006) - Shang et al. [2014] Shang, C., Yang, F., Huang, D., Lyu, W.: Data-driven soft sensor development based on deep learning technique. Journal of Process Control 24, 223–233 (2014) - Napier and Aldrich [2017] Napier, L., Aldrich, C.: An isamill™ soft sensor based on random forests and principal component analysis. IFAC-PapersOnLine 50, 1175–1180 (2017) - Amihai et al. [2018a] Amihai, I., Gitzel, R., Kotriwala, A., Pareschi, D., Subbiah, S., Sosale, G.: An industrial case study using vibration data and machine learning to predict asset health. In: 2018 IEEE 20th Conference on Business Informatics (CBI), vol. 1, pp. 178–185 (2018). IEEE - Amihai et al. [2018b] Amihai, I., Chioua, M., Gitzel, R., Kotriwala, A., Pareschi, D., Sosale, G., Subbiah, S.: Modeling machine health using gated recurrent units with entity embeddings and k-means clustering. In: 2018 IEEE 16th International Conference on Industrial Informatics (INDIN), pp. 212–217 (2018). IEEE - Kolokas et al. [2020] Kolokas, N., Vafeiadis, T., Ioannidis, D., Tzovaras, D.: Fault prognostics in industrial domains using unsupervised machine learning classifiers. Simulation Modelling Practice and Theory, 102109 (2020) - Abdul et al. [2018] Abdul, A., Vermeulen, J., Wang, D., Lim, B.-Y., Kankanhalli, M.: Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, pp. 1–18 (2018). Association for Computing Machinery - Adadi and Berrada [2018] Adadi, A., Berrada, M.: Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access 6, 52138–52160 (2018) https://doi.org/10.1109/ACCESS.2018.2870052 - Saeed and Omlin [2023] Saeed, W., Omlin, C.: Explainable ai (xai): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263, 110273 (2023)