# Towards Transparent AI: A Survey on Explainable Language Models
**Authors**: \fnmAvash\surPalikhe, \fnmZichong\surWang, \fnmZhipeng\surYin, \fnmRui\surGuo, \fnmQiang\surDuan, \fnmJie\surYang
> apali007@fiu.edu
> ziwang@fiu.edu
> zyin007@fiu.edu
> wenbin.zhang@fiu.edu[[[[
[1] \fnm Wenbin \sur Zhang
1] \orgname Florida International University, \city Miami, \country United States 2] \orgname University of Florida, \orgaddress \street Gainesville, \country United States 3] \orgname Pennsylvania State University, \orgaddress \street Montgomery County, \country United States 4] \orgname University of Wollongong, \orgaddress \street Wollongong, \country Australia
Abstract
Language Models (LMs) have significantly advanced natural language processing and enabled remarkable progress across diverse domains, yet their black-box nature raises critical concerns about the interpretability of their internal mechanisms and decision-making processes. This lack of transparency is particularly problematic for adoption in high-stakes domains, where stakeholders need to understand the rationale behind model outputs to ensure accountability. On the other hand, while explainable artificial intelligence (XAI) methods have been well studied for non-LMs, they face many limitations when applied to LMs due to their complex architectures, considerable training corpora, and broad generalization abilities. Although various surveys have examined XAI in the context of LMs, they often fail to capture the distinct challenges arising from the architectural diversity and evolving capabilities of these models. To bridge this gap, this survey presents a comprehensive review of XAI techniques with a particular emphasis on LMs, organizing them according to their underlying transformer architectures: encoder-only, decoder-only, and encoder-decoder, and analyzing how methods are adapted to each while assessing their respective strengths and limitations. Furthermore, we evaluate these techniques through the dual lenses of plausibility and faithfulness, offering a structured perspective on their effectiveness. Finally, we identify open research challenges and outline promising future directions, aiming to guide ongoing efforts toward the development of robust, transparent, and interpretable XAI methods for LMs.
keywords: XAI, LMs, Explainability, Transparency, Accountability
1 Introduction
In the evolving field of natural language processing (NLP), language models (LMs) such as BERT [1], T5 [2], GPT-4 [3], and LLaMA-2 [4] have demonstrated remarkable progress across a wide range of tasks and application domains, including machine translation, code generation, medical diagnosis, and personalized education [5, 6, 7, 8]. Despite these advancements, LMs inherently exhibit a black-box nature due to their considerable parameter scales, extensive training data, and complex architectures, making it challenging to understand their internal mechanisms and decision-making processes [9]. This lack of transparency introduces several critical limitations, including diminished trust, reduced accountability, and increased difficulty in identifying biases or debugging errors in model outputs. Consequently, these challenges become particularly critical when LMs are deployed in high-stakes domains such as healthcare, finance, and law, where transparency and accountability are essential because errors can lead to serious consequences [10, 9, 11].
On the other hand, explainable artificial intelligence (XAI) methods developed for transparency and accountability in non-LM domains have been extensively studied [12, 13, 14, 15]. For example, feature attribution techniques such as Integrated Gradients [16] are widely adopted in non-LM settings to interpret model predictions by quantifying the contribution of individual input features. However, applying such methods directly to LM is difficult due to their considerable parameter scale and broad generalization abilities [17]. This has motivated researchers to design XAI approaches tailored to LMs. For instance, Chain-of-Thought (CoT) prompting [18, 19] exposes intermediate reasoning steps to improve transparency, while self-explanation methods generate human-readable justifications alongside predictions [20, 21]. Although various existing surveys have examined XAI in the context of LMs, they often overlook the architecture-specific challenges that arise from the diverse combinations of transformer encoder and decoder components [22, 23, 24, 25, 26]. Since each architecture processes and represents information differently [27], these variations lead to distinct explainability issues. Consequently, a unified framework is essential to systematically examine XAI methods across model architectures and to address the unique interpretability challenges posed by LMs.
To this end, this survey provides a systematic review of XAI methods for language models, organizing them by their underlying transformer architectures, discussing their evaluation mechanisms, and identifying open challenges along with future research directions. To the best of our knowledge, this is the first survey to present a comprehensive framework of XAI methods specifically tailored to LMs, introducing a standardized taxonomy based on transformer architectures, analyzing architecture-specific challenges, and assessing existing evaluation strategies. By comparing explainability methods from an architecture-aware perspective, we offer a holistic view of current approaches and their effectiveness. The main contributions of this paper are as follows:
- Proposes a taxonomy of XAI methods for LMs structured around their underlying transformer architectures.
- Provides a comparative analysis of these methods, highlighting key features and limitations across different design choices.
- Explore evaluation strategies for explanations, emphasizing reliability through the dual perspectives of faithfulness and plausibility.
2 Background
2.1 Language Models
LMs have become fundamental to modern NLP, driving significant progress in applications such as machine translation, dialogue generation, question answering, and code synthesis [28, 3]. Most LMs are built upon transformer architectures [29], which leverage self-attention mechanisms to capture contextual dependencies. Based on their architectural design, LMs can be broadly categorized into three groups: encoder-only, decoder-only, and encoder-decoder models, each with distinct features and applications as detailed below.
Encoder-only models, such as BERT [1] and RoBERTa [30], consist solely of the encoder component and process input using bidirectional self-attention. They are pretrained with the masked language modeling objective and excel at tasks requiring text comprehension [31]. In contrast, decoder-only models, including GPT-4 [3] and LLaMA-2 [4], consist exclusively of the decoder component and generate text autoregressively via causal self-attention [32]. By predicting each token conditioned on the preceding tokens, these models excel at tasks involving free-form generation, such as dialogue, storytelling, and code synthesis [27]. Finally, encoder-decoder models, such as T5 [2] and BART [33], integrate both components connected through cross-attention. Trained on sequence-to-sequence objectives, they are well-suited for tasks requiring both understanding and generation, including machine translation and summarization [31].
2.2 Explainable Artificial Intelligence (XAI)
Explainable Artificial Intelligence (XAI) seeks to make AI systems more interpretable and transparent by providing human-understandable insights into their internal reasoning and decision-making processes [23]. As AI continues to be deployed in high-stakes domains such as healthcare, finance, education, and law, the ability to explain predictions has become critical to building trust, ensuring accountability, and meeting regulatory requirements [5, 6, 7, 8]. This growing demand for interpretability has fueled the rapid development of XAI methods and frameworks.
Within this context, non-LMs such as random forests or neural networks typically possess relatively simpler architectures and have been more extensively studied from an explainability perspective [34]. These models generally process shorter contexts and utilize simpler feature representations, which makes their decision-making mechanisms easier to analyze. Consequently, a wide range of XAI techniques have been proposed and successfully applied to these models [12, 13, 14, 15]. For instance, methods, such as Integrated Gradients [16] approximate model predictions by estimating feature contributions, while counterfactual explanation methods identify minimal input perturbations required to alter a modelâs decision, offering interpretable insights into the factors influencing predictions [35]. Due to the simple structure and limited contextual complexity of these models, such approaches tend to yield explanations that are generally more faithful and human-interpretable compared to LMs.
2.3 XAI for LMs
Unlike traditional non-LM models with relatively simpler architectures, language models pose unique explainability challenges due to their transformer-based designs and massive parameter scales [31]. Existing XAI techniques developed for classical models often fail to transfer directly to LMs, as they cannot adequately capture attention flows, contextual embeddings, or emergent reasoning behaviors [17]. Moreover, LMs are built on diverse transformer architectures, encoder-only, decoder-only, and encoderâdecoder, each defined by distinct attention mechanisms and pre-training objectives [27]. These architectural differences necessitate tailored approaches, making the design of effective XAI techniques for LMs complex and essential.
Moreover, the tension between plausibility and faithfulness is a well-known challenge in explainable artificial intelligence across many types of models [22]. For classical models such as random forests or neural networks, XAI techniques often generate explanations that appear plausible to humans but do not always faithfully reflect the modelâs actual decision process [36]. In language models, this problem is amplified by their transformer based architectures, massive parameter scales, and emergent reasoning behaviors [31]. Explanations that seem convincing may overlook the true underlying dynamics of attention flows, contextual embeddings, or reasoning chains. This distinction is especially critical in high stakes applications such as healthcare, finance, and law, where misleading yet plausible explanations can compromise trust, accountability, and safety [37]. Addressing this challenge requires XAI methods explicitly tailored to LMs architectures, alongside rigorous evaluation frameworks that jointly assess plausibility and faithfulness, ensuring explanations are both human interpretable and faithful to the modelâs internal reasoning [38].
3 Taxonomy
To address the diverse explainability challenges posed by language models and to provide actionable guidelines, we present a novel taxonomy for categorizing XAI methods based on their underlying transformer architectures. This taxonomy not only structures existing work but also offers practical guidance for selecting and developing methods suited to different model families. As illustrated in Figure 1, the classification organizes approaches into three groups. First, XAI methods for encoder-only models interpret contextual embeddings by leveraging bidirectional self-attention to analyze internal representations. Second, XAI methods for decoder-only models frequently rely on prompt engineering and in-context learning strategies to reveal the reasoning processes of these autoregressive models with massive parameter scales. Third, XAI methods for encoder-decoder models provide interpretability at both encoding and decoding stages, exploiting cross-attention to trace the flow of information across components. By standardizing XAI methods along these architectural lines, the taxonomy enables meaningful comparisons of their mechanisms and characteristics, highlights architecture specific challenges, and clarifies how explanatory effectiveness can be evaluated.
<details>
<summary>images/xai-llms-taxonomy.png Details</summary>
### Visual Description
\n
## Diagram: XAI Methods in LMs
### Overview
This diagram illustrates the relationship between different Explainable AI (XAI) methods applied to Large Language Models (LMs). It depicts a central node labeled "XAI methods in LMs" with various methods branching out from it, categorized by model architecture (Encoder-only, Decoder-only, Encoder-Decoder). The diagram shows how these methods can be further categorized into specific techniques.
### Components/Axes
The diagram consists of interconnected rectangular boxes representing different XAI methods. The central box is labeled "XAI methods in LMs". The branches represent different model architectures:
* **Encoder-only** (Green)
* **Decoder-only** (Red)
* **Encoder-Decoder** (Blue)
The methods branching from these architectures are:
* Feature attribution
* Classifier-based probing
* Parameter-free probing
* Attention-based
* ICL-based (In-Context Learning)
* CoT prompting (Chain-of-Thought prompting)
* Mechanistic Interpretability
* Self-explanation
### Detailed Analysis or Content Details
The diagram can be broken down into three main sections based on the model architecture:
**1. Encoder-only (Green):**
* Branches into four methods: Feature attribution, Classifier-based probing, Parameter-free probing, and Attention-based.
* The Attention-based method further branches into Feature attribution.
**2. Decoder-only (Red):**
* Branches into three methods: Feature attribution, ICL-based, and CoT prompting.
* CoT prompting branches into Mechanistic Interpretability.
**3. Encoder-Decoder (Blue):**
* Branches into three methods: Classifier-based probing, Attention-based, and Self-explanation.
The arrows indicate a flow or relationship between the model architecture and the XAI methods. The diagram suggests that each model architecture supports a variety of XAI techniques, and some techniques can be applied in multiple architectures.
### Key Observations
* Feature attribution and Attention-based methods appear to be versatile, being applicable across all three model architectures.
* Decoder-only models seem to emphasize methods related to prompting and interpretability (ICL-based, CoT prompting, Mechanistic Interpretability).
* The diagram doesn't provide quantitative data, but rather a qualitative overview of the relationships between XAI methods and model architectures.
### Interpretation
The diagram illustrates the diverse landscape of XAI techniques available for understanding Large Language Models. The categorization by model architecture highlights that the choice of XAI method may depend on the underlying model structure. The branching structure suggests that some methods build upon others (e.g., CoT prompting leading to Mechanistic Interpretability). The diagram serves as a useful visual guide for researchers and practitioners interested in applying XAI to LMs, showing the breadth of options available and potential connections between them. The diagram suggests a hierarchical relationship, where the central "XAI methods in LMs" node represents a broad category, and the branches represent increasingly specific techniques. The diagram does not provide any information about the effectiveness or limitations of each method, but rather focuses on their categorization and relationships.
</details>
Figure 1: The proposed taxonomy for XAI in language models.
4 Overview
Based on the proposed taxonomy, this section presents an overview of XAI methodologies for language models, as summarized in Table LABEL:tab:xai-methods. The table organizes prior studies across six key dimensions: (1) Method, indicating the specific XAI technique used (e.g., LIME, TransSHAP, ROME); (2) Architecture (Arch.), specifying the applicable model type, encoder-only, decoder-only, or encoderâdecoder; (3) Type, denoting the broader class of approach, such as feature attribution, ICL-based, attention-based, or self-explanation; (4) Model, listing the representative language model (e.g., BERT, GPT-2, T5); (5) Scope, distinguishing whether the explanation is local (L) to individual predictions or global (G) to overall model behavior; and (6) Integration (Inte.), identifying when interpretability is introduced, either post hoc (PH) or at prompt time (PT). This structured comparison facilitates a deeper understanding of methodological distinctions, architectural compatibility, and interpretability timing, thereby supporting informed evaluation and selection of XAI techniques for various LM settings.
Table 1: Summary of different methodologies used for explainable AI (XAI) in language models (LMs). Abbreviations: Arch. = Architecture, Inte. = Integration, L = Local, G = Global, PH = Post-hoc, PT = Prompt-time.
| [39] | LIME | Encoder-only | Feature attribution-based | BERT | L | PH |
| --- | --- | --- | --- | --- | --- | --- |
| [40] | TransSHAP | Encoder-only | Feature attribution-based | BERT | L | PH |
| [41] | Edge probing | Encoder-only | Classifier-based probing | BERT | L, G | PH |
| [41] | Token representation probing | Encoder-only | Classifier-based probing | BERT | L | PH |
| [42] | Perturbation masking | Encoder-only | Parameter-free probing | BERT | G | PH |
| [42] | DIRECTPROBE | Encoder-only | Parameter-free probing | BERT | G | PH |
| [43] | BertViz | Encoder-only | Attention-based | BERT | L | PH |
| [44] | ExBERT | Encoder-only | Attention-based | BERT | L | PH |
| [45] | AML | Decoder-only | Feature attribution-based | LLaMA-2 7B, Mistral 7B | L | PH |
| [46] | Progressive Inference | Decoder-only | Feature attribution-based | GPT-2, LLaMA-2 | L | PH |
| [47] | ICL with priors & mappings | Decoder-only | ICL-based analysis | GPT-3, PaLM | G | PH |
| [48] | Saliency-Guided ICL Analysis | Decoder-only | ICL-based analysis | Instruct-GPT, GPT-2 | L | PH |
| [49] | SEA-CoT | Decoder-only | CoT prompting | LLaMA-2 | L | PT |
| [50] | Least-to-Most Prompting | Decoder-only | CoT prompting | GPT-3 | L | PT |
| [51] | ROME | Decoder-only | Mechanistic interpretability | GPT-2 XL | L | PH |
| [52] | ACDC | Decoder-only | Mechanistic interpretability | GPT-2 Small | G | PH |
| [53] | Program generation explanations | Encoder-decoder | Feature attribution-based | T5, CodeT5, CodeReviewer, CodeT5+ | L | PH |
| [54] | TextGenSHAP | Encoder-decoder | Feature attribution-based | T5-large, T5-XXL, T5-FiD | L | PH |
| [55] | Document-level discourse probing | Encoder-decoder | Classifier-based probing | T5, BART | G | PH |
| [56] | Implicit meaning representation probing | Encoder-decoder | Classifier-based probing | T5, BART | G | PH |
| [20] | Attention-Guided Decoding | Encoder-decoder | Attention-based | T5, BART | L | PH |
| [21] | Attention Head Masking | Encoder-decoder | Attention-based | BART, PEGASUS | G | PH |
| [57] | WT5 | Encoder-decoder | Self-explanation | WT5-Base, WT5-11B | L | PT |
| [58] | Few-Shot NLE Transfer | Encoder-decoder | Self-explanation | T5 | L | PT |
5 XAI Methods for LMs
This section provides a comprehensive overview of XAI methods applied to LMs, categorized according to their underlying transformer architectures: encoder-only, decoder-only, and encoder-decoder. For encoder-only models, XAI approaches, such as feature attribution-based methods and classifier-based probing primarily focus on interpreting contextual embeddings and analyzing bidirectional self-attention mechanisms to explain model predictions. In contrast, decoder-only models adopt techniques such as chain-of-thought (CoT) prompting and in-context learning (ICL), which are designed to interpret the autoregressive generation process and often leverage prompt engineering strategies. Finally, encoder-decoder models utilize methods such as attention-based techniques and self-explanation approaches that aim to provide interpretability across both encoding and decoding stages by utilizing cross-attention mechanisms. The following sections discuss in detail the XAI approaches for each category.
5.1 XAI Methods for Encoder-only LMs
XAI methods for encoder-only models seek to provide insights into the internal mechanisms and decision-making processes of models that rely solely on an encoder architecture, such as BERT [1] and RoBERTa [30]. These models employ a bidirectional self-attention mechanism to generate contextualized embeddings, making explanations particularly focused on how contextual information influences token representations [39]. We discuss several key explainability approaches for these models, including feature attribution-based methods [39], classifier-based probing [41], parameter-free probing [42] and attention-based methods [43].
5.1.1 Feature Attribution-based Methods
Feature attribution-based methods evaluate the relevance of individual features, such as words or phrases, to a modelâs prediction [22]. They assign a score to each feature, interpreting its contribution to the output. These methods are particularly well-suited for encoder-only models, which are pre-trained with a masked language modeling (MLM) objective [1]. This objective encourages bidirectional context modeling, yielding stable and context-rich embeddings that can be reliably analyzed to assess the influence of each feature on the final prediction [59].
One popular line of research extends the commonly used feature attribution technique, LIME, from non-LMs to LMs. Specifically, LIME explains a modelâs predictions by approximating them locally with an interpretable model. Building on this technique, SzczepaĆski et al. [39] proposed an approach that adapts LIME to provide explainability for encoder-only LMs such as BERT, fine-tuned for fake news detection. In their method, it explains the predictions of the encoder-only model by highlighting words and their corresponding weights, which represent their impact on the prediction probability. The increase in weight for these words raises the likelihood of the sentence being classified as fake and vice versa. However, LIME has limitations because it may not satisfy additive attributions defined by Lundberg and Lee [40], such as local accuracy, consistency, and missingness. For example, it often fails to ensure local accuracy, as the sum of the feature contributions does not necessarily equal the modelâs actual output for a given instance. Similarly, it doesnât guarantee consistency, where increasing a featureâs importance in the model may not always increase its attribution score. Finally, LIME may not ensure missingness, as features that are absent or masked can still be assigned non-zero attribution scores, potentially leading to misleading interpretations. These shortcomings can reduce the reliability of LIME-based explanations, especially when applied to language models.
Some methods address these limitations by extending SHAP [40] to language models. Specifically, SHAP is an XAI technique that introduces Shapley values as a unified measure of additive feature importance to explain the model decisions. To adapt this technique to encoder-only LMs, such as BERT, Kokalj et al. [17] proposed a method called TransSHAP. They demonstrate the method in a tweet sentiment classification task, where TransSHAP explains BERTâs predictions for positive and negative sentiment using visualization. The visualizer allows the user to compare the direction of the impact, whether positive or negative, as well as the magnitude of the impact for individual words. Furthermore, they also compared their explanations to LIME and found that it scored slightly better in terms of overall user preference, though in specific elements, such as positioning of visualization elements, TransSHAP performed slightly better. However, there are limitations in using SHAP due to challenges in feature removal and estimating Shapley values [22]. For instance, it is unclear how to select the right baseline for choosing appropriate methods for feature removal. Similarly, calculating Shapley values is computationally expensive, as the complexity grows exponentially with the number of features. Despite these challenges, SHAP is widely used and adapted to LMs, providing sequential visualization explanations for interpreting model predictions.
5.1.2 Classifier-based Probing
Classifier-based probing involves training lightweight classifiers on top of encoder-only models, such as BERT [60], to explain the internal representations learned by the model [55]. Unlike feature attribution-based methods, which measure the contribution of input features to a prediction, this approach focuses on understanding what information is encoded within the modelâs hidden representations. The process typically begins by freezing the model parameters and generating representations for input words, phrases, or sentences. These representations are then used to train classifiers that identify the linguistic properties or reasoning abilities captured by the model [22]. This method is particularly suitable for encoder-only models, as they are pretrained using a masked language modeling (MLM) objective, which enables them to process data bidirectionally and produce strong contextualized word embeddings for each token [1].
To utilize classifier-based probing for interpreting encoder-only models, Tenney et al. [41] introduced edge probing, which investigates how models encode sentence structure across a range of syntactic, semantic, local and long-range phenomena. It focuses on core NLP labeling tasks, such as part-of-speech tagging [61], semantic role labeling [62], and coreference resolution [63]. Furthermore, this study finds that contextual models like BERT show notable improvements over non-contextual counterparts, particularly on syntactic tasks compared to semantic ones. These findings suggest that contextual embeddings encode syntactic features more strongly than higher-level semantic information. However, a key limitation of edge probing is that it does not investigate the underlying reasons behind layer-wise behavior [60], as it does not clarify why certain layers specialize in specific linguistic properties. Additionally, it overlooks the functional role of individual token representations, offering limited insights into how specific tokens contribute to the modelâs internal decision-making process.
To address these limitations, Mohebbi et al. [60] investigate the role of token representations in BERTâs representation space, aiming to explain performance trends across various probing tasks. Their analysis spans a set of surface, syntactic, and semantic tasks, showing that BERT typically encodes the knowledge required for these tasks within specific token representations, particularly in the higher layers. This study finds that most positional information decreases through layers whereas sentence-ending tokens play a partial role in transmitting positional knowledge to higher layers of the model. Additionally, grammatical number and tense information was analyzed throughout the model, observing that BERT encodes verb tense and noun number in the sentence-ending token. However, despite the strong performance of these classifier-based probing techniques, Hewitt and Liang [64] raise concerns about their faithfulness. Specifically, they argue that high accuracy in probing tasks does not necessarily confirm that the model internally represents the probed features, as the classifier might learn to exploit spurious correlations rather than revealing true model behavior. Consequently, the probing results should be interpreted with caution, as they may not fully capture the underlying mechanisms driving the modelâs representations. One effective way to mitigate these concern is through parameter-free probing, which we discuss in the next section.
5.1.3 Parameter-free probing
Parameter-free probing techniques avoid training external classifiers or adding additional parameters. Instead, they directly analyze outputs from encoder-only models, such as activation patterns, attention weights, or contextualized embeddings, to infer the presence of linguistic structures [42]. This avoids a key pitfall of classifier-based probing, where the probe itself may inadvertently encode task-specific information in its own parameters rather than faithfully revealing the knowledge inherently represented by the model [64].
To interpret encoder-only models using a parameter-free probing technique, Wu et al. [42] propose perturbed masking, which estimates inter-word correlations and extracts global syntactic information. The method performs a two-stage perturbation: first, a target word is masked; then, an additional word in the sentence is masked to observe how the targetâs contextualized representation shifts. This shift quantifies the second wordâs influence on the first. From these perturbations, impact matrices are derived from BERTâs outputs, capturing inter-word dependencies similar to attention patterns but without relying on intermediate representations. Using these matrices, syntactic trees are reconstructed, revealing that BERT encodes rich syntactic structures.
Similarly, Zhou et al. [65] point out that training classifiers as probes can be unreliable because different representations may require different classifiers, and the choice of classifier can significantly influence the estimated quality of a representation. To address this limitation, they propose a parameter-free technique, DIRECTPROBE [65], an XAI method designed to interpret and evaluate the quality of representations for NLP tasks without relying on auxiliary classifiers. In this approach, instead of assessing performance using a specific classifier, it approximates the version space that represents the set of all classifiers consistent with the training data by analyzing the geometric structure of the embedding space. Furthermore, through hierarchical clustering, the method partitions the feature space into contiguous regions associated with different labels, providing an interpretable view of how representations organize information. Together, these procedures explain model representations while addressing the challenges posed by training auxiliary classifiers.
5.1.4 Attention-based Methods
Encoder-only models such as BERT are built on transformer architectures that employ a fully attention-based mechanism, specifically bidirectional self-attention [1]. Although previously discussed XAI approaches utilize techniques, such as quantifying each inputâs contribution through feature-attribution or training lightweight classifiers to probe what the model has learned, attention-based methods provide a more direct view into model reasoning by visualizing self-attention distributions and showing exactly how weight is allocated across tokens. To leverage this, various tools have been developed to visualize attention, providing concise summaries of relevant information and facilitating user interaction with LMs [44].
To interpret the attention mechanism in encoder-only models, Vig et al. [43] introduced BertViz, an open-source tool designed to visualize attention at multiple scales, each offering a distinct perspective on the modelâs attention behavior. The tool features a high-level model view, which displays all layers and attention heads within a single interface, and a low-level neuron view, which reveals how individual neurons interact to produce attention patterns. It demonstrates the tool on the BERT model and presents various primary use cases, such as identifying relevant attention heads and linking neurons to model behavior. However, interpreting models solely based on attention weights as faithful explanations can be problematic [66]. Specifically, attention distributions are not guaranteed to reflect the true reasoning process of the model, as different configurations of attention weights can yield similar outputs. Consequently, while BertViz provides valuable visual insights, conclusions drawn from attention patterns alone should be approached with caution, as they may not accurately represent the underlying decision-making mechanisms.
To address these concerns, Hoover et al. [44] introduced ExBERT, an interactive visualization tool that provides a dynamic and intuitive view of both attention mechanisms and internal representations within transformer models. The tool comprises the attention view, offering an interactive interface allowing users to examine aggregated attention patterns, and the corpus view, presenting aggregate statistics that summarize the hidden representations of a selected token or group of tokens. Using attention visualization and nearest-neighbor search techniques, ExBERT explores what information attention and representations capture, and also helps uncover latent patterns that influence model predictions in text inputs. Despite its various applications, ExBERT is primarily constrained to local analyses, as it focuses on presenting statistics across a few nearest neighbors for one token at a time [44]. Consequently, it provides limited insights into ability to provide a global understanding of the modelâs representational space.
5.2 XAI Methods for Decoder-only LMs
XAI methods for decoder-only LMs, such as GPT-4 [3] and LLaMA-2 [4], seek to interpret and analyze the reasoning processes of autoregressive architectures that generate text sequentially based on left-to-right context. Unlike XAI methods for encoder-only LMs, which focus on bidirectional self-attention, explainability techniques for decoder-only models account for their unidirectional attention mechanisms [31]. Moreover, the massive parameter scales and generative capabilities of these models pose significant challenges for interpretability, often necessitating the use of prompt engineering strategies [67]. To address these challenges, various explainability techniques have been proposed, including feature attribution-based methods [45], in-context learning (ICL)-based approaches [28], chain-of-thought (CoT) prompting [49], and mechanistic interpretability [51], which will be discussed in the following sections.
5.2.1 Feature Attribution-based Methods
Feature attribution-based methods aim to quantify how much each individual input token contributes to the modelâs final output. As discussed in Section 5.1.1, these methods are well-suited for encoder-only models as they are pretrained using MLM objective and leverage bidirectional self-attention. However, applying them to decoder-only language models poses challenges due to their autoregressive architecture, where token generation is conditioned on the left-hand context. Consequently, recent efforts have focused on designing feature attribution techniques aligned with the autoregressive decoding process inherent in these models.
For instance, to interpret decoder-only models using feature attribution, Barkan et al. [45] proposed Attributive Masking Learning (AML), a method that explains language model predictions by learning input masks. AML trains an attribution model to identify influential tokens in the input for a given prediction. The optimization follows two complementary objectives: (i) masking as much input data as possible while preserving the modelâs original prediction, and (ii) ensuring that applying the complement of the mask significantly alters the prediction. This dual strategy yields explanations aligned with the metric of interest. However, the study has notable limitations. Only one of the five proposed masking approaches, replacement by designated tokens was implemented, leaving alternatives such as replacement by random or contextually predicted tokens unexplored. Moreover, the analysis did not consider broader aspects of explainability, such as stability and fairness.
As another example, Kariyappa et al. [46] introduced Progressive Inference, a framework for computing input attributions to explain predictions of decoder-only sequence classification models. The core idea is that the classification head can generate intermediate predictions at different points in the input sequence, which, due to the causal attention mechanism, depend only on preceding tokens. This allows obtaining predictions on masked subsequences with negligible computational overhead. Using this insight, the framework proposes two methods: Single Pass-Progressive Inference (SP-PI), which derives attributions from differences between consecutive intermediate predictions, and Multi Pass-Progressive Inference (MP-PI), which leverages Kernel SHAP [40] and multiple masked inputs to produce higher-quality attributions. However, this framework has key limitations. For instance, both methods in this framework assume that masked versions of the inputs are valid inputs for the model. Furthermore, MP-PIâs attributions differ from SHAP values because the samples in progressive inference are correlated due to masked attention and intermediate predictions may not accurately represent the modelâs prediction on equivalent masked inputs.
5.2.2 In-Context Learning (ICL)-based Methods
In-Context Learning (ICL)-based methods explain decoder-only LMs by leveraging their ability to perform tasks directly from examples provided in the input prompt without updating model parameters [68]. Unlike feature attribution-based methods, which explain predictions by quantifying the contribution of individual input tokens, ICL-based methods focus on understanding how models leverage contextual demonstrations to guide their outputs, improving interpretability by making the modelâs reasoning process more transparent. This makes them particularly well-suited for decoder-only LMs, as outputs are inherently generated through causal self-attention over left-to-right context, allowing explanations to align with the modelâs autoregressive behavior [22].
To enable explainability through in-context learning (ICL) in decoder-only models, Wei et al. [47] interpret how LMs are influenced by semantic priors and inputâlabel mappings. In this study, they explain this behavior of LMs using two approaches. The first approach is flipped-label ICL, where all the exemplar labels are flipped, creating a conflict between semantic prior knowledge and inputâlabel mappings. In this case, although smaller language models tend to ignore in-context flipped labels and rely primarily on semantic knowledge acquired during pretraining, larger models can override these semantic priors when given contradictory exemplars showing emergent abilities. The second approach is semantically unrelated label ICL (SUL-ICL) where the labels are semantically unrelated to the task as semantic cues are removed with model only performing ICL using input-label mappings. In this case, it indicates that the ability to do SUL-ICL also emerges primarily with scale, with sufficiently large models even capable of performing linear classification in such settings. However, this approach has limitations. In particular, while the study provides empirical evidence that larger models exhibit emergent behavior as the model scale increases, the underlying reasons behind the emergence of these complex abilities remain unclear [69]. Additionally, this study does not investigate the effect of input distribution.
To address this limitation of not probing input distributions, Li et al. [48] explain the model behavior of decoder-only models under ICL using a fine-grained interpretation of input distributions by editing different components of the input text for task-specific purposes. For example, in sentiment analysis, sentiment-indicative terms in the demonstration inputs are replaced with sentiment-neutral ones, finding that such neutral input perturbations have a smaller impact on model performance compared to changing the ground-truth labels. Furthermore, this approach uses contrastive inputâlabel demonstration pairs constructed through label flipping, input modification, and complementary explanations, and compares their saliency maps using both qualitative and quantitative analyses. With this, it interprets the effects of altering ground-truth labels and adding complementary explanations, providing a better understanding of which parts of the demonstrations contribute most to predictions. However, this study has limitations. In particular, it relies on a relatively small sample size, highlighting the need for larger datasets for more robust evaluation. Additionally, all demonstrations used for ICL are randomly selected, which may influence the modelâs reasoning in certain cases.
5.2.3 Chain-of-Thought (CoT) Prompting
Chain-of-Thought (CoT) prompting enables decoder-only language models to articulate intermediate reasoning steps before producing a final answer [18]. In contrast to In-Context Learning (ICL)-based methods, which rely on demonstrations to learn task-specific patterns without exposing the underlying reasoning, CoT prompting encourages models to generate step-by-step rationales in natural language [70]. This technique aligns well with the autoregressive nature of decoder-only LMs, which facilitates the generation of coherent reasoning chains alongside answers. As a result, CoT prompting not only enhances performance on tasks requiring multi-step reasoning but also improves interpretability by providing insight into the modelâs decision-making process [22].
To enhance explainability in decoder-only LMs, Zhou et al. [50] proposed Least-to-Most prompting, a CoT-based strategy designed to make the modelâs intermediate reasoning steps more interpretable while improving performance. The approach operates in two stages: first, it decomposes a complex problem into a sequence of simpler subproblems; second, it solves these subproblems sequentially, where each solution builds on the outcomes of the previous ones. Both stages are implemented via few-shot prompting, requiring no additional training or fine-tuning. By explicitly structuring the reasoning process, this method improves interpretability while achieving strong performance on tasks involving symbolic manipulation, compositional generalization, and mathematical reasoning. However, this approach has limitations. Its explainability is constrained by poor cross-domain generalization since decomposition prompts designed for one domain (e.g., mathematical word problems) are often ineffective for others (e.g., commonsense reasoning). As a result, maintaining transparency and performance requires domain-specific prompt engineering.
Similarly, another Chain-of-Thought prompting technique is proposed by Jie et al. [71], presenting a multifaceted evaluation of interpretability, examining dimensions such as faithfulness, robustness, and utility across multiple commonsense reasoning benchmarks. Unlike studies restricted to a single prompting strategy, this work investigates a broad spectrum of widely used prompting techniques, enabling a more exhaustive assessment. Furthermore, a simple explainability alignment method, termed Self-Entailment Alignment Chain-of-Thought (SEA-CoT), is introduced to enhance interpretability within the reasoning chain. This method operates similarly to Self-Consistency [70] but additionally enforces consistency between the intermediate reasoning steps and the supporting context. However, despite its utility for enhancing explainability, this study does not explore techniques that ground the modelâs responses using external knowledge, leaving open issues related to self-hallucination, where language models produce plausible but non-factual content. Moreover, it does not investigate hybrid approaches, such as neuro-symbolic AI [72], which combine the learning capabilities of neural networks with the decision-making frameworks of symbolic systems which are inherently interpretable.
5.2.4 Mechanistic Interpretability
Mechanistic interpretability aims to understand the internal computations of decoder-only LMs by reverse-engineering how their parameters, neurons, and attention heads collectively contribute to generating predictions [73]. In contrast to interpretability approaches such as CoT prompting, which infer reasoning patterns from observable outputs, mechanistic interpretability focuses on uncovering how models internally represent, manipulate, and propagate information during inference [74]. This approach is particularly well-suited for decoder-only LMs due to their autoregressive architecture with causal self-attention, which enables a structured flow of information from left to right, allowing researchers to trace how previous tokens influence subsequent predictions [31].
To interpret decoder-only LMs using mechanistic interpretability, Meng et al. [51] analyze how factual associations are stored and recalled in autoregressive transformer models. They identify localized, directly editable computations within mid-layer feed-forward modules that mediate factual predictions. Using a causal intervention, they pinpoint neuron activations critical for these predictions and introduce Rank-One Model Editing (ROME) [51] to modify specific associations. The results show that ROME proves effective on zero-shot relation extraction tasks and a challenging counterfactual dataset, maintaining both specificity and generalization. These findings highlight the central role of mid-layer feed-forward modules in storing factual knowledge and demonstrate the feasibility of directly manipulating computational mechanisms for model editing. However, this approach has limitations. For instance, ROME and causal tracing primarily focus on factual associations, leaving other forms of learned knowledge, such as logical, spatial, and numerical reasoning, largely unexplored. Additionally, despite successful modification of factual associations, models may still generate plausible yet unsupported facts, which constrains their reliability as sources of accurate information.
On the other hand, Conmy et al. [52] automate a key step in mechanistic interpretability by identifying connections between abstract neural units that form a circuit. Specifically, they locate a subgraph of the model responsible for a target behavior. To achieve this, they introduce Automatic Circuit DisCovery (ACDC) [66], an algorithm that emulates researchersâ circuit-identification workflows by adapting Subnetwork Probing [75] and Head Importance Score for Pruning [76]. The study validates ACDC by reproducing prior results, such as successfully rediscovering all five component types in a circuit in GPT-2 Small [28] computing the Greater-Than operation. However, ACDC has certain limitations, as it struggles to fully automate activation patching. In particular, it may overlook certain classes of abstract units that are part of the circuit, such as the negative name mover heads from IOI [77]. Moreover, the algorithmâs behavior is highly sensitive to hyperparameter configurations and metric selection, which can result in inconsistent and non-robust performance across different settings.
5.3 XAI Methods for Encoder-decoder LMs
XAI methods for encoder-decoder LMs, such as T5 [2] and BART [33], seek to explain the internal decision-making processes of models with a dual architecture that combines both an encoder and a decoder component [27]. Unlike XAI techniques for encoder-only LMs, which primarily focus on interpreting bidirectional self-attention, and decoder-only LMs, which emphasize causal self-attention for autoregressive generation, explainability methods for encoder-decoder architectures emphasize cross-attention mechanisms that transfer information between the encoder and decoder, as well as the sequence-to-sequence objectives on which these models are pretrained [31]. To interpret these models, various explainable AI (XAI) techniques have been proposed, including feature attributionâbased methods [45], classifier-based probing [55], attention-based methods [20], and self-explanation methods [58], as discussed below.
5.3.1 Feature Attribution-based Methods
Feature attribution-based methods, as discussed in Sections 5.1.1 and 5.2.1, aim to quantify the contribution of individual input tokens or features to a modelâs predictions [40]. In contrast to their application in encoder-only and decoder-only LMs, these methods for encoder-decoder models consider both the bidirectional contextual representations generated by the encoder and the auto-regressive decoding process in the decoder [2]. Consequently, feature importance is influenced not only by the relationships among the input tokens but also by the cross-attention mechanisms that facilitate information transfer between the encoder and decoder [54]. To address these complexities, researchers have proposed various approaches tailored to sequence-to-sequence models while considering their dual architecture.
For example, to apply feature attribution-based methods for XAI in encoder-decoder models, Liu et al. [53] conduct an empirical study on various encoder-decoder LMs, including CodeT5 [78], CodeReviewer [79], and CodeT5+ [78], to evaluate the explainability of automated program generation approaches. This study employs gradient-based SHAP [40] to examine and interpret the modelsâ decision-making processes by highlighting tokens that significantly contribute to code transformation. The analysis demonstrates that these models often overlook critical tokens and exhibit limited robustness. Furthermore, the findings reveal that, while the models can effectively recognize code grammar and structural patterns, their performance remains sensitive to changes in input sequences. However, this study has certain limitations. For instance, the evaluation was conducted using different program generation models with their default architectures and hyperparameter settings as reported in the original papers. While this ensures consistency across models when analyzing their explainability, it may not reflect their optimal performance, as hyperparameter tuning could potentially yield better results.
Similarly, another XAI technique for encoder-decoder LMs, proposed by Enouen et al. [54], is TextGenSHAP, this approach extends Shapley values for text generation while maintaining computational efficiency for large-scale encoder-decoder models, such as T5-large [80] and T5-XXL [80]. Specifically, it focuses on explaining open-ended text generation using long prompts, particularly for abstractive question answering from retrieval-augmented documents. Moreover, TextGenSHAP achieves notable runtime improvements for token- and document-level explanations and further enhances retrieval-augmented generation by localizing important words and re-ranking retrieved passages. Furthermore, experiments on open-domain QA benchmarks demonstrate that TextGenSHAP significantly improves both retrieval recall and prediction accuracy. However, despite these improvements, the approach has certain limitations. In particular, generating explanations can still be computationally expensive for many real-world applications, making integration challenging. Furthermore, the study does not explore scenarios involving models distributed across multiple GPUs or TPUs, meaning it cannot guarantee speedups under hardware-specific optimizations.
5.3.2 Classifier-based Probing
Classifier-based probing, as discussed in Section 5.1.2, has proven effective for interpreting encoder-only language models by training shallow classifiers on frozen model representations to reveal model behavior and prediction logic [60]. This approach can be extended to encoder-decoder models, where probing is applied across the sequence-to-sequence architecture involving both the encoder and decoder. However, the added complexity of encoder-decoder modelsâparticularly the interplay introduced by components such as cross-attentionâposes significant challenges for interpretability. These interactions complicate the analysis of internal representations and obscure how predictions are formed [2].
To apply classifier-based probing to encoderâdecoder LMs, Koto et al. [55] introduced document-level discourse probing to interpret the ability of models such as T5 [2] and BART [33] to capture discourse-level relations. This approach helps identify whether, and at which layers, the model captures discourse structure. The study shows that BARTâs encoder layers perform best in capturing discourse information for probing tasks such as discourse connective [81], RST nuclearity [82] and RST relation [82] while the decoder performs comparatively poorly. This explains why decoder layers are less effective for language understanding than encoder layers and instead focus more on sequence generation. Similar trends were observed in the T5 encoderâdecoder model, which is also trained as a denoising autoencoder. Moreover, this study demonstrates that discourse knowledge is unevenly distributed across model layers, influenced by their pre-training objective and model architecture.
On the other hand, Li et al. [56] utilize classifier-based probing to investigate whether the effectiveness of encoder-decoder LMs such as T5 [2] and BART [33] arises solely from capturing surface-level word co-occurrence patterns or whether these models also represent and reason about the described world. They identify contextual word representations that function as models of entities and situations as they evolve within a discourse. These representations exhibit similarities to dynamic semantic models, as they enable a linear readout of an entityâs properties and relations and can be manipulated with predictable effects on language generation. This study suggests that encoder-decoder models rely, at least in part, on dynamic representations of meaning and implicit simulation of entity states, even when trained solely on textual data. However, it has limitations. For instance, the LM outputs and implicit state representations are far from perfect; even in relatively simple tasks, the best-case scenario recovers complete information states correctly only about half of the time. This indicates that the semantic representations lack sufficient expressive power and reliability, falling short of what is required for a human-like explanation.
5.3.3 Attention-based Methods
Attention-based methods, as discussed in Section 5.1.4, aim to interpret model behavior by analyzing the distribution of attention weights across tokens [2]. While attention mechanisms have been widely used for interpretability in encoder-only LMs, which rely on bidirectional self-attention, encoder-decoder architectures introduce additional complexity due to their dual structure comprising encoder and decoder components linked through cross-attention [31]. This architectural design makes it challenging to attribute model decisions and to understand how information flows across different layers.
To interpret the inner workings of encoder-decoder LMs using attention-head methods, Juraska et al. [20] propose a semantic attention-guided decoding method (SEAGUIDE) which visualizes the attention weight distribution across individual cross-attention layers in the decoder for different inputs, revealing multiple universal patterns that can be used to determine whether input slots are mentioned or omitted in the generated output. Specifically, the first pattern appears in the lowest attention layers, whose role is to track the corresponding input token associated with the token generated by the decoder in the preceding step. This pattern, called the verbatim slot mention pattern, can be captured by maximizing the weight across all attention heads in the first layer of the decoder. On the other hand, paraphrased slot mention patterns are captured in the higher layers when a corresponding token is about to be generated next. In these higher layers, the cross-attention weights gradually shift toward input tokens corresponding to information most likely to appear next in the output, capturing increasingly abstract concepts. Furthermore, the unrealized slot mention pattern is captured by averaging the attention weights over all layers at the final decoding step, which helps to reduce any undesired side effects of identifying paraphrased slot mention patterns. However, SEAGUIDEâs ability to detect slot-related errors is limited, as it focuses primarily on missing and incorrect slot mentions. It also struggles to reliably identify duplicate slots because the decoder inherently attends to certain input tokens across multiple non-consecutive steps.
Another attention-based method for explaining encoder-decoder models is proposed by Cao et al. [21], which introduces an inference-time attention head masking mechanism to interpret encoder-decoder attentions in abstractive summarization. To explain the content selection behavior of individual heads, the study investigates leveraging encoder-decoder attentions to guide content selection and enhance summary informativeness. Specifically, it applies attention head masking based on oracle content selection labels, constructed by aligning the reference summary with the source article and iteratively identifying the longest common subsequences between them. The findings show that applying oracle masking to the top attention layers yields the greatest improvement in summary informativeness. The study also explores whether masking multiple heads enhances content selection and whether these heads act synergistically. By progressively masking an increasing number of heads at each layer, it finds that the largest performance gain occurs when masking 15 out of 16 heads in the top layer, indicating strong collaboration among multiple heads for content selection. Furthermore, the study analyzes the types of words attention heads focus on: for each generated token during decoding, the input token with the highest attention weight (the attendee) is categorized into groups such as SALIENT, CONTENT, and FIRST/LAST tokens. The results reveal that higher layers tend to focus on tokens directly copied into the output, whereas lower layers attend more to salient words that are excluded from the current generation. However, this approach has limitations. For tasks that demand a comprehensive understanding of the source such as XSum [83] the method may falter. Specifically masking âunimportantâ tokens can inadvertently strip away essential contextual information [84], leading to unstable performance and diminished effectiveness.
5.3.4 Self-explanation Methods
Self-explanation-based methods aim to make encoder-decoder LMs more interpretable by generating human-readable justifications alongside task predictions. Unlike attention-based approaches that rely on interpreting internal attention weights to infer model reasoning [20, 21], these methods produce explanations as part of the output sequence, leveraging the generative capacity of the decoder [57]. This aligns with encoder-decoder architectures, where explanation and prediction can be treated as a unified text generation task [85]. It also enhances transparency by directly linking the modelâs decision-making process with the generated explanations, enabling users to better understand and evaluate the rationale behind the predictions [57, 58].
A popular approach to implement self-explanations for encoder-decoder models is introduced by Narang et al. [57] through WT5, which leverages the text-to-text framework to train models to generate natural language explanations alongside their predictions. This method does not require any modifications to the loss function, training objectives, or decoding procedures; instead, the model is simply trained to produce an explanation after generating the predicted output. The experimental results of this technique demonstrate that it achieves state-of-the-art performance on explainability benchmarks while also enabling learning from a limited number of annotated explanations and facilitating the transfer of rationalization capabilities across different datasets. However, a key limitation of this approach is that it typically requires large amounts of annotated explanations during training, which can be costly and time-consuming to obtain. For many real-world applications, this reliance on extensive data becomes a significant bottleneck, limiting the scalability and practical deployment of such methods.
Extending the paradigm of self-explanation techniques for encoder-decoder LMs, Yordanov et al. [58] propose three few-shot transfer learning approaches for generating natural language explanations (NLEs) and incorporate an additional method adapted from Erliksson et al. [86] to enhance computational efficiency. These four approaches integrate multi-task learning with fine-tuning to enable knowledge transfer from a parent task, which has abundant training labels, to a child task that has limited NLE annotations but sufficient labels. Specifically, the framework transfers explanatory capabilities from a large-scale natural language inference dataset (e-SNLI) [87] to two downstream child tasks: (1) complex pronoun resolution cases, for which the authors develop the small-e-WinoGrande dataset of NLEs based on WinoGrande [88], and (2) commonsense validation (ComVE) [89]. Moreover, the results highlight the potential of few-shot out-of-domain transfer learning for NLEs and provide insights into the effectiveness of different learning strategies in this context. However, this approach has certain limitations. Although the proposed training methods can be applied to languages other than English, they require the availability of a large parent dataset containing NLE annotations. Additionally, achieving effective performance may also depend on the existence of a high-quality pre-trained generative language model for the target language.
6 Evaluation of Explanations
After discussing various XAI methods within a systematic taxonomy based on the transformer architectures of LMs, it is essential to evaluate the quality and reliability of the explanations generated by these approaches. These explanations are typically assessed not only for their understandability to humans but also for how accurately they reflect the modelâs underlying reasoning process [37]. This dual requirement has led to two complementary evaluation dimensions: plausibility and faithfulness [22]. Specifically, plausibility measures the extent to which explanations align with human intuition and appear reasonable, whereas faithfulness examines whether explanations reliably capture the modelâs true decision-making process [36]. Together, these dimensions provide a structured framework for evaluating and comparing interpretability methods across encoder-only, decoder-only, and encoder-decoder architectures.
6.1 Plausibility
Plausibility refers to the extent to which an explanation is perceived as reasonable or convincing by humans [90]. To evaluate explanations using plausibility, researchers have proposed several methodologies:
- DeYoung et al. [38] introduce the Evaluating Rationales and Simple English Reasoning (ERASER), which measures how closely model-generated explanations align with human-annotated rationales across multiple datasets. This technique incorporates two types of metrics depending on whether models perform discrete or soft selection.
In the discrete setting, it employs the Intersection-over-Union (IoU) metric [91], which permits credit assignment for partial matches between predicted rationale span and human-annotated rationale span. Here, the predicted rationale span refers to a contiguous segment of input text highlighted by the model as its explanation, while the human-annotated rationale span denotes the corresponding gold evidence marked by annotators. Formally, IoU is defined as:
$$
\mathrm{IoU}=\frac{\lvert B_{p}\cap B_{gt}\rvert}{\lvert B_{p}\cup B_{gt}\rvert} \tag{1}
$$
where $B_{p}$ denotes the predicted rationale span and $B_{gt}$ denotes the human-annotated ground-truth rationale span.
A prediction is considered a match when the overlap between the predicted rationale and a ground-truth rationale exceeds $50\%$ , i.e., $\mathrm{IoU}>0.5$ . This threshold ensures that only predicted spans sufficiently aligned with human-annotated rationales are counted as correct. These matched predictions are then used to compute alignment metrics such as F1-score, which measure how well the modelâs explanations agree with human rationales.
In the soft selection setting, ERASER evaluates token rankings by rewarding models for assigning higher importance scores to tokens annotated by humans. Specifically, it calculates the Area Under the PrecisionâRecall Curve (AUPRC) [38], obtained by sweeping a threshold over token-level scores:
$$
\mathrm{AUPRC}=\sum_{n=1}^{N}\left(R_{n}-R_{n-1}\right)P_{n}, \tag{2}
$$
where $P_{n}$ and $R_{n}$ denote precision and recall at the $n$ -th threshold, respectively.
Here, AUPRC quantifies how well a model ranks human-annotated rationale tokens higher than irrelevant tokens. A higher AUPRC indicates that the model consistently assigns greater importance to human-labeled rationales, thus exhibiting better plausibility.
- Ding and Koehn [92] propose a test to evaluate the plausibility of explanations by leveraging lexical agreement annotations as a form of ground truth. They divide the nouns in the prefix into two distinct sets: the cue set $C$ , which contains nouns sharing the same morphological number as the verb in the sentence, and the attractor set $A$ , which contains nouns with a different morphological number from the verb in the sentence. Formally, let $\psi(w)$ denote the saliency score assigned to a word $w$ by the explanation method. Then, based on the prediction $y$ made by the model $M$ , the test is conducted under one of the following two scenarios:
Expected: When $y$ corresponds to the verb with the correct number, the interpretation passes the test if
$$
\max_{w\in C}\psi(w)>\max_{w\in A}\psi(w) \tag{3}
$$
In this case, a plausible explanation should assign higher importance to cue words than to attractors, indicating that the model relied on the correct grammatical cues when making the prediction.
Alternative: When $y$ corresponds to the verb with the incorrect number, the interpretation passes the test if
$$
\max_{w\in C}\psi(w)<\max_{w\in A}\psi(w) \tag{4}
$$
Here, the explanation is considered plausible if the attractor words receive higher importance than the cue words, showing that the model was misled by the wrong grammatical cues.
- Shen et al. [93] evaluate plausibility based on how well the rationales provided by the model agree with human-annotated ones. To quantify this, the approach adopts the token-level F1-score, computed from the overlap between predicted and annotated rationale tokens. Since an instance may contain multiple golden rationale sets, the set achieving the highest F1-score with the predicted rationale is selected as the ground truth for the current prediction:
$$
\text{Token-F1}=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{2\times P_{i}\times R_{i}}{P_{i}+R_{i}}\right),\quad P_{i}=\dfrac{|S^{p}_{i}\cap S^{g}_{i}|}{|S^{p}_{i}|},\quad R_{i}=\dfrac{|S^{p}_{i}\cap S^{g}_{i}|}{|S^{g}_{i}|} \tag{5}
$$
where $P_{i}$ is the precision, which measures the proportion of predicted rationale tokens that overlap with the human rationale, $R_{i}$ is the recall, which measures the proportion of human rationale tokens that overlap with the predicted rationale, $N$ denotes the total number of test instances, $S^{p}_{i}$ and $S^{g}_{i}$ represent the rationale sets of the $i$ -th instance provided by the model and the human, respectively.
A higher Token-F1 indicates better alignment between model-predicted and human-annotated rationales, reflecting greater plausibility, while a lower score suggests less interpretable and less human-aligned explanations.
The techniques discussed above help evaluate the plausibility of explanations produced by various XAI methods across different categories. However, several studies emphasize that evaluating the plausibility of explanations is fundamentally different from assessing their correctness, and the two criteria should be treated separately [37]. For example, Jacovi et al. [90] argue that explanations perceived as plausible do not necessarily reflect the modelâs true reasoning process. Similarly, Petsiuk et al. [94] suggest that excluding humans from the evaluation loop can lead to more objective assessments by focusing on the classifierâs decision-making process rather than subjective judgments. Such approaches reduce potential human bias while saving time and computational resources. Furthermore, Gilpin et al. [95] highlight that explanations perceived as unreasonable may indicate either limitations in the modelâs ability to process information in a reasonable way or shortcomings in the explanation generation method itself.
6.2 Faithfulness
While plausibility-based evaluations focus on whether explanations appear reasonable to humans, such rationales may not necessarily correspond to the modelâs true decision-making process [90]. In contrast, faithfulness refers to the extent to which an explanation accurately captures the internal reasoning of the model, ensuring that the identified rationale is causally linked to and meaningfully contributes to the modelâs prediction [96]. To assess the faithfulness of explanations, researchers have introduced various evaluation methodologies:
- DeYoung et al. [38] introduce two widely adopted metrics for evaluating faithfulness: comprehensiveness and sufficiency. Firstly, comprehensiveness measures whether the selected rationales include all features necessary for the modelâs prediction. The comprehensiveness score is defined as:
$$
\text{Comprehensiveness}=m(x_{i})_{j}-m(\tilde{x}_{i})_{j} \tag{6}
$$
where $x_{i}$ denotes the input instance, $\tilde{x}_{i}$ denotes the contrast example obtained by removing predicted rationale $r_{i}$ from $x_{i}$ , and $m(·)_{j}$ denotes the modelâs predicted probability for the predicted class $j$ in a classification setting.
A higher score indicates greater faithfulness, as it implies that removing the rationales significantly decreases the modelâs confidence. Conversely, a low score suggests that the rationales contributed little to the decision, and a negative score indicates an even less faithful scenario where the model becomes more confident after removing the rationales.
Secondly, sufficiency, in contrast, measures whether the extracted rationales alone are adequate for the model to reproduce its original decision. Formally, it is defined as:
$$
\text{Sufficiency}=m(x_{i})_{j}-m(r_{i})_{j} \tag{7}
$$
where $x_{i}$ denotes the input instance, $r_{i}$ denotes the predicted rationale, and $m(·)_{j}$ denotes the modelâs predicted probability for the predicted class $j$ in a classification setting.
Lower sufficiency scores imply higher faithfulness, as they indicate that $r_{i}$ alone contains enough information to make the same prediction with minimal confidence loss. In contrast, higher scores suggest that the rationales are insufficient and the model relies on additional context from the rest of the input.
- Ding and Koehn [92] propose consistency tests to evaluate the faithfulness of explanations, focusing on two aspects: model consistency, which measures the stability of feature importance scores under changes in model configuration, and input consistency, which measures the stability of scores under perturbations to the input.
To evaluate model consistency, the similarity between feature importance scores of the original model and a smaller distilled model trained to mimic it is computed. Although the original model and the distilled model differ in configuration, the distillation process encourages the smaller model to preserve the decision behavior of the original. This setup provides a meaningful test of whether explanations truly capture the modelâs internal reasoning. Formally, model consistency is defined as:
$$
\text{ModelConsistency}(M,M^{\prime},F)=\rho\left(\Psi_{M}(F),\Psi_{M^{\prime}}(F)\right), \tag{8}
$$
where $M$ denotes the original model, $M^{\prime}$ is the smaller distilled model trained to mimic $M$ , $\Psi_{M}(F)$ and $\Psi_{M^{\prime}}(F)$ represent the feature importance scores, $F$ denotes the input features, and $\rho(·,·)$ is the Pearson correlation coefficient.
Here, higher model consistency indicates that the saliency method produces stable and faithful explanations across models with similar decision behavior, whereas a low model consistency suggests sensitivity to configuration changes, undermining faithfulness.
To evaluate input consistency, tokens in the input are substituted while preserving the underlying decision, and the similarity between feature importance scores of the original and perturbed inputs is measured. Formally, input consistency is defined as:
$$
\text{Input Consistency}(F,F^{\prime})=\rho\left(\Psi(F),\Psi(F^{\prime})\right), \tag{9}
$$
where $F$ and $F^{\prime}$ denote the original and perturbed input features, respectively; $\Psi(F)$ and $\Psi(F^{\prime})$ denote their corresponding feature importance scores; and $\rho(·,·)$ represents the Pearson correlation coefficient.
Here, higher input consistency indicates that the feature importance remains stable under controlled changes in input, thus signifying greater faithfulness of the interpretation. Conversely, a lower input consistency suggests that the attribution is sensitive to irrelevant variations, indicating reduced faithfulness.
- Shen et al. [93] evaluate the faithfulness of language models by measuring the consistency of rationales under input perturbations by adopting Mean Average Precision (MAP) [97]. This metric quantifies the order consistency between two ranked token lists: the rationale extracted from the original input and that from the perturbed input. Formally, MAP is defined as:
$$
\text{MAP}=\frac{1}{|X^{p}|}\sum_{i=1}^{|X^{p}|}\frac{\sum_{j=1}^{i}G(x_{j}^{p},X_{1:i}^{o})}{i} \tag{10}
$$
where $X^{o}$ and $X^{p}$ denote the ranked token lists obtained from the original and perturbed inputs, respectively, and $|X^{p}|$ is the number of tokens in $X^{p}$ . The subset $X^{o}_{1:i}$ contains the top- $i$ tokens from $X^{o}$ . The indicator function $G(x,Y)$ returns $1$ if $xâ Y$ and $0$ otherwise.
A higher MAP score indicates that the ranking of important tokens remains consistent under perturbations, demonstrating stronger faithfulness of the rationale. In contrast, a lower MAP score reflects weaker faithfulness, suggesting that the rationale does not reliably capture the modelâs reasoning.
- Chrysostomou et al. [98] evaluate faithfulness by measuring the impact of removing the most relevant tokens identified by explanations. Specifically, they introduce two metrics: Decision Flip â Most Informative Token and Decision Flip â Fraction of Tokens.
Firstly, Decision Flip â Most Informative Token measures the percentage of predictions that change when the single most important token is removed. Formally, it is defined as:
$$
\text{DF}_{\text{MIT}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\!\left[m(x_{i})\neq m(x_{i}\setminus t_{i}^{*})\right] \tag{11}
$$
where $N$ is the total number of test instances, $m(·)$ denotes the model prediction, $x_{i}$ is the input instance, $t_{i}^{*}$ is the most important token according to the explanation method, and $\mathbf{1}[·]$ is the indicator function.
Here, a higher score indicates stronger faithfulness, since removing the most important token is sufficient to change the modelâs prediction. Conversely, a lower score suggests that the top-ranked token was not truly decisive for the modelâs decision.
Secondly, Decision Flip â Fraction of Tokens computes the average proportion of tokens that must be removed to induce a change in the modelâs prediction. Formally, it is defined as:
$$
\text{DF}_{\text{Frac}}=\frac{1}{N}\sum_{i=1}^{N}\frac{|R_{i}|}{|x_{i}|} \tag{12}
$$
where $R_{i}$ is the minimal set of top-ranked tokens that need to be removed from $x_{i}$ to flip the prediction, $|x_{i}|$ is the sequence length, and $N$ is the total number of test instances.
Here, a lower score indicates stronger faithfulness, as only a small proportion of top-ranked tokens need to be removed to alter the prediction. Conversely, a higher score means that many tokens must be removed, suggesting the explanation is less focused on the truly influential evidence.
The approaches discussed above are employed to assess the faithfulness of explanations across different explainability methods. Moreover, faithfulness thus complements plausibility by ensuring that explanations are not only persuasive to humans but also accurately reflect the modelâs internal reasoning. Achieving a balance between these perspectives is crucial for developing explanations that are both trustworthy and interpretable.
7 Challenges and Future Directions
Despite significant progress in developing XAI techniques for LMs, several open challenges remain. This section discusses these challenges and outlines potential avenues for future research, focusing on key aspects such as the scalability of explanations, the need for methods that operate without ground-truth rationales, the trade-offs between performance and interpretability, the explainability of multimodal systems, and the broader ethical and societal implications. The following subsections examine each of these challenges and highlight promising directions for addressing them.
7.1 Scalability of Explanations
As language models continue to grow in size and complexity, generating faithful and interpretable explanations becomes increasingly challenging. Many existing XAI techniques, such as attribution-based methods [40], attention visualizations [20], and mechanistic interpretability [66], require extensive computations or repeated model evaluations, leading to substantial resource demands. These challenges are further amplified in large-scale models with hundreds of billions of parameters, where token-level or layer-wise explanations often become impractical for real-time or resource-constrained applications. Future research should explore strategies that improve the efficiency of explanation techniques while maintaining their interpretability, enabling scalable and practical solutions for modern LMs.
7.2 Explanation without Ground Truths
Ground-truth explanations for language models are often inaccessible, making it challenging to evaluate the quality and reliability of generated explanations [22]. While existing benchmarks, such as ERASER [38], provide human-annotated rationales for plausibility and faithfulness assessment, their coverage is limited in terms of tasks and domains. Moreover, for many open-ended tasks such as summarization, dialogue generation, or creative writing, multiple valid explanations may exist, and identifying a single correct rationale becomes inherently ambiguous [37]. This lack of standardized ground-truth references highlights the need for alternative evaluation strategies, such as those focusing on internal model consistency, causal alignment, and cross-metric validation, rather than relying exclusively on annotated datasets.
7.3 Performance vs. Interpretability
Recent advances in XAI techniques such as chain-of-thought prompting, feature attribution methods, and self-explanation strategies have enhanced the interpretability of language models by providing greater transparency into their decision-making processes [53, 58, 50]. However, these techniques often introduce additional computational overhead and, in some cases, lead to reductions in task performance [99]. Conversely, models optimized exclusively for predictive accuracy may yield limited or no meaningful explanations, constraining their applicability in high-stakes domains where accountability and transparency are critical. Addressing this problem requires research efforts aimed at developing approaches that balance predictive performance with interpretability, ensuring that explanations remain reliable without significantly compromising model effectiveness.
7.4 Explainibility of Multimodal models
Recent advances in multimodal models, which integrate information from diverse modalities such as text, images, audio, and video, have expanded the capabilities of language models but also introduced new challenges for explainability [100]. Unlike unimodal LMs, where explanations can focus solely on textual inputs and outputs, multimodal systems require understanding how information is combined, aligned, and propagated across different modalities. Existing XAI techniques, which are primarily designed for text-only models, often fail to capture these cross-modal interactions and dependencies [101]. Addressing this challenge requires XAI frameworks that jointly analyze contributions from multiple modalities, enabling interpretability while preserving the distinct roles of each modality in the decision-making process.
7.5 Ethical and Societal Concerns
As language models are increasingly deployed in high-stakes domains such as healthcare, finance, education, and law, their explanations carry significant ethical and societal implications [5, 6, 7]. Explanations that are incomplete, misleading, or biased can undermine user trust and lead to harmful outcomes. Moreover, the absence of standardized explainability practices complicates regulatory compliance and accountability, especially when decisions directly affect individuals or communities. Addressing this challenge requires developing explanation frameworks that emphasize fairness, transparency, and inclusivity while mitigating biases in both models and their generated explanations [22]. Furthermore, explainability methods should consider diverse user needs, ensuring that explanations remain accessible and meaningful across different contexts and populations.
8 Conclusion
In this survey, we provided a comprehensive review of XAI techniques for language models, systematically organizing existing methods according to their underlying transformer architectures: encoder-only, decoder-only, and encoder-decoder. By analyzing a wide range of approaches, we highlight how architectural differences shape the design and effectiveness of explainability methods by examining their strengths and limitations. We also discussed evaluation strategies through the complementary lenses of plausibility and faithfulness, emphasizing the need for explanations that are both interpretable to humans and faithful to model reasoning. Despite significant progress in XAI, several challenges remain, including scalability, balancing performance with interpretability, and addressing ethical concerns, presenting opportunities for future research.
References
- \bibcommenthead
- Devlin et al. [2018] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
- Raffel et al. [2020] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), 1â67 (2020)
- Achiam et al. [2023] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
- Touvron et al. [2023] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
- Lee et al. [2020] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36 (4), 1234â1240 (2020) https://doi.org/10.1093/bioinformatics/btz682
- Chen et al. [2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)
- Ng and Fung [2024] Ng, C., Fung, Y.: Educational personalized learning path planning with large language models. arXiv preprint arXiv:2407.11773 (2024)
- Lopes et al. [2020] Lopes, A., Nogueira, R., Lotufo, R., Pedrini, H.: Lite Training Strategies for Portuguese-English and English-Portuguese Translation (2020). https://arxiv.org/abs/2008.08769
- Schwartz et al. [2024] Schwartz, I.S., Link, K.E., Daneshjou, R., CortĂ©s-Penfield, N.: Black box warning: large language models and the future of infectious diseases consultation. Clinical infectious diseases 78 (4), 860â866 (2024)
- Li et al. [2023] Li, Y., Wang, S., Ding, H., Chen, H.: Large language models in finance: A survey. In: Proceedings of the Fourth ACM International Conference on AI in Finance. ICAIF â23, pp. 374â382. Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3604237.3626869 . https://doi.org/10.1145/3604237.3626869
- Shu et al. [2024] Shu, D., Zhao, H., Liu, X., Demeter, D., Du, M., Zhang, Y.: Lawllm: Law large language model for the us legal system. In: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 4882â4889 (2024)
- Jung et al. [2021] Jung, Y.-J., Han, S.-H., Choi, H.-J.: Explaining cnn and rnn using selective layer-wise relevance propagation. IEEE Access 9, 18670â18681 (2021)
- Papanastasopoulos et al. [2020] Papanastasopoulos, Z., Samala, R.K., Chan, H.-P., Hadjiiski, L., Paramagul, C., Helvie, M.A., Neal, C.H.: Explainable ai for medical imaging: deep-learning cnn ensemble for classification of estrogen receptor status from breast mri. In: Medical Imaging 2020: Computer-aided Diagnosis, vol. 11314, pp. 228â235 (2020). SPIE
- Freeborough [2022] Freeborough, W.: Investigating explainablity methods in recurrent neural network architectures for financial time-series data. Masterâs thesis, University of the Witwatersrand, Johannesburg (South Africa) (2022)
- Burkart and Huber [2021] Burkart, N., Huber, M.F.: A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research 70, 245â317 (2021)
- Sundararajan et al. [2017] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. 34th International Conference on Machine Learning, 1â11 (2017)
- Kokalj et al. [2021] Kokalj, E., Ć krlj, B., LavraÄ, N., Pollak, S., Robnik-Ć ikonja, M.: Bert meets shapley: Extending shap explanations to transformer-based classifiers. In: Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, pp. 16â21 (2021)
- Wei et al. [2022] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824â24837 (2022)
- Kojima et al. [2022] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems 35, 22199â22213 (2022)
- Juraska and Walker [2021] Juraska, J., Walker, M.: Attention is indeed all you need: Semantically attention-guided decoding for data-to-text nlg. In: Proceedings of the 14th International Conference on Natural Language Generation, pp. 416â431 (2021)
- Cao and Wang [2021] Cao, S., Wang, L.: Attention head masking for inference time content selection in abstractive summarization. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5008â5016 (2021)
- Zhao et al. [2024] Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., Du, M.: Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology 15 (2), 1â38 (2024)
- Luo and Specia [2024] Luo, H., Specia, L.: From Understanding to Utilization: A Survey on Explainability for Large Language Models (2024). https://arxiv.org/abs/2401.12874
- Mohammadi et al. [2025] Mohammadi, H., Bagheri, A., Giachanou, A., Oberski, D.L.: Explainability in Practice: A Survey of Explainable NLP Across Various Domains (2025). https://arxiv.org/abs/2502.00837
- Dwivedi et al. [2023] Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., Ranjan, R.: Explainable ai (xai): Core ideas, techniques, and solutions. ACM Comput. Surv. 55 (9) (2023) https://doi.org/10.1145/3561048
- Islam et al. [2021] Islam, S.R., Eberle, W., Ghafoor, S.K., Ahmed, M.: Explainable Artificial Intelligence Approaches: A Survey (2021). https://arxiv.org/abs/2101.09429
- Wang et al. [2025] Wang, Z., Chu, Z., Doan, T.V., Ni, S., Yang, M., Zhang, W.: History, development, and principles of large language models: an introductory survey. AI and Ethics 5 (3), 1955â1971 (2025)
- Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877â1901 (2020)
- Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ć., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
- Liu et al. [2019] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). https://arxiv.org/abs/1907.11692
- Minaee et al. [2025] Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., Gao, J.: Large Language Models: A Survey (2025). https://arxiv.org/abs/2402.06196
- Kalyan et al. [2022] Kalyan, K.S., Rajasekharan, A., Sangeetha, S.: Ammu: A survey of transformer-based biomedical pretrained language models. Journal of Biomedical Informatics 126, 103982 (2022) https://doi.org/10.1016/j.jbi.2021.103982
- Lewis et al. [2019] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019). https://arxiv.org/abs/1910.13461
- Gulowaty and WoĆșniak [2021] Gulowaty, B., WoĆșniak, M.: Extracting interpretable decision tree ensemble from random forest. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1â8 (2021). IEEE
- Dai et al. [2022] Dai, X., Keane, M.T., Shalloo, L., Ruelle, E., Byrne, R.M.: Counterfactual explanations for prediction and diagnosis in xai. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 215â226 (2022)
- Chen et al. [2023] Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., McKeown, K.: Do models explain themselves? counterfactual simulatability of natural language explanations. arXiv preprint arXiv:2307.08678 (2023)
- Nauta et al. [2023] Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., Keulen, M., Seifert, C.: From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Comput. Surv. 55 (13s) (2023) https://doi.org/10.1145/3583558
- DeYoung et al. [2019] DeYoung, J., Jain, S., Rajani, N.F., Lehman, E., Xiong, C., Socher, R., Wallace, B.C.: Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv:1911.03429 (2019)
- SzczepaĆski et al. [2021] SzczepaĆski, M., Pawlicki, M., Kozik, R., ChoraĆ, M.: New explainability method for bert-based model in fake news detection. Scientific Reports 11 (1), 23705 (2021) https://doi.org/10.1038/s41598-021-03100-6
- Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems (2017)
- Tenney et al. [2019] Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R.T., Kim, N., Van Durme, B., Bowman, S.R., Das, D., et al.: What do you learn from context? probing for sentence structure in contextualized word representations. In: 7th International Conference on Learning Representations, ICLR 2019 (2019)
- Wu et al. [2020] Wu, Z., Chen, Y., Kao, B., Liu, Q.: Perturbed masking: Parameter-free probing for analyzing and interpreting bert. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4166â4176 (2020)
- Vig [2019] Vig, J.: A multiscale visualization of attention in the transformer model. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37â42. Association for Computational Linguistics, Florence, Italy (2019)
- Hoover et al. [2020] Hoover, B., Strobelt, H., Gehrmann, S.: exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 187â196. Association for Computational Linguistics, Online (2020)
- Barkan et al. [2024] Barkan, O., Toib, Y., Elisha, Y., Weill, J., Koenigstein, N.: Llm explainability via attributive masking learning. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 9522â9537 (2024)
- Kariyappa et al. [2024] Kariyappa, S., LĂ©cuĂ©, F., Mishra, S., Pond, C., Magazzeni, D., Veloso, M.: Progressive inference: explaining decoder-only sequence classification models using intermediate predictions. In: Proceedings of the 41st International Conference on Machine Learning, pp. 23238â23255 (2024)
- Wei et al. [2023] Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al.: Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846 (2023)
- Li et al. [2023] Li, S., Nie, Y., Pang, B., Chang, K.-W., Yang, Y.: Analyzing in-context learning with saliency methods: A case study on contrastive demonstrations. arXiv preprint arXiv:2307.05052 (2023)
- Zhang et al. [2024] Zhang, D., Sun, Y., Wang, Z., Zhang, Y., Shen, Y., Zhu, K., Wang, W.Y.: Sea-cot: Self-entailment-alignment chain-of-thought prompting for explainable reasoning. In: Findings of the Association for Computational Linguistics: NAACL 2024 (2024). https://aclanthology.org/2024.findings-naacl.138.pdf
- Zhou et al. [2022] Zhou, D., SchâÌarli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al.: Least-to-most prompting enables complex reasoning in large language models. The Eleventh International Conference on Learning Representations (2022)
- Meng et al. [2022] Meng, K., Bau, D., Andonian, A., Belinkov, Y.: Locating and editing factual associations in gpt. In: Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 17359â17372 (2022)
- Conmy et al. [2023] Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., Garriga-Alonso, A.: Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems 36, 16318â16352 (2023)
- Liu et al. [2024] Liu, Y., Tantithamthavorn, C., Liu, Y., Li, L.: On the reliability and explainability of language models for program generation. ACM Trans. Softw. Eng. Methodol. 33 (5) (2024)
- Enouen et al. [2024] Enouen, J., Nakhost, H., Ebrahimi, S., Arik, S., Liu, Y., Pfister, T.: TextGenSHAP: Scalable post-hoc explanations in text generation with long documents. In: Findings of the Association for Computational Linguistics: ACL 2024, pp. 13984â14011. Association for Computational Linguistics, Bangkok, Thailand (2024)
- Koto et al. [2021] Koto, F., Lau, J.H., Baldwin, T.: Discourse probing of pretrained language models. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3849â3864. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.301 . https://aclanthology.org/2021.naacl-main.301/
- Li et al. [2021] Li, B.Z., Nye, M., Andreas, J.: Implicit representations of meaning in neural language models. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1813â1827. Association for Computational Linguistics, Online (2021)
- Narang et al. [2020] Narang, S., Raffel, C., Lee, K., Roberts, A., Fiedel, N., Malkan, K.: Wt5?! training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546 (2020)
- Yordanov et al. [2022] Yordanov, Y., Kocijan, V., Lukasiewicz, T., Camburu, O.-M.: Few-shot out-of-domain transfer learning of natural language explanations in a label-abundant setup. In: Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3486â3501 (2022)
- Wu and Ong [2021] Wu, Z., Ong, D.C.: On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification (2021). https://arxiv.org/abs/2101.00196
- Mohebbi et al. [2021] Mohebbi, H., Modarressi, A., Pilehvar, M.T.: Exploring the role of bert token representations to explain sentence probing results. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 792â806 (2021)
- Belinkov et al. [2017] Belinkov, Y., MĂ rquez, L., Sajjad, H., Durrani, N., Dalvi, F., Glass, J.: Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1â10. Asian Federation of Natural Language Processing, Taipei, Taiwan (2017). https://aclanthology.org/I17-1001/
- He et al. [2018] He, L., Lee, K., Levy, O., Zettlemoyer, L.: Jointly predicting predicates and arguments in neural semantic role labeling. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 364â369. Association for Computational Linguistics, Melbourne, Australia (2018)
- Lee et al. [2018] Lee, K., He, L., Zettlemoyer, L.: Higher-order coreference resolution with coarse-to-fine inference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 687â692. Association for Computational Linguistics, New Orleans, Louisiana (2018)
- Hewitt and Liang [2019] Hewitt, J., Liang, P.: Designing and Interpreting Probes with Control Tasks (2019). https://arxiv.org/abs/1909.03368
- Zhou and Srikumar [2021] Zhou, Y., Srikumar, V.: DirectProbe: Studying representations without classifiers. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5070â5083. Association for Computational Linguistics, Online (2021)
- Brunner et al. [2020] Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., Wattenhofer, R.: On Identifiability in Transformers (2020). https://arxiv.org/abs/1908.04211
- Chen et al. [2023] Chen, B., Zhang, Z., Langrené, N., Zhu, S.: Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735 (2023)
- [68] Liu, F.: Towards understanding in-context learning with contrastive demonstrations and saliency maps. In: NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning
- Chan et al. [2022] Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., Hill, F.: Data distributional properties drive emergent in-context learning in transformers. Advances in neural information processing systems 35, 18878â18891 (2022)
- Wang et al. [2022] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)
- Wei Jie et al. [2024] Wei Jie, Y., Satapathy, R., Goh, R., Cambria, E.: How interpretable are reasoning explanations from prompting large language models? In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2148â2164. Association for Computational Linguistics, Mexico City, Mexico (2024). https://doi.org/10.18653/v1/2024.findings-naacl.138 . https://aclanthology.org/2024.findings-naacl.138/
- Colelough and Regli [2025] Colelough, B.C., Regli, W.: Neuro-symbolic ai in 2024: A systematic review. arXiv preprint arXiv:2501.05435 (2025)
- Olah et al. [2020] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., Carter, S.: Zoom in: An introduction to circuits. Distill 5 (3), 00024â001 (2020)
- Nanda et al. [2023] Nanda, N., Chan, L., Lieberum, T., Smith, J., Steinhardt, J.: Progress measures for grokking via mechanistic interpretability. In: The Eleventh International Conference on Learning Representations (2023)
- Cao et al. [2021] Cao, S., Sanh, V., Rush, A.: Low-complexity probing via finding subnetworks. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (eds.) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 960â966. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.74 . https://aclanthology.org/2021.naacl-main.74/
- Michel et al. [2019] Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? Advances in neural information processing systems 32 (2019)
- Wang et al. [2023a] Wang, K.R., Variengien, A., Conmy, A., Shlegeris, B., Steinhardt, J.: Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In: The Eleventh International Conference on Learning Representations (2023). https://openreview.net/forum?id=NpsVSN6o4ul
- Wang et al. [2023b] Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., Hoi, S.: CodeT5+: Open code large language models for code understanding and generation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1069â1088. Association for Computational Linguistics, Singapore (2023)
- Li et al. [2022] Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., Sundaresan, N.: Automating Code Review Activities by Large-Scale Pre-training (2022)
- Chung et al. [2024] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), 1â53 (2024)
- Nie et al. [2019] Nie, A., Bennett, E., Goodman, N.: DisSent: Learning sentence representations from explicit discourse relations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4497â4510. Association for Computational Linguistics, Florence, Italy (2019)
- Mann and Thompson [1986] Mann, W.C., Thompson, S.A.: Assertions from discourse structure. In: Strategic Computing - Natural Language Workshop: Proceedings of a Workshop Held at Marina del Rey, California, May 1-2, 1986 (1986). https://aclanthology.org/H86-1024/
- Narayan et al. [2018] Narayan, S., Cohen, S.B., Lapata, M.: Donât give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In: Riloff, E., Chiang, D., Hockenmaier, J., Tsujii, J. (eds.) Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797â1807. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1206 . https://aclanthology.org/D18-1206/
- Ma et al. [2021] Ma, Y., Lan, Z., Zong, L., Huang, K.: Global-aware beam search for neural abstractive summarization. Advances in Neural Information Processing Systems 34, 16545â16557 (2021)
- Rajani et al. [2019] Rajani, N.F., McCann, B., Xiong, C., Socher, R.: Explain yourself! leveraging language models for commonsense reasoning. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4932â4942 (2019)
- Erliksson et al. [2021] Erliksson, K.F., Arpteg, A., Matskin, M., Payberah, A.H.: Cross-domain transfer of generative explanations using text-to-text models, pp. 76â89. Springer, Berlin, Heidelberg (2021). https://doi.org/10.1007/978-3-030-80599-9_8
- Camburu et al. [2018] Camburu, O.-M., RocktĂ€schel, T., Lukasiewicz, T., Blunsom, P.: e-snli: natural language inference with natural language explanations. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPSâ18, pp. 9560â9572. Curran Associates Inc., Red Hook, NY, USA (2018)
- Sakaguchi et al. [2020] Sakaguchi, K., Le Bras, R., Bhagavatula, C., Choi, Y.: Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), 8732â8740 (2020) https://doi.org/10.1609/aaai.v34i05.6399
- Wang et al. [2020] Wang, C., Liang, S., Jin, Y., Wang, Y., Zhu, X., Zhang, Y.: SemEval-2020 task 4: Commonsense validation and explanation. In: Herbelot, A., Zhu, X., Palmer, A., Schneider, N., May, J., Shutova, E. (eds.) Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 307â321. International Committee for Computational Linguistics, Barcelona (online) (2020). https://doi.org/10.18653/v1/2020.semeval-1.39 . https://aclanthology.org/2020.semeval-1.39/
- Jacovi and Goldberg [2020] Jacovi, A., Goldberg, Y.: Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198â4205. Association for Computational Linguistics, ??? (2020)
- Everingham et al. [2010] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88 (2), 303â338 (2010) https://doi.org/10.1007/s11263-009-0275-4
- Ding and Koehn [2021] Ding, S., Koehn, P.: Evaluating saliency methods for neural language models. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y. (eds.) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5034â5052. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.399 . https://aclanthology.org/2021.naacl-main.399/
- Shen et al. [2022] Shen, Y., Wang, L., Chen, Y., Xiao, X., Liu, J., Wu, H.: An interpretability evaluation benchmark for pre-trained language models. arXiv preprint arXiv:2207.13948 (2022)
- Petsiuk et al. [2018] Petsiuk, V., Das, A., Saenko, K.: Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018)
- Gilpin et al. [2018] Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: An overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80â89 (2018). https://doi.org/10.1109/DSAA.2018.00018
- Mathew et al. [2021] Mathew, B., Saha, P., Yimam, S.M., Biemann, C., Goyal, P., Mukherjee, A.: Hatexplain: A benchmark dataset for explainable hate speech detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14867â14875 (2021)
- Wang et al. [2021] Wang, L., Liu, H., Peng, S., Tang, H., Xiao, X., Chen, Y., Wu, H., Wang, H.: Dutrust: A sentiment analysis dataset for trustworthiness evaluation. arXiv preprint arXiv:2108.13140 (2021)
- Chrysostomou and Aletras [2021] Chrysostomou, G., Aletras, N.: Improving the faithfulness of attention-based explanations with task-specific information for text classification. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 477â488. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.acl-long.40 . https://aclanthology.org/2021.acl-long.40/
- Crook et al. [2023] Crook, B., SchlĂŒter, M., Speith, T.: Revisiting the performance-explainability trade-off in explainable artificial intelligence (xai). In: 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), pp. 316â324 (2023). IEEE
- Rodis et al. [2024] Rodis, N., Sardianos, C., Radoglou-Grammatikis, P., Sarigiannidis, P., Varlamis, I., Papadopoulos, G.T.: Multimodal explainable artificial intelligence: A comprehensive review of methodological advances and future research directions. IEEE Access 12, 159794â159820 (2024)
- Mersha et al. [2024] Mersha, M., Lam, K., Wood, J., AlShami, A.K., Kalita, J.: Explainable artificial intelligence: A survey of needs, techniques, applications, and future direction. Neurocomputing 599, 128111 (2024) https://doi.org/10.1016/j.neucom.2024.128111
- Wang et al. [2023] Wang, Z., Saxena, N., Yu, T., Karki, S., Zetty, T., Haque, I., Zhou, S., Kc, D., Stockwell, I., Bifet, A., et al.: Preventing discriminatory decision-making in evolving data streams. In: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT) (2023)
- Zhang et al. [2023] Zhang, W., Wang, Z., Kim, J., Cheng, C., Oommen, T., Ravikumar, P., Weiss, J.: Individual fairness under uncertainty. In: 26th European Conference on Artificial Intelligence, pp. 3042â3049 (2023)
- Wang et al. [2023a] Wang, Z., Wallace, C., Bifet, A., Yao, X., Zhang, W.: Fg 2 an: Fairness-aware graph generative adversarial networks. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 259â275 (2023). Springer Nature Switzerland
- Wang et al. [2023b] Wang, Z., Narasimhan, G., Yao, X., Zhang, W.: Mitigating multisource biases in graph neural networks via real counterfactual samples. In: 2023 IEEE International Conference on Data Mining (ICDM), pp. 638â647 (2023). IEEE
- Chinta et al. [2023] Chinta, S.V., Fernandes, K., Cheng, N., Fernandez, J., Yazdani, S., Yin, Z., Wang, Z., Wang, X., Xu, W., Liu, J., et al.: Optimization and improvement of fake news detection using voting technique for societal benefit. In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1565â1574 (2023). IEEE
- Wang et al. [2024] Wang, Z., Chu, Z., Doan, T.V., Ni, S., Yang, M., Zhang, W.: History, development, and principles of large language models-an introductory survey. AI and Ethics, 2024 (2024)
- Chu et al. [2024] Chu, Z., Wang, Z., Zhang, W.: Fairness in large language models: A taxonomic survey. ACM SIGKDD explorations newsletter 26 (1), 34â48 (2024)
- Yin et al. [2024] Yin, Z., Wang, Z., Zhang, W.: Improving fairness in machine learning software via counterfactual fairness thinking. In: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, pp. 420â421 (2024)
- Wang et al. [2023] Wang, Z., Zhou, Y., Haque, I., Lo, D., Zhang, W.: Towards fair machine learning software: Understanding and addressing model bias through counterfactual thinking. arXiv preprint arXiv:2302.08018 (2023)
- Wang et al. [2024] Wang, Z., Qiu, M., Chen, M., Salem, M.B., Yao, X., Zhang, W.: Toward fair graph neural networks via real counterfactual samples. Knowledge and Information Systems, 1â25 (2024)
- Chinta et al. [2024] Chinta, S.V., Wang, Z., Yin, Z., Hoang, N., Gonzalez, M., Quy, T.L., Zhang, W.: Fairaied: Navigating fairness, bias, and ethics in educational ai applications. arXiv preprint arXiv:2407.18745 (2024)
- Wang et al. [2024a] Wang, Z., Dzuong, J., Yuan, X., Chen, Z., Wu, Y., Yao, X., Zhang, W.: Individual fairness with group awareness under uncertainty. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 89â106 (2024). Springer Nature Switzerland
- Wang et al. [2024b] Wang, Z., Palikhe, A., Yin, Z., Zhang, W.: Fairness definitions in language models explained. arXiv preprint arXiv:2407.18454 (2024)
- Wang et al. [2024c] Wang, Z., Chu, Z., Blanco, R., Chen, Z., Chen, S.-C., Zhang, W.: Advancing graph counterfactual fairness through fair representation learning. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2024). Springer Nature Switzerland
- Wang and Zhang [2024] Wang, Z., Zhang, W.: Group fairness with individual and censorship constraints. In: 27th European Conference on Artificial Intelligence (2024)
- Wang et al. [2024] Wang, Z., Dzuong, J., Yuan, X., Chen, Z., Wu, Y., Yao, X., Zhang, W.: Individual fairness with group awareness under uncertainty. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2024). Springer Nature Switzerland
- Yin et al. [2024] Yin, Z., Agarwal, S., Kashif, A., Gonzalez, M., Wang, Z., Liu, S., Liu, Z., Wu, Y., Stockwell, I., Xu, W., et al.: Accessible health screening using body fat estimation by image segmentation. In: 2024 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 405â414 (2024)
- Wang et al. [2025a] Wang, Z., Yin, Z., Zhang, Y., Yang, L., Zhang, T., Pissinou, N., Cai, Y., Hu, S., Li, Y., Zhao, L., et al.: Fg-smote: Towards fair node classification with graph neural network. ACM SIGKDD Explorations Newsletter 26 (2), 99â108 (2025)
- Wang et al. [2025b] Wang, Z., Yin, Z., Liu, F., Liu, Z., Lisetti, C., Yu, R., Wang, S., Liu, J., Ganapati, S., Zhou, S., et al.: Graph fairness via authentic counterfactuals: Tackling structural and causal challenges. ACM SIGKDD Explorations Newsletter 26 (2), 89â98 (2025)
- Wang et al. [2025c] Wang, Z., Chu, Z., Viet Doan, T., Wang, S., Wu, Y., Palade, V., Zhang, W.: Fair graph u-net: A fair graph learning framework integrating group and individual awareness. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 28485â28493 (2025)
- Wang et al. [2025d] Wang, Z., Yin, Z., Zhang, X., Zhang, Y., He, X., Wang, S., Song, H., Yang, L., Zhang, W.: Towards fair graph-based machine learning software: unveiling and mitigating graph model bias. AI and Ethics, 1â18 (2025)
- Yin et al. [2025] Yin, Z., Wang, Z., Xu, W., Zhuang, J., Mozumder, P., Smith, A., Zhang, W.: Digital forensics in the age of large language models. arXiv preprint arXiv:2504.02963 (2025)
- Chinta et al. [2025] Chinta, S.V., Wang, Z., Palikhe, A., Zhang, X., Kashif, A., Smith, M.A., Liu, J., Zhang, W.: Ai-driven healthcare: Fairness in ai healthcare: A survey. PLOS Digital Health 4 (5), 0000864 (2025)
- Wang and Zhang [2025] Wang, Z., Zhang, W.: Fdgen: A fairness-aware graph generation model. In: Proceedings of the 42nd International Conference on Machine Learning (2025). PMLR
- Wang et al. [2025a] Wang, Z., Yin, Z., Yang, L., Zhuang, J., Yu, R., Kong, Q., Zhang, W.: Fairglite: Fair graph representation learning with limited demographics. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2025). Springer Nature Switzerland
- Wang et al. [2025b] Wang, Z., Yin, Z., Yap, R., Zhang, X., Hu, S., Zhang, W.: Redefining fairness: A multi-dimensional perspective and integrated evaluation framework. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2025). Springer Nature Switzerland
- Zhang and Ntoutsi [2019] Zhang, W., Ntoutsi, E.: Faht: an adaptive fairness-aware decision tree classifier. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 1480â1486 (2019)
- Zhang [2024] Zhang, W.: Ai fairness in practice: Paradigm, challenges, and prospects. Ai Magazine 45 (3), 386â395 (2024)
- Zhang and Weiss [2022] Zhang, W., Weiss, J.C.: Longitudinal fairness with censorship. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 12235â12243 (2022)
- Zhang et al. [2023] Zhang, W., Hernandez-Boussard, T., Weiss, J.: Censored fairness through awareness. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 14611â14619 (2023)
- Zhang et al. [2025] Zhang, W., Zhou, S., Walsh, T., Weiss, J.C.: Fairness amidst nonâiid graph data: A literature review. AI Magazine 46 (1), 12212 (2025)
- Zhang et al. [2022] Zhang, W., Pan, S., Zhou, S., Walsh, T., Weiss, J.C.: Fairness amidst non-iid graph data: Current achievements and future directions. arXiv preprint arXiv:2202.07170 (2022)
- Wang et al. [2024] Wang, Z., Qiu, M., Chen, M., Salem, M.B., Yao, X., Zhang, W.: Towards fair graph neural networks via real counterfactual samples (2024)
- Wang et al. [2025] Wang, Z., Yin, Z., Yap, R., Zhang, W.: Ai fairness beyond complete demographics: Current achievements and future directions. In: 28th European Conference on Artificial Intelligence (2025)
- Chu et al. [2024] Chu, Z., Ni, S., Wang, Z., Feng, X., Li, C., Hu, X., Xu, R., Yang, M., Zhang, W.: History, development, and principles of large language models-an introductory survey. arXiv preprint arXiv:2402.06853 (2024)
- Saxena et al. [2023] Saxena, N.A., Zhang, W., Shahabi, C.: Missed opportunities in fair ai. In: Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pp. 961â964 (2023). SIAM
- Zhang et al. [2019] Zhang, W., Tang, X., Wang, J.: On fairness-aware learning for non-discriminative decision-making. In: International Conference on Data Mining Workshops (ICDMW), pp. 1072â1079 (2019)
- Zhang et al. [2020] Zhang, W., et al.: Flexible and adaptive fairness-aware learning in non-stationary data streams. In: IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 399â406 (2020)
- Zhang and Zhao [2020] Zhang, W., Zhao, L.: Online decision trees with fairness. arXiv preprint arXiv:2010.08146 (2020)
- Zhang [2020] Zhang, W.: Learning fairness and graph deep generation in dynamic environments (2020)
- Zhang et al. [2021] Zhang, W., Bifet, A., Zhang, X., Weiss, J.C., Nejdl, W.: Farf: A fair and adaptive random forests classifier. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 245â256 (2021). Springer
- Zhang and Weiss [2021] Zhang, W., Weiss, J.: Fair decision-making under uncertainty. In: 2021 IEEE International Conference on Data Mining (ICDM) (2021). IEEE
- Zhang and Weiss [2023] Zhang, W., Weiss, J.C.: Fairness with censorship and group constraints. Knowledge and Information Systems, 1â24 (2023)
- Zhang et al. [2016] Zhang, W., Tang, J., Wang, N.: Using the machine learning approach to predict patient survival from high-dimensional survival data. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2016)
- Zhang and Wang [2018] Zhang, W., Wang, J.: Content-bootstrapped collaborative filtering for medical article recommendations. In: IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2018)
- Zhang et al. [2021] Zhang, M., Zhang, F., Zhang, J., Chaddad, A., Guo, F., Zhang, W., Zhang, J., Evans, A.: Autoencoder for neuroimage. In: International Conference on Database and Expert Systems Applications, pp. 84â90 (2021). Springer
- Zhang et al. [2018] Zhang, W., Wang, J., Jin, D., Oreopoulos, L., Zhang, Z.: A deterministic self-organizing map approach and its application on satellite data based cloud type classification. In: IEEE International Conference on Big Data (Big Data) (2018)
- Tang et al. [2021] Tang, X., Zhang, W., Yu, Y., Turner, K., Derr, T., Wang, M., Ntoutsi, E.: Interpretable visual understanding with cognitive attention network. In: International Conference on Artificial Neural Networks, pp. 555â568 (2021). Springer
- Zhang et al. [2021] Zhang, W., Zhang, L., Pfoser, D., Zhao, L.: Disentangled dynamic graph deep generation. In: Proceedings of the SIAM International Conference on Data Mining (SDM), pp. 738â746 (2021)
- Yazdani et al. [2024] Yazdani, S., Saxena, N., Wang, Z., Wu, Y., Zhang, W.: A comprehensive survey of image and video generative ai: Recent advances, variants, and applications (2024)
- Liu et al. [2021] Liu, Z., Wang, R., Japkowicz, N., Tang, D., Zhang, W., Zhao, J.: Research on unsupervised feature learning for android malware detection based on restricted boltzmann machines. Future Generation Computer Systems 120, 91â108 (2021)
- Liu et al. [2023] Liu, Z., Wang, R., Japkowicz, N., Gomes, H.M., Peng, B., Zhang, W.: Segdroid: An android malware detection method based on sensitive function call graph learning. Expert Systems with Applications, 121125 (2023)
- Cai et al. [2023] Cai, Y., Youngstrom, D., Zhang, W.: Exploring approaches for teaching cybersecurity and ai for k-12. In: 2023 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1559â1564 (2023). IEEE
- Guyet et al. [2022] Guyet, T., Zhang, W., Bifet, A.: Incremental mining of frequent serial episodes considering multiple occurrences. In: 22nd International Conference on Computational Science, pp. 460â472 (2022). Springer
- Zhang [2024] Zhang, W.: Fairness with censorship: Bridging the gap between fairness research and real-world deployment. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 22685â22685 (2024)
- Wang et al. [2025] Wang, Z., Yin, Z., Palikhe, A., Zhang, W.: Fairness in language models: A tutorial. In: Proceedings of the 34th ACM International Conference on Information and Knowledge Management (2025)
- Zhang [2025] Zhang, W.: Online and customizable fairness-aware learning. Knowledge and Information Systems, 1â28 (2025)
- [159] Yin, Z., Wang, Z., Palikhea, A., Liu, Z., Liu, J., Zhang, W.: Amcr: A framework for assessing and mitigating copyright risks in generative models. In: ECAI 2025
- Wang et al. [2025a] Wang, Z., Yin, Z., Zhang, W.: A unified framework for fair graph generation: Theoretical guarantees and empirical advances. Advances in Neural Information Processing Systems (2025)
- Wang et al. [2025b] Wang, Z., Wu, A., Moniz, N., Hu, S., Knijnenburg, B., Zhu, X., Zhang, W.: Towards fairness with limited demographics via disentangled learning, 565â573 (2025) https://doi.org/%****âŁsn-article.bblâŁLineâŁ2725âŁ****10.24963/ijcai.2025/64 . Main Track
- Wang et al. [2025c] Wang, Z., Liu, F., Pan, S., Liu, J., Saeed, F., Qiu, M., Zhang, W.: fairgnn-wod: Fair graph learning without complete demographics, 556â564 (2025) https://doi.org/10.24963/ijcai.2025/63 . Main Track
- Zhang et al. [2025] Zhang, J., Wang, Z., Palikhe, A., Yin, Z., Zhang, W.: Datasets for fairness in language models: An in-depth survey. arXiv preprint arXiv:2506.23411 (2025)