# Towards Transparent AI: A Survey on Explainable Large Language Models
(2025)
Abstract.
Large Language Models (LLMs) have played a pivotal role in advancing Artificial Intelligence (AI). However, despite their achievements, LLMs often struggle to explain their decision-making processes, making them a ’black box’ and presenting a substantial challenge to explainability. This lack of transparency poses a significant obstacle to the adoption of LLMs in high-stakes domain applications, where interpretability is particularly essential. To overcome these limitations, researchers have developed various explainable artificial intelligence (XAI) methods that provide human-interpretable explanations for LLMs. However, a systematic understanding of these methods remains limited. To address this gap, this survey provides a comprehensive review of explainability techniques by categorizing XAI methods based on the underlying transformer architectures of LLMs: encoder-only, decoder-only, and encoder-decoder models. Then these techniques are examined in terms of their evaluation for assessing explainability, and the survey further explores how these explanations are leveraged in practical applications. Finally, it discusses available resources, ongoing research challenges, and future directions, aiming to guide continued efforts toward developing transparent and responsible LLMs. copyright: acmlicensed journalyear: 2025
1. Introduction
In the rapidly advancing field of natural language processing, large language models (LLMs) like BERT (J. and et al, 2018), T5 (C. and et al., 2023), GPT-4 (J. and et al, 2023), and LLaMA-2 (H. and et al, 2023) have demonstrated impressive capabilities across domains such as machine translation, code generation, medical diagnosis, and personalized education (J. and et al., 2020; M. and et al, 2021; C. and et al, 2024; A. and et al., 2020). However, their ’black box’ nature, stemming from vast parameters and extensive training data, obscures internal mechanisms and decision-making processes, complicating explainability. This lack of transparency can lead to unintended issues like model hallucinations and harmful content, posing significant challenges in high-stake fields like health, finance, and law, where interpretability is critical to avoid serious errors. Without insight into decision-making, user trust erodes, and ethical concerns escalate.
Researchers have developed various explainable artificial intelligence (XAI) methods to address concerns about LLMs, revealing their internal processes and decision-making mechanisms with clear, human-level explanations. This is crucial for building user trust, ensuring ethical high-stake decisions, and identifying issues like model hallucinations and biases (e.g., gender bias in translating ’doctor’ to male and ’nurse’ to female pronouns). XAI also aids in debugging by highlighting attention patterns or context errors, improving model performance and reliability for real-world use. However, current XAI literature lacks a systematic focus on LLMs, with many surveys covering traditional models broadly and few addressing the unique challenges of transformer-based architectures (encoder-decoder components with distinct attention mechanisms). A systematic survey is needed to standardize taxonomies and tailor solutions for these architectures’ specific issues.
This survey offers a systematic review of explainable artificial intelligence (XAI) methods in large language models (LLMs), highlighting recent advances, evaluation mechanisms, explanation applications, and future research directions. To the best of our knowledge, this is the first comprehensive survey providing a standard taxonomy based on transformer architectures (encoder-only, decoder-only, and encoder-decoder models), addressing specific challenges in each, and exploring evaluation and application of these explanations. By comparing various XAI methods across architecture-aware perspectives, it delivers a holistic view of techniques and their use cases. Key contributions include: 1) A detailed review of XAI methods with a novel taxonomy, comparing techniques and use cases across LLM architectures; 2) Analysis of evaluation mechanisms and practical applications of XAI explanations; 3) Identification of available resources, research challenges, and future directions for transparent, ethically aligned models.
2. Background
2.1. Large Language Models
Large Language Models (LLMs) have become fundamental to modern natural language processing (NLP), driving progress in applications such as machine translation, dialogue generation, question answering, and code synthesis (B. and et al., 2020b; J. and et al, 2023). These models are primarily built upon the Transformer architecture (A. and et al., 2017), which uses self-attention mechanisms to encode contextual relationships between tokens. Trained on massive text corpora, LLMs learn to capture statistical and semantic patterns in language (H. and et al, 2023), enabling them to generalize across a wide variety of tasks. As the size of these models grows, in terms of both the number of parameters and the diversity of training data, they begin to exhibit emergent behaviors such as in-context learning, multi-task generalization, and the ability to follow natural language instructions (J. and et al, 2022).
Despite their impressive capabilities, LLMs remain largely opaque. Their internal decision-making processes are encoded in high-dimensional and deeply layered representations, making it difficult for users to understand how specific predictions are made (R. and et al, 2021; H. and et al., 2024b). This lack of interpretability presents risks in safety-critical domains such as healthcare, law, and education, where incorrect or biased outputs can have serious consequences. Therefore, improving the transparency and accountability of LLMs has become an urgent priority, motivating the development of explainability methods tailored to their unique architecture and behavior.
2.2. Explainability in LLMs
Explainability refers to the ability to make a model’s internal logic and outputs understandable to humans. This property is essential for ensuring transparency, detecting failure modes, and enabling responsible AI deployment in domains such as healthcare, law, and finance (C. and et al., 2019; H. and et al., 2024b). In the context of large language models (LLMs), explainability is particularly challenging due to their black-box nature. These models rely on deeply stacked Transformer layers and encode knowledge in high-dimensional hidden states, which makes tracing how specific outputs are generated inherently difficult (J. and et al., 2022; P. and et al., 2024).
To address these challenges, researchers have proposed various strategies aimed at increasing the interpretability of LLMs. Some methods focus on saliency-based attribution, which highlights the contribution of input tokens to the model’s output (Y. and et al., 2024a), while others use probing techniques to examine the internal representations at different layers. More recent techniques such as Chain-of-Thought (CoT) prompting have demonstrated that prompting format can shape how intermediate reasoning steps are expressed (J. and et al., 2022). Despite these advances, many current methods are limited in scope, failing to generalize across tasks or model architectures. Importantly, LLMs differ significantly in architecture and each type poses distinct interpretability challenges.
3. Taxonomy
We present a novel taxonomy for categorizing XAI methods based on the underlying transformer architecture of LLMs. This classification organizes the XAI approaches into three fundamental groups, as illustrated in Figure 1. First, XAI methods in encoder-only LLMs are designed to interpret contextual embeddings, leveraging the bidirectional structure of the models to effectively analyze internal representations. Second, XAI methods in decoder-only LLMs rely on prompt engineering and in-context learning strategies to uncover the internal reasoning processes of these models, which are often opaque due to their massive scale and autoregressive nature. Third, XAI methods in encoder-decoder LLMs can provide insights at both the encoding and decoding stages, utilizing the dual architecture to interpret the flow of information through the cross-attention mechanism. This standardized taxonomy offers a comprehensive review of XAI techniques tailored to distinct architectures, enabling meaningful comparisons of their underlying structures and characteristics. It also addresses the unique challenges of achieving explainability in each type of architecture and highlights how these methods can be used for practical applications.
<details>
<summary>extracted/6574933/images/xai-llms-taxonomy.png Details</summary>
### Visual Description
## Diagram: XAI Methods in LLMs
### Overview
The image is a diagram illustrating different XAI (Explainable AI) methods used in Large Language Models (LLMs). It categorizes these methods based on the LLM architecture they are applied to: Encoder-only, Decoder-only, and Encoder-decoder. Each category lists specific XAI techniques.
### Components/Axes
* **Main Title:** XAI methods in LLMs (located at the top center)
* **Categories (Top Level):**
* Encoder-only (Green box, left side)
* Decoder-only (Yellow box, center)
* Encoder-decoder (Blue box, right side)
* **XAI Methods (Sub-categories):** Each category has a list of XAI methods associated with it. These are contained in rounded boxes.
### Detailed Analysis or ### Content Details
**1. Encoder-only (Green):**
* Feature attribution
* Classifier-based probing
* Parameter-free probing
* Attention-based
**2. Decoder-only (Yellow):**
* Feature attribution
* ICL-based
* CoT prompting
* Mechanistic Interpretability
**3. Encoder-decoder (Blue):**
* Feature attribution
* Classifier-based probing
* Attention-based
* Self-explanation
The arrows indicate the relationship between the LLM architecture type (Encoder-only, Decoder-only, Encoder-decoder) and the specific XAI methods applicable to each.
### Key Observations
* "Feature attribution" and "Classifier-based probing" are common XAI methods used across Encoder-only and Encoder-decoder architectures.
* "Feature attribution" is also used in Decoder-only architectures.
* Decoder-only architectures have unique methods like "ICL-based", "CoT prompting", and "Mechanistic Interpretability".
* Encoder-decoder architectures have a unique method called "Self-explanation".
* Encoder-only architectures have a unique method called "Parameter-free probing".
### Interpretation
The diagram provides a structured overview of XAI methods in the context of different LLM architectures. It highlights the diversity of techniques available for understanding and interpreting the behavior of these models. The categorization helps in selecting appropriate XAI methods based on the specific architecture being used. The presence of common methods across different architectures suggests fundamental interpretability approaches, while unique methods indicate architecture-specific considerations.
</details>
Figure 1. Taxonomy of XAI methods in LLMs.
Table 1. Summary of different methodologies used for XAI in LLMs
| (M. and et al., 2021) | LIME | Feature attribution-based | Encoder-only | BERT | External | Local | Post-hoc | Fake news detection classification |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (M. and et al., 2017) | TransSHAP | Feature attribution-based | Encoder-only | BERT | External | Local | Post-hoc | Tweet sentiment classification |
| (I. and et al., 2019) | Edge probing | Classifier-based probing | Encoder-only | BERT | Internal | Local, Global | Post-hoc | NLP labeling tasks |
| (I. and et al., 2019) | Token representation probing | Classifier-based probing | Encoder-only | BERT | Internal | Local | Post-hoc | Surface, syntax, and semantic probing tasks |
| (Z. and et al., 2020) | Perturbation masking | Parameter-free probing | Encoder-only | BERT | Internal | Global | Post-hoc | syntactic parsing, discourse dependency parsing, aspect based sentiment classification |
| (Z. and et al., 2020) | DIRECTPROBE | Parameter-free probing | Encoder-only | BERT | Internal | Global | Post-hoc | Supersense-role, Supersense-function, POS, Dependency Relation, Semantic Relation |
| (J. and et al., 2019b) | BertViz | attention-based | Encoder-only | BERT | Internal | Local | Post-hoc | Detecting model bias, locating relevant attention heads, linking neurons to model behaviour |
| (B. and et al., 2020a) | ExBERT | attention-based | Encoder-only | BERT | Internal | Local | Post-hoc | Dependency relations, coreference, entity relationships |
| (B. and et al., 2024) | AML | Feature attribution-based | Decoder-only | GPT-2 | Internal | Local | Post-hoc | Faithful input attribution |
| (S. and et al., 2024) | Progressive Inference | Feature attribution-based | Decoder-only | OPT, GPT-2 | External | Local | Post-hoc | Stepwise token attribution |
| (L. and et al., 2023) | Complementary Explanations | ICL-based analysis | Decoder-only | GPT-3 | External | Local | Prompt-time | Prompt structure analysis |
| (S. and et al., 2023) | Saliency-Guided ICL Analysis | ICL-based analysis | Decoder-only | GPT-2, GPT-Neo | External | Local | Post-hoc | ICL prompt token attribution |
| (D. and et al., 2024) | SEA-CoT | CoT prompting | Decoder-only | GPT-3.5 | External | Local | Prompt-time | CoT reasoning consistency |
| (D. and et al., 2022) | Least-to-Most Prompting | CoT prompting | Decoder-only | GPT-3 | External | Local | Prompt-time | Hierarchical CoT reasoning |
| (K. and et al., 2022) | ROME | Mechanistic interpretability | Decoder-only | GPT-2 | Internal | Local | Post-hoc | Factual memory editing |
| (W. and et al., 2024) | ACDC | Mechanistic interpretability | Decoder-only | GPT-2 | Internal | Local | Post-hoc | Abstract reasoning attribution |
| (Y. and et al., 2024b) | Program generation explanations | Feature attribution-based | Encoder-decoder | T5, CodeT5, CodeReviewer, CodeT5+ | External | Local | Post-hoc | Code repair, code review, code translation, and text-to-code generation |
| (J. and et al., 2024) | TextGenSHAP | Feature attribution-based | Encoder-decoder | T5-large, T5-XXL and T5-FiD | External | Local | Post-hoc | Abstractive question answering from retrieval-augmented documents |
| (F. and et al., 2021) | Document-level discourse probing | Classifier-based probing | Encoder-decoder | T5, BART | Internal | Global | Post-hoc | Sentence ordering, discourse connective prediction, RST nuclearity prediction, RST relation prediction and cloze story test |
| (Z. and et al., 2021a) | Implicit meaning representation probing | Classifier-based probing | Encoder-decoder | T5, BART | Internal | Global | Post-hoc | Next-instruction prediction, action and game-response generation task |
| (J. and et al., 2021) | Attention-Guided Decoding | Attention-based | Encoder-decoder | T5 | Internal | Local | Post-hoc | Data-to-text NLG alignment |
| (S. and et al., 2021) | Attention Head Masking | Attention-based | Encoder-decoder | BART, PEGASUS | Internal | Local | Post-hoc | Summarization content selection |
| (S. and et al., 2020) | WT5 | Self-explanation | Encoder-decoder | T5-small, T5-large | Internal | Local | Prompt-time | Answer justification generation |
| (Y. and et al., 2022) | Few-Shot NLE Transfer | Self-explanation | Encoder-decoder | T5-base | External | Local | Prompt-time | Explanation transfer across domains |
4. Overview
Based on the proposed taxonomy, this section offers an overview of explainable artificial intelligence (XAI) methodologies for large language models (LLMs), as detailed in Table 1. Studies are categorized across eight dimensions: Paper, Method, Type, Architecture, Model, Access, Scope, and Integration, with Application highlighting practical use cases. The Method (e.g., feature attribution-based, classifier-based probing, attention-based) and Type define the interpretability approach, while Architecture (encoder-only, decoder-only, encoder-decoder) and Model (e.g., BERT, GPT-2, T5) identify targeted LLM frameworks. Access (internal or external) and Scope (local or global) reflect data dependency and interpretability range, and Integration (post-hoc, prompt-time) indicates the implementation stage. Applications cover diverse tasks like fake news detection, sentiment classification, code repair, and abstractive question answering, showcasing the methods’ ability to tackle specific interpretability challenges across LLM architectures.
5. XAI methods in LLMs
This section provides a detailed overview of XAI methods, categorized by three major model architectures, each presenting unique interpretability challenges and driving distinct methodological advances. The first category, encoder-only models, focuses on interpreting stable contextual embeddings from their bidirectional structure. The second, decoder-only models, targets autoregressive generation, often using prompt engineering to assess behavior. The third, encoder-decoder models, emphasizes interactions between encoder and decoder components, with a focus on cross-attention mechanisms. This categorization reveals shared methodological foundations and architecture-specific considerations influencing current and future XAI research.
5.1. XAI methods in encoder-only LLMs
XAI methods for encoder-only models explain contextual representations from their bidirectional attention mechanism (J. and et al, 2018), producing stable, semantically rich embeddings analyzed via techniques like feature attribution and classifier-based probing (H. and et al., 2024b). Masked language modeling (MLM) training simplifies explanation by allowing XAI to trace input token impacts on internal states or predictions, avoiding complexities of generative decoding or encoder-decoder interactions. Key methods include feature attribution, classifier-based probing, parameter-free probing, and attention-based approaches.
5.1.1. Feature attribution-based methods
Feature attribution-based methods assess the relevance of features (e.g., words, phrases) to a model’s prediction (H. and et al., 2024b), assigning scores reflecting each feature’s contribution to the output. These techniques suit encoder-only models, pretrained with masked language modeling (MLM) (J. and et al, 2018), which integrates bidirectional left and right contexts, yielding stable, context-rich embeddings for analyzing feature influence on predictions (Z. and et al., 2021b).
A widely used feature attribution-based XAI method, LIME (T. and et al., 2016), faithfully explains model predictions by locally approximating them with an interpretable model. Szczepański et al. (M. and et al., 2021) applied LIME to BERT-based models fine-tuned for fake news detection, highlighting words and their weights to show their impact on prediction probability—higher weights increase the likelihood of a sentence being classified as fake, and vice versa. However, LIME is limited, failing to meet additive attribution properties like local accuracy, consistency, and missingness (M. and et al., 2017).
This issue is tackled by SHAP (M. and et al., 2017), an effective feature attribution technique that uses Shapley values to measure unique additive feature importance. Kokalj et al. (E. and et al., 2021) extended SHAP for encoder-only models with TransSHAP, adapting it to BERT for tweet sentiment classification, visualizing predictions for positive and negative sentiments. The tool compares impact direction and magnitude per word, outperforming LIME in some visualization aspects, though LIME scored slightly higher in overall user preference. Challenges with SHAP include selecting feature removal methods and managing the computational complexity of estimating Shapley values (H. and et al., 2024b).
5.1.2. Classifier-based probing
Classifier-based probing trains lightweight classifiers on top of encoder-only models like BERT (H. and et al., 2021), starting by freezing parameters and generating representations for input words, phrases, or sentences. Unlike feature attribution methods, which assess input feature contributions to predictions, this approach uses probing classifiers to identify linguistic properties or reasoning abilities (H. and et al., 2024b). It suits encoder-only models, pretrained with masked language modeling (MLM) (J. and et al, 2018), producing robust contextualized embeddings for each token.
To explore word-level contextualized representations in encoder-only models like BERT, Tenney et al. (I. and et al., 2019) introduced edge probing, which examines how models encode sentence structure across syntactic, semantic, local, and long-range phenomena, targeting core NLP tasks like part-of-speech tagging (I. and et al., 2019), semantic role labeling (L. and et al., 2018), and coreference resolution (K. and et al., 2018). Their study shows BERT outperforms non-contextual models, especially in syntactic tasks over semantic ones, indicating stronger encoding of syntactic features. However, edge probing lacks insight into layer-wise behavior and individual token functions (H. and et al., 2021).
To overcome this limitation, Mohebbi et al. (H. and et al., 2021) examine token representations in BERT’s space to explain performance trends across surface, syntactic, and semantic tasks, revealing that higher layers encode required knowledge, with most positional information diminishing through layers while sentence-ending tokens partly transmit it. They also find BERT encodes verb tense and noun number in these tokens, using a diagnostic classifier with attribution methods for qualitative insights. However, Hewitt and Liang (J. and et al., 2019a) question probing classifier faithfulness with control tasks, suggesting parameter-free probing—discussed next—as a solution.
5.1.3. Parameter-free probing
Parameter-free probing techniques bypass training external classifiers or adding parameters, directly analyzing outputs like activation patterns, attention weights, or contextualized embeddings from encoder-only models to detect linguistic structures (Z. and et al., 2020). This addresses a key flaw in classifier-based probing, where the probe may learn downstream task aspects, encoding them in its parameters rather than faithfully mirroring the model’s inherent knowledge (J. and et al., 2019a).
Wu et al. (Z. and et al., 2020) introduce perturbed masking, a parameter-free probing technique to estimate inter-word correlations and extract global syntactic information. This method uses a two-stage process: first, masking a target word, then observing shifts in its contextualized representation when another word is masked, quantifying the second word’s influence. Impact matrices derived from BERT’s output capture these correlations, reflecting attention mechanisms’ dependency structures, though computed from outputs rather than intermediate representations. Algorithms extract syntactic trees from these matrices, showing BERT encodes rich syntactic properties, with the method also assessing its ability to model long document sequences.
Likewise, Zhou et al. (Y. and et al., 2021) note that training classifiers as probes is unreliable due to varying representations requiring different classifiers, which can skew representation quality estimates. To counter this, they propose DIRECTPROBE (Y. and et al., 2021), a parameter-free heuristic method leveraging version space (M. and et al., 1982) to analyze embedding geometry, determining task suitability without classifiers.
5.1.4. Attention-based methods
Encoder-only models like BERT rely on transformer architectures with a fully attention-based, bidirectional self-attention mechanism (J. and et al, 2018). While prior XAI methods, such as feature attribution or lightweight classifier probing, assess input contributions, attention-based methods offer a direct view into model reasoning by visualizing self-attention distributions and token weight allocation. Various tools have been developed to leverage this, providing concise summaries and enhancing user interaction with LLMs (B. and et al., 2020a).
To examine the attention mechanism, Vig et al. (J. and et al., 2019b) developed BertViz, an open-source tool visualizing attention across multiple scales, offering perspectives via a high-level model view (showing all layers and heads) and a low-level neuron view (revealing neuron interactions in attention patterns). Applied to the encoder-only BERT model, it supports three use cases: detecting bias, identifying key attention heads, and linking neurons to behavior. However, relying solely on attention for faithful explanations can yield misleading results (G. and et al., 2020).
To address this, Hoover et al. (B. and et al., 2020a) introduced ExBERT, an interactive tool visualizing attention mechanisms and internal representations in transformer models. It features an attention view for examining aggregated patterns and a corpus view for summarizing hidden representations of selected tokens via statistics. Using attention visualization and nearest-neighbor search, ExBERT reveals captured information and potential biases in text inputs, though it is limited to local analyses, focusing on a few neighbors per token (B. and et al., 2020a).
5.2. XAI methods in decoder-only LLMs
XAI methods for decoder-only LLMs address their autoregressive, left-to-right generation, presenting unique interpretability challenges and opportunities. Unlike encoder-only LLMs with stable, easily probed contextual embeddings, decoder-only models need techniques tailored to their unidirectional nature. Key methods include feature attribution-based approaches, in-context learning (ICL)-based methods, chain-of-thought (CoT) prompting, and mechanistic interpretability.
5.2.1. Feature Attribution-based Methods
Feature attribution-based methods quantify individual input tokens’ contributions to model outputs. As noted in Section 5.1.1, they suit encoder-only models with bidirectional context. However, their application to decoder-only large language models (LLMs) is challenging due to the autoregressive, left-to-right token generation, where conventional bidirectional methods fail to capture the causal structure (H. and et al., 2024b). Recent efforts thus focus on developing attribution techniques tailored to this autoregressive decoding process.
Attributive Masking Learning (AML) (B. and et al., 2024) uses a gradient-based approach to identify a minimal, sufficient input token subset preserving the model’s output distribution, unlike post-hoc approximations. AML employs a mask generator that learns binary token masks by minimizing Kullback-Leibler (KL) divergence (K. and et al., 1951) between full and masked predictions, offering faithful, causally grounded explanations via model gradients and internal activations. However, its dependence on internals limits use with closed-source LLMs like GPT-4 (J. and et al, 2023) or Claude (Anthropic, 2025), and its interpretability may falter in high-compositionality tasks (e.g., multi-hop reasoning) or ambiguous language due to unstable masking responses.
In contrast, Progressive Inference (S. and et al., 2024) provides a parameter-free, model-agnostic method for black-box access, extending the input sequence token by token to track output probability evolution and build an influence curve for each prefix element. Aligned with autoregressive decoding, it explains generative behavior without needing model weights or gradients, broadening its use with proprietary models. However, its token-wise marginal attribution overlooks complex distant token dependencies, reducing explanatory power in tasks like summarization requiring global semantic understanding. Compared to AML, it sacrifices interpretive depth for accessibility, ideal for diagnostic cases with restricted model access.
5.2.2. In-Context Learning (ICL)-based Methods
In-context learning (ICL) enables decoder-only large language models (LLMs) like GPT-3 and GPT-4 (A. and et al., 2023b) to perform novel tasks using a few prompt examples without parameter updates (B. and et al., 2020b), a key feature of these architectures. However, its mechanisms are poorly understood due to the lack of gradient updates and opaque prompt processing. Thus, ICL explainability methods focus on interpreting how models extract and use prompt information, emphasizing structure, semantics, and token-level saliency.
To explore ICL behavior, Lampinen et al. (K. and et al, 2022) used contrastive prompts, substituting semantic content with syntactically coherent but meaningless tokens, revealing that models retain task performance without semantic cues. This suggests decoder-only LLMs depend more on syntactic structures and token positions than true semantic understanding, indicating that their generalization may arise from statistical pattern matching in prompt formats rather than abstract task comprehension.
Expanding on this, Li et al. (S. and et al., 2023) developed a saliency-guided method to examine how decoder-only LLMs interpret prompts at the token level, using output loss gradients to create saliency maps visualizing token influence. Their findings show attention focuses heavily on final tokens (e.g., answers or labels), suggesting models prioritize contrastive signals (e.g., correct vs. incorrect patterns) over holistic representations. This token-level analysis indicates ICL relies more on local alignment and positional regularities than deep semantic integration, underscoring the need to integrate prompt structure analysis with saliency-based explanations for a complete understanding.
5.2.3. Chain-of-Thought (CoT) Prompting
Chain-of-thought (CoT) prompting adds intermediate reasoning steps to decoder-only large language models (LLMs) outputs, turning opaque predictions into interpretable rationales (J. and et al., 2022; T. and et al., 2022). Aligned with autoregressive generation, CoT allows real-time tracking of reasoning paths, boosting performance on complex tasks like math or logic puzzles while embedding interpretability in outputs. Unlike in-context learning (ICL)-based methods, which depend on prompt structure for task behavior, CoT offers explicit reasoning insights in the response.
This makes CoT ideal for decoder-only models, whose left-to-right generation supports incremental, interpretable reasoning. A key method, SEA-CoT (D. and et al., 2024), enhances logical consistency by using a natural language inference (NLI) model to verify each reasoning step against prior ones, filtering incoherent transitions for greater faithfulness and local coherence. However, its reliance on external NLI tools and focus on pairwise step relationships neglects global reasoning validity.
An alternative, Least-to-Most prompting (D. and et al., 2022), breaks complex problems into ordered subtasks (e.g., defining variables, setting equations, solving a math word problem) from easiest to hardest, fostering incremental, clear, and controllable reasoning chains. Unlike SEA-CoT’s post-hoc filtering, it structures reasoning during prompt creation. Though effective in logical and mathematical domains, it demands extensive prompt engineering and struggles with unstructured tasks like open-ended text generation. Together, these methods show how intermediate reasoning design—via validation or decomposition—enhances decoder-only model interpretability.
5.2.4. Mechanistic Interpretability
Mechanistic interpretability methods explain decoder-only large language models (LLMs) by revealing internal computational structures driving behavior, tracing causal pathways to identify roles of specific neurons, attention heads, or feedforward layers in reasoning and prediction (O. and et al., 2020; N. and et al., 2023). Ideal for autoregressive architectures with deeply stacked transformer blocks, this approach isolates modular components (e.g., subnetworks for specific functions). Unlike attribution-based or prompt-driven methods, it provides structural, fine-grained insights based on model parameters and activations.
A key technique, ROME (Rank-One Model Editing) (K. and et al., 2022), offers a localized editing framework to modify factual knowledge in GPT-2 (R. and et al., 2019) by targeting specific feedforward layer neurons, showing some facts are stored in editable subnetworks (e.g., changing “Paris is the capital of France” to “Florida is the capital of France”). This links outputs to identifiable components, enhancing explainability and enabling direct edits. However, ROME is limited to factual recall, assumes fixed activation paths, and struggles with abstract or multi-hop reasoning requiring complex interactions.
Building on this, ACDC (W. and et al., 2024) advances mechanistic interpretability by integrating activation patching, attribution, and pruning to identify minimal subgraphs driving broader cognitive tasks like analogical reasoning or number comparison. Related efforts, such as Path Patching (R. and et al., 2022) and Transformer Circuits (O. and et al., 2022), analyze attention and MLP contributions to trace signal propagation, uncovering reusable, modular components. However, these methods demand full white-box access and are computationally intensive, hindering scalability. Despite these limitations, they are vital for structurally grounded explanations and improving safe, controllable LLMs.
5.3. XAI methods in encoder-decoder LLMs
Encoder-decoder LLMs employ a two-part architecture with a cross-attention mechanism, integrating an encoder for understanding and a decoder for generation, enabling explainability at both stages. Unlike encoder-only models, they enhance generative capabilities, and compared to decoder-only models, they offer better input representation control. Key explainability methods include feature attribution, classifier-based methods, attention-based methods, and self-explanation techniques.
5.3.1. Feature attribution-based methods
Feature attribution-based methods, as outlined in Sections 5.1.1 and 5.2.1, assign relevance scores to each feature. Unlike their application in encoder-only and decoder-only LLMs, in encoder-decoder models, these methods evaluate token relevance across both input and output sequences (J. and et al., 2024), reflecting the sequence-to-sequence nature by considering contributions from preceding output tokens as well as input tokens (C. and et al., 2023).
To apply feature attribution-based methods for XAI in encoder-decoder models, Liu et al. (Y. and et al., 2024b) conducted a study on program generation, using a gradient-based SHAP (M. and et al., 2017) to interpret decision-making. They found models like CodeT5 (Y. and et al., 2023), CodeReviewer (Z. and et al., 2022), and CodeT5+ (Y. and et al., 2023) prioritize keywords and identifiers over operators and separators, indicating strong code grammar and structure recognition. However, performance dropped significantly when removing even minor tokens, revealing robustness issues. Despite SHAP’s utility, its scalability struggles with large encoder-decoder models and long sequences, often causing impractical delays (J. and et al., 2024).
To address this, Enouen et al. (J. and et al., 2024) introduced TextGenSHAP, a more efficient enhancement of Shapley values for large-scale encoder-decoder models like T5-large (H. and et al., 2022), T5-XXL (H. and et al., 2022), and T5-FiD (G. and et al., 2021). Targeting open-ended text generation with long prompts, especially abstractive question answering with retrieval-augmented documents, TextGenSHAP improves recall in document retrieval systems. Though a major step forward in applying SHAP to large encoder-decoder models, computational costs for generating explanations remain a practical concern.
5.3.2. Classifier-based methods
As noted in Section 5.1.2, classifier-based probing enhances explainability for encoder-only LLMs by training shallow classifiers on frozen representations to interpret predictions and behavior (H. and et al., 2021). In encoder-decoder models, this approach uniquely targets both encoder and decoder components (C. and et al., 2023), addressing their complex interactions, which complicates interpretation.
To apply classifier-based probing to encoder-decoder LLMs, Koto et al. (F. and et al., 2021) introduced document-level discourse probing to assess models like T5 (C. and et al., 2023) and BART (M. and et al., 2019) in capturing discourse relations, identifying active layers. The study reveals BART’s encoder layers excel in tasks like discourse connective (A. and et al., 2019), RST nuclearity (W. and et al., 1986), and RST relation (W. and et al., 1986), while its decoder lags, focusing more on generation than understanding. Similar patterns in T5, a denoising autoencoder, show discourse knowledge is unevenly distributed across layers, shaped by pre-training objectives and architecture.
Extending classifier-based probing to implicit meaning, Li et al. (Z. and et al., 2021a) explored how encoder-decoder LLMs like BART and T5 encode discourse-level information states, using a probing method to recover them. Results show implicit meanings are linearly decoded from entity mentions, capturing state changes and semantic details of final states, enhancing insight into these models’ inner workings. However, limitations include recovering complete states only about half the time, with probing representations lacking the expressiveness for human-like language generation.
5.3.3. Attention-based methods
Attention-based methods leverage attention weights, especially cross-attention in encoder-decoder models, to interpret how input influences output generation (C. and et al., 2023). As noted in Section 5.1.4, self-attention aids interpretability in encoder-only LLMs, but encoder-decoder models’ cross-attention layers directly connect encoder inputs to decoder outputs (C. and et al., 2023). This makes attention a key tool for tasks like summarization or translation, where tracing source-to-target sequences is vital (B. and et al., 2020a; J. and et al., 2019b), providing intuitive explanations by aligning attention distributions with semantic relevance.
To interpret encoder-decoder LLMs, Juraska et al. (J. and et al., 2021) introduced attention-guided decoding, analyzing cross-attention activations during decoding to identify source tokens influencing target tokens. By tracking attention weights across decoder steps and rescoring beams for semantic consistency, it reveals information flow from input to output, attributing output elements to specific inputs, aligning with human judgment in translation or summarization. However, its reliability depends on well-calibrated attention maps, as diffusion in lengthy or noisy inputs can diminish interpretive clarity, reducing explanation faithfulness.
In contrast, attention head masking (S. and et al., 2021) uses an intervention-based approach, disabling specific attention heads at inference to assess their causal importance by observing output impacts, providing head-level attribution and insights into internal decision pathways rather than just token tracing. Though precise in isolating functional roles, it struggles with scalability due to combinatorial complexity and may miss distributed patterns where multiple weak heads collectively influence decisions.
5.3.4. Self-explanation methods
Self-explanation-based methods enhance encoder-decoder LLM interpretability by generating human-readable justifications with task predictions, unlike attention-based methods analyzing internal mechanisms (J. and et al., 2021; S. and et al., 2021). Leveraging the decoder’s generative capacity (S. and et al., 2020), they produce explanations within the output sequence, aligning with encoder-decoder architectures as a unified text generation task (F. and et al., 2019). This offers an intuitive interface for assessing reasoning, especially in trust-demanding tasks like QA, classification, and commonsense inference (S. and et al., 2020; Y. and et al., 2022), embedding reasoning into outputs for user-aligned interpretability without post-hoc analysis.
A key self-explanation method, WT5 (S. and et al., 2020), builds on the T5 architecture, training models to output both answers and explanations, e.g., responding to “What is the capital of France?” with “Paris” and “The capital of France is Paris, so the answer is Paris.” This format links supporting facts to conclusions, boosting user trust and, in some cases, task performance via joint training (S. and et al., 2020). However, it relies on extensive explanation-annotated data, and it’s uncertain whether explanations reflect causal reasoning or just label patterns from training.
Extending this approach, Yordan et al. (Y. and et al., 2022) introduced Few-Shot Out-of-Domain NLE generation, targeting tasks with limited labeled explanations but abundant labels. Their method pretrains a model on a well-annotated source domain, then adapts it to a target domain with ample labels but scarce explanation data by aligning representation spaces and fine-tuning on limited examples, enabling quality explanation generation under data-scarce conditions (Y. and et al., 2022). Unlike WT5’s rich supervision assumption, this enhances scalability across domains, though few-shot, knowledge-driven methods struggle with generalizing explanation quality and style, particularly in abstract reasoning.
6. Challenges and Future Directions
Explainable artificial intelligence (XAI) for large language models (LLMs) is crucial for transparency and trust in high-stakes fields like healthcare and law (A. and et al., 2023a; B. and et al, 2020). Despite progress in XAI methods for encoder-only, decoder-only, and encoder-decoder architectures (Section 5), notable challenges remain, requiring innovative solutions. Below, we outline four key challenges and future directions to enhance explainable LLMs.
(1) Explanation and Emergent Abilities Ground truth explanations for LLMs are typically inaccessible, with no benchmark datasets to evaluate global explanations of individual components. This hinders assessing explanation faithfulness and fidelity, especially for emergent abilities that arise as models scale and training data grows, even without explicit training for such tasks. Open questions include: (1) How can we design algorithms to reflect LLM decision-making without ground truth? (2) What model architectures and data characteristics drive emergent skills in LLMs like ChatGPT or LLaMA? Solutions may involve human evaluations, synthetic datasets, and investigating the interplay between model design and training data (H. and et al., 2024a).
(2) Paradigm Comparison and Shortcut Learning The fine-tuning and prompting paradigms show divergent performance in natural language inference (NLI) and out-of-distribution (OOD) tasks, relying on distinct reasoning strategies. Recent studies reveal LLMs use shortcuts like lexical or position bias in fine-tuning and falter with long contexts in prompting. Key questions are: (1) How do their prediction rationales differ? (2) Can OOD robustness be linked to reasoning variations? Tackling this involves comparing explanatory differences and reducing shortcut dependence to enhance generalization (H. and et al., 2024a).
(3) Interpretability Efficiency and Redundancy High-dimensional representations and deep transformer layers in LLMs like BERT (J. and et al, 2018) and GPT-4 (J. and et al, 2023) render decision tracing computationally intensive. XAI methods like SHAP (M. and et al., 2017; J. and et al., 2024) require substantial resources, hindering scalability. Additionally, attention redundancy in models like BERT-base and OPT-66B indicates pruning redundant heads could boost efficiency with minimal performance impact. Future efforts should prioritize efficient XAI techniques and model compression strategies (H. and et al., 2024a).
(4) Ethics and Temporal Analysis The lack of interpretability raises ethical risks like bias and misinformation, requiring audits and alignment with human values. Tools like BertViz (J. and et al., 2019b) expose biases, but research overlooks training dynamics, depending on post-hoc analysis. Temporal studies, such as detecting Syntactic Attention Structure (SAS) in pre-training, indicate dynamic analysis could reveal causal patterns. Emphasizing interpretability with scale and safety is vital for ethical LLM development (H. and et al., 2024a).
7. Conclusion
Large language models have transformed natural language processing, but their opaque nature hinders transparency and trust in critical domains like healthcare and law. This survey offers a comprehensive review of explainable AI methods for these models, introducing a novel taxonomy based on encoder-only, decoder-only, and encoder-decoder architectures, and examining techniques such as feature attribution, probing, attention-based methods, and chain-of-thought prompting. Despite progress, challenges in complexity, standardization, generalization, efficiency, and ethics underscore the need for innovative solutions to achieve scalable, faithful, and ethical explanations, guiding future efforts toward transparent, responsible, and trustworthy models.
References
- (1)
- A. and et al. (2023a) Chaddad A. and et al. 2023a. Survey of Explainable AI Techniques in Healthcare. Sensors 23 (01 2023), 634.
- A. and et al. (2023b) Katherine A. and et al. 2023b. Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety in high-school students. Big Data and Cognitive Computing 7, 3 (2023), 124.
- A. and et al. (2020) Lopes A. and et al. 2020. Lite Training Strategies for Portuguese-English and English-Portuguese Translation. arXiv:2008.08769 [cs.CL]
- A. and et al. (2019) Nie A. and et al. 2019. DisSent: Learning Sentence Representations from Explicit Discourse Relations. In ACL. 4497–4510.
- A. and et al. (2017) Vaswani A. and et al. 2017. Attention is all you need. In NeurIPS. 30.
- Anthropic (2025) Anthropic. 2025. Official Documentation for Anthropic APIs and Models. Accessed on June 16, 2025.
- B. and et al (2020) Hutchinson B. and et al. 2020. Accountability of AI under the law: The role of explanation. arXiv:1711.01134 [cs.AI]
- B. and et al. (2020a) Hoover B. and et al. 2020a. exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models. In ACL. 187–196.
- B. and et al. (2024) Oren B. and et al. 2024. LLM Explainability via Attributive Masking Learning. In EMNLP. 9522–9537.
- B. and et al. (2020b) Tom B. and et al. 2020b. Language models are few-shot learners. NeurIPS 33 (2020), 1877–1901.
- C. and et al (2024) Ng C. and et al. 2024. Educational personalized learning path planning with large language models. arXiv:2407.11773 [cs.AI]
- C. and et al. (2019) Rudin C. and et al. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 5 (2019), 206–215.
- C. and et al. (2023) Raffel C. and et al. 2023. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs.LG]
- D. and et al. (2022) Zhou D. and et al. 2022. Least-to-most prompting enables complex reasoning in large language models. ICLR (2022).
- D. and et al. (2024) Zhang D. and et al. 2024. SEA-CoT: Self-Entailment-Alignment Chain-of-Thought Prompting for Explainable Reasoning. In NAACL.
- E. and et al. (2021) Kokalj E. and et al. 2021. BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In EACL Hackashop. 16–21.
- F. and et al. (2021) Koto F. and et al. 2021. Discourse Probing of Pretrained Language Models. In NAACL-HLT. 3849–3864.
- F. and et al. (2019) Rajani N. F. and et al. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In ACL. 4932–4942.
- G. and et al. (2020) Brunner G. and et al. 2020. On Identifiability in Transformers. arXiv:1908.04211 [cs.CL]
- G. and et al. (2021) Izacard G. and et al. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In EACL. 874–880.
- H. and et al. (2022) Chung H. and et al. 2022. Scaling Instruction-Finetuned Language Models. arXiv:2210.11416
- H. and et al. (2024a) Luo H. and et al. 2024a. From Understanding to Utilization: A Survey on Explainability for Large Language Models. arXiv:2401.12874 [cs.CL]
- H. and et al. (2021) Mohebbi H. and et al. 2021. Exploring the Role of BERT Token Representations to Explain Sentence Probing Results. In EMNLP. 792–806.
- H. and et al (2023) Touvron H. and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 [cs.AI]
- H. and et al. (2024b) Zhao H. and et al. 2024b. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology 15, 2 (2024), 1–38.
- I. and et al. (2019) Tenney I. and et al. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In ICLR.
- J. and et al (2023) Achiam J. and et al. 2023. Gpt-4 technical report. arXiv:2303.08774
- J. and et al (2018) Devlin J. and et al. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 [cs.CL]
- J. and et al. (2024) Enouen J. and et al. 2024. TextGenSHAP: Scalable Post-Hoc Explanations in Text Generation with Long Documents. In ACL. 13984–14011.
- J. and et al. (2019a) Hewitt J. and et al. 2019a. Designing and Interpreting Probes with Control Tasks. arXiv:1909.03368 [cs.CL]
- J. and et al. (2021) Juraj J. and et al. 2021. Attention Is Indeed All You Need: Semantically Attention-Guided Decoding for Data-to-Text NLG. In INLG. 416–431.
- J. and et al. (2020) Lee J. and et al. 2020. BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining. Bioinformatics 36, 4 (2020), 1234–1240.
- J. and et al. (2019b) Vig J. and et al. 2019b. A Multiscale Visualization of Attention in the Transformer Model. In ACL. 37–42.
- J. and et al. (2022) Wei J. and et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 35 (2022), 24824–24837.
- J. and et al (2022) Wei J. and et al. 2022. Emergent abilities of large language models. arXiv:2206.07682 [cs.AI]
- J. and et al. (2022) Yu J. and et al. 2022. INTERACTION: A Generative XAI Framework for Natural Language Inference Explanations. arXiv:2209.01061 [cs.CL]
- K. and et al. (2018) Lee K. and et al. 2018. Higher-Order Coreference Resolution with Coarse-to-Fine Inference. In NAACL-HLT. 687–692.
- K. and et al (2022) Lampinen K. and et al. 2022. Can language models learn from explanations in context? arXiv:2204.02329 [cs.AI]
- K. and et al. (2022) Meng K. and et al. 2022. Locating and editing factual associations in GPT. In NeurIPS. 17359–17372.
- K. and et al. (1951) Solomon K. and et al. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics 22, 1 (March 1951), 79–86.
- L. and et al. (2023) Andrew L. and et al. 2023. Complementary Explanations: In-Context Learning via Composition of Natural Language and Computation Traces. In ACL.
- L. and et al. (2018) He L. and et al. 2018. Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling. In ACL. 364–369.
- M. and et al (2021) Chen M. and et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374 [cs.LG]
- M. and et al. (2019) Lewis M. and et al. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv:1910.13461 [cs.CL]
- M. and et al. (2017) Lundberg S. M. and et al. 2017. A Unified Approach to Interpreting Model Predictions. In NeurIPS.
- M. and et al. (1982) Mitchell T. M. and et al. 1982. Generalization as search. Artificial Intelligence 18, 2 (1982), 203–226.
- M. and et al. (2021) Szczepański M. and et al. 2021. New explainability method for BERT-based model in fake news detection. Scientific Reports 11, 1 (2021), 23705.
- N. and et al. (2023) Nanda N. and et al. 2023. Progress measures for grokking via mechanistic interpretability. In ICLR.
- O. and et al. (2020) Chris O. and et al. 2020. Zoom in: An introduction to circuits. Distill 5, 3 (2020), e00024–001.
- O. and et al. (2022) Catherine O. and et al. 2022. In-context learning and induction heads. arXiv:2209.11895 [cs.CL]
- P. and et al. (2024) Mavrepis P. and et al. 2024. XAI for All: Can Large Language Models Simplify Explainable AI? arXiv:2401.13110 [cs.AI]
- R. and et al. (2019) Alec R. and et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- R. and et al (2021) Bommasani R. and et al. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258 [cs.AI]
- R. and et al. (2022) Wang R. and et al. 2022. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. In ICLR.
- S. and et al. (2021) Cao S. and et al. 2021. Attention Head Masking for Inference Time Content Selection in Abstractive Summarization. In NAACL-HLT. 5008–5016.
- S. and et al. (2024) Kariyappa S. and et al. 2024. Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictions. In ICML. 23238–23255.
- S. and et al. (2023) Li S. and et al. 2023. Analyzing In-Context Learning with Saliency Methods: A Case Study on Contrastive Demonstrations. (2023). arXiv:2307.05052 [cs.CL]
- S. and et al. (2020) Narang S. and et al. 2020. Wt5?! training text-to-text models to explain their predictions. arXiv:2004.14546 [cs.CL]
- T. and et al. (2022) Kojima T. and et al. 2022. Large language models are zero-shot reasoners. NeurIPS 35 (2022), 22199–22213.
- T. and et al. (2016) Ribeiro M. T. and et al. 2016. ”Why should I trust you?”: Explaining the predictions of any classifier. In KDD. 1135–1144.
- W. and et al. (2024) Gurnee W. and et al. 2024. Language Models Represent Space and Time. ICLR.
- W. and et al. (1986) Mann W. and et al. 1986. Assertions from Discourse Structure. In SC-NL Workshop.
- Y. and et al. (2024a) Dang Y. and et al. 2024a. Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey. arXiv:2412.02104
- Y. and et al. (2024b) Liu Y. and et al. 2024b. On the Reliability and Explainability of Language Models for Program Generation. ACM Trans. Softw. Eng. Methodol. 33, 5, Article 126 (2024), 26 pages.
- Y. and et al. (2023) Wang Y. and et al. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In EMNLP. 1069–1088.
- Y. and et al. (2022) Yordan Y. and et al. 2022. Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant Setup. In EMNLP. 3486–3501.
- Y. and et al. (2021) Zhou Y. and et al. 2021. DirectProbe: Studying Representations without Classifiers. In NAACL-HLT. 5070–5083.
- Z. and et al. (2021a) Li Z. and et al. 2021a. Implicit Representations of Meaning in Neural Language Models. In ACL-IJCNLP. 1813–1827.
- Z. and et al. (2022) Li Z. and et al. 2022. Automating Code Review Activities by Large-Scale Pre-training. arXiv:2203.09095 [cs.CL]
- Z. and et al. (2020) Wu Z. and et al. 2020. Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. In ACL. 4166–4176.
- Z. and et al. (2021b) Wu Z. and et al. 2021b. On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification. arXiv:2101.00196