2509.10929v1

# Clarifying Model Transparency: Interpretability versus Explainability in Deep Learning with MNIST and IMDB Examples **Authors**: Mitali Raj > School of Cyber Security and Digital ForensicsNational Forensic Sciences UniversityNew Delhi, Indiamitali.btmtcs22215@nfsu.ac.in Abstract The impressive capabilities of deep learning models are often counterbalanced by their inherent opacity, commonly termed the "black box" problem, which impedes their widespread acceptance in high-trust domains. In response, the intersecting disciplines of interpretability and explainability, collectively falling under the Explainable AI (XAI) umbrella, have become focal points of research. Although these terms are frequently used as synonyms, they carry distinct conceptual weights. This document offers a comparative exploration of interpretability and explainability within the deep learning paradigm, carefully outlining their respective definitions, objectives, prevalent methodologies, and inherent difficulties. Through illustrative examinations of the MNIST digit classification task and IMDB sentiment analysis, we substantiate a key argument: interpretability generally pertains to a model’s inherent capacity for human comprehension of its operational mechanisms (global understanding), whereas explainability is more commonly associated with post-hoc techniques designed to illuminate the basis for a model’s individual predictions or behaviors (local explanations). For example, feature attribution methods can reveal why a specific MNIST image is recognized as a ’7’, and word-level importance can clarify an IMDB sentiment outcome. However, these local insights do not render the complex underlying model globally transparent. A clear grasp of this differentiation, as demonstrated by these standard datasets, is vital for fostering dependable and sound artificial intelligence. Index Terms: Deep Learning, Explainable AI (XAI), Model Interpretability, Prediction Explainability, Black Box Systems, Trustworthy AI, MNIST, IMDB, Case Study Analysis, Local Explanations, Global Understanding. publicationid: pubid: XXX-X-XXXX-XXXX-X/XX/$31.00 © 2026 IEEE I Introduction Deep learning (DL) has instigated transformative progress across numerous domains, such as computer vision and natural language processing, frequently demonstrating performance levels that surpass human benchmarks. Nevertheless, the sophisticated, non-linear architecture of DL models, which can encompass millions of adjustable parameters, frequently obscures their internal decision-making pathways. This "black box" attribute fuels apprehensions regarding reliability, accountability, equity, and security, particularly when deployed in critical applications. To allay such concerns, significant research efforts have been directed towards enhancing the transparency of DL models. Central to this endeavor are two pivotal concepts: interpretability and explainability. Despite their common interchangeable usage, a more detailed examination uncovers crucial distinctions. This paper endeavors to analyze and contrast these notions, anchoring the discussion with tangible examples derived from established DL applications: image recognition on the MNIST dataset and text-based sentiment classification using the IMDB dataset. These case studies will effectively illustrate how specific prediction explanations can be achieved even when a model’s overall global interpretability remains limited. II Defining Interpretability and Explainability No universally endorsed definitions exist for interpretability or explainability, which has led to a degree of terminological fluidity in academic discourse. Nonetheless, practical definitions can be formulated based on prevalent usage and influential research. II-A Interpretability - Definition: The notion of interpretability centers on the extent to which a human observer can grasp the causal relationships or the underlying functional principles governing a model’s decision-making (Lipton, 2018 [3]; Doshi-Velez & Kim, 2017 [2]). The emphasis is on comprehending how the model functions at a systemic level. - Focus: Intrinsic characteristics of the model, architectural transparency, global understanding. - Goal: The construction of models that are fundamentally comprehensible to humans, whether in their complete form (global interpretability) or regarding their individual components. - Example Question: “By what general process does this Convolutional Neural Network (CNN) convert raw pixel data, through its successive layers, into a final digit classification?” II-B Explainability (within XAI) - Definition: Explainability pertains to the capacity to furnish an interface or supplementary data that elucidates a model’s operational behavior, especially its individual predictions, in a manner that is intelligible to humans (Arrieta et al., 2020 [1]). This often entails the application of post-hoc methods to pre-trained models. - Focus: Justification of model outputs after the fact, understanding at the level of individual predictions, local explanations. - Goal: The generation of clear rationales for specific instances or for the model’s general behavioral patterns, frequently without needing to make the model’s entire internal architecture transparent. - Example Question: “What factors led the model to classify this particular MNIST image as a ’3’ rather than an ’8’?” or “Which specific words in this IMDB review instance were most decisive for its ’positive’ sentiment label?” III Comparative Analysis The principal distinctions between interpretability and explainability are outlined in Table I. TABLE I: Distinguishing Features of Interpretability and Explainability (XAI) | Core Objective | Comprehend the model’s internal logic/system (Global Understanding). | Understand the rationale for a specific decision/prediction (Local Explanation). | | --- | --- | --- | | Methodological Timing | Primarily intrinsic to model design (ante-hoc). | Frequently extrinsic, applied post-training (post-hoc). | | Preferred Model Type | Leans towards simpler, inherently transparent architectures. | Applicable to complex, opaque models (typical of DL). | | Reach of Understanding | Can be global (entire model) or modular (components). | Often instance-specific (local predictions), though global approximations are possible. | | Role of Human User | Human can directly understand the model’s functioning. | Human receives an explanation produced by an auxiliary process. | | Exemplary Techniques | Simpler models (e.g., linear regression, decision trees), rule induction, concept-based systems (e.g., ProtoPNet). | Feature attribution maps (e.g., Grad-CAM), LIME, SHAP, counterfactual reasoning, attention visualization. | | Interrelation | An interpretable model is by nature explainable. | An explainable model is not necessarily interpretable. | IV Illustrative Case Studies: Anchoring Explainability Concepts To provide a concrete illustration of this distinction—especially the point that explainability (yielding local explanations) can be attained for opaque models lacking comprehensive interpretability (global understanding)—we investigate two prevalent deep learning scenarios. IV-A Case Study 1: MNIST Handwritten Digit Recognition <details> <summary>figure1.png Details</summary>  ### Visual Description ## Heatmap: SHAP Value Distribution for Digit '7' ### Overview The image consists of two panels: 1. **Left Panel**: A pixelated black-and-white digit '7' on a black background. 2. **Right Panel**: A heatmap overlaying the digit '7', colored in red (positive values) and blue (negative values), with a color scale labeled "SHAP value" at the bottom. ### Components/Axes - **Color Scale (Legend)**: - Position: Bottom center, spanning the width of the heatmap. - Labels: - Numerical range: `-6` (blue) to `6` (red), with a note `1e-9` at the far right. - Gradient: Blue (negative) → White (neutral) → Red (positive). - Text: "SHAP value" explicitly labeled. - **Heatmap**: - Position: Right panel, overlaid on the digit '7'. - Color Coding: - Red: Positive SHAP values (contributing to the model's output). - Blue: Negative SHAP values (reducing the model's output). - Structure: - The digit '7' is segmented into pixels, each colored based on SHAP value. - Red dominates the top horizontal stroke and right vertical stroke of the '7'. - Blue dominates the lower-left diagonal stroke of the '7'. ### Detailed Analysis - **SHAP Value Distribution**: - The heatmap reveals that the top horizontal and right vertical segments of the '7' have the highest positive SHAP values (red), indicating these pixels strongly influence the model's prediction. - The lower-left diagonal stroke has negative SHAP values (blue), suggesting these pixels reduce the model's confidence in the '7' classification. - The color scale’s `1e-9` note likely indicates the scale’s maximum absolute value (e.g., `±6e-9`), though this is ambiguous without explicit units. - **Pixel-Level Trends**: - Red pixels cluster along the upper and right edges of the '7', forming a continuous band. - Blue pixels form a diagonal band in the lower-left quadrant of the digit. - Neutral (white) pixels are sparse, concentrated near the center of the '7'. ### Key Observations 1. **Dominant Contributions**: The model relies heavily on the top and right segments of the '7' for classification. 2. **Negative Influence**: The lower-left diagonal stroke acts as a "corrective" feature, reducing the model’s output. 3. **Scale Ambiguity**: The `1e-9` notation on the color scale is unclear—it may represent a scaling factor (e.g., `SHAP values × 1e-9`) or a typo. ### Interpretation - **Model Behavior**: The SHAP values highlight that the model prioritizes the upper and right edges of the digit '7' for recognition. The negative values in the lower-left suggest the model may penalize deviations in this region. - **Visualization Purpose**: This heatmap explains how individual pixels contribute to the model’s output, useful for debugging or improving digit recognition systems. - **Uncertainties**: - The `1e-9` label on the color scale lacks context (e.g., units or scaling rationale). - The heatmap’s resolution is low, making precise SHAP value localization challenging. ## Notes - No non-English text is present. - All textual elements (labels, scale, legend) are in English. - The heatmap’s color-to-value mapping is consistent with standard SHAP visualization conventions. </details> Figure 1: SHAP-based local explanation of a CNN’s prediction for an MNIST digit classified as ’7’. Left: Original input image. Right: SHAP heatmap indicating pixel-level feature contributions; red areas contribute positively to the prediction ’7’, while blue areas contribute negatively. The MNIST dataset consists of 70,000 grayscale images depicting handwritten digits (0-9). CNNs represent the standard, high-efficacy models for this recognition task. - The Opaque Nature of the Model: A standard CNN for MNIST typically incorporates several convolutional and pooling layers, followed by fully-connected layers, amounting to potentially millions of parameters. Although the architectural blueprint is human-designed, deciphering precisely how a particular configuration of learned filter weights across all layers culminates in a digit classification presents a formidable challenge. The model’s global operational mechanics remain largely obscure, precluding an easy articulation of its learned comprehensive rule-set for achieving global understanding. - Attaining Local Explainability (Post-hoc): Notwithstanding the CNN’s opacity, explanations for individual predictions are obtainable. - Feature Attribution Maps (e.g., Grad-CAM [7], SHAP): For an image classified as a ’7’, such methods can produce a visual heatmap or assign scores that pinpoint the input image pixels most critical to this ’7’ determination (e.g., the horizontal bar and diagonal stroke characteristic of a ’7’). This offers a localized, post-hoc rationale: “The model designated this as ’7’ due to its focus on these ’7’-indicative pixel areas.” (See Fig. 1). - LIME [5]: LIME can locally approximate the CNN’s decision boundary for a given image by introducing perturbations (like toggling superpixels) and training a simpler, understandable model (such as a sparse linear regressor) on these variations. This would identify which superpixels steered the decision towards ’7’. - Takeaway from MNIST Analysis: XAI tools empower us to effectively generate a local explanation for why a specific image is assigned a particular digit label by highlighting crucial input features. Nevertheless, these explanations do not transform the entire CNN into a globally interpretable system. We gain insight into the ’why’ for an individual instance, but not necessarily the complete ’how’ of the model’s overarching learned function. IV-B Case Study 2: IMDB Movie Review Sentiment Analysis <details> <summary>figure2.png Details</summary>  ### Visual Description ## Bar Chart: LIME Explanation for 'positive' prediction (Instance 5) ### Overview The chart visualizes the contribution of individual words to a machine learning model's positive prediction for a specific instance (Instance 5). Words are ranked by their impact, with horizontal red bars representing their contribution values. All values are negative, indicating inverse relationships between word presence and the positive prediction outcome. ### Components/Axes - **Y-Axis (Categories)**: Words listed vertically, ordered by contribution magnitude (top = largest impact). Labels: `movie`, `cheap`, `on`, `this`, `least`, `channel`, `and`, `i`, `doesn`, `see`. - **X-Axis (Values)**: Numerical scale from **-0.010** (left) to **0.000** (right), representing contribution strength. - **Legend**: Located on the right, with a single red bar labeled as the contribution metric. - **Title**: Positioned at the top center. ### Detailed Analysis - **Word Contributions**: - `movie`: **-0.010** (longest bar, farthest left). - `cheap`: **-0.009** (second-longest). - `on`: **-0.008** (third-longest). - `this`: **-0.007** (fourth-longest). - `least`: **-0.006** (fifth-longest). - `channel`: **-0.005** (sixth-longest). - `and`: **-0.004** (seventh-longest). - `i`: **-0.003** (eighth-longest). - `doesn`: **-0.002** (ninth-longest). - `see`: **-0.001** (shortest bar, closest to 0.000). ### Key Observations 1. **Inverse Relationship**: All contributions are negative, suggesting that the presence of these words reduces the model's confidence in a positive prediction. 2. **Dominant Words**: `movie` and `cheap` have the strongest negative impact (-0.010 and -0.009, respectively). 3. **Gradual Decline**: Contribution magnitudes decrease progressively from `movie` (-0.010) to `see` (-0.001). 4. **Minimal Impact**: `see` has the weakest effect (-0.001), barely influencing the prediction. ### Interpretation The model's positive prediction for this instance is most strongly influenced by the **absence** of words like `movie` and `cheap`, which are associated with negative sentiment or context. Conversely, words like `see` have negligible impact. This suggests the model interprets the lack of certain keywords (e.g., `movie`, `cheap`) as a positive signal, while their presence detracts from the prediction. The gradual decline in contribution magnitude indicates a hierarchy of word importance, with `movie` being the most critical factor. **Note**: LIME explanations often highlight features that *increase* prediction confidence. Negative values here imply these words act as "negative indicators" for the positive class, requiring their absence for a favorable outcome. </details> Figure 2: LIME explanation for a positive sentiment prediction on an IMDB movie review. Although the predicted label is positive, the words shown (e.g., "cheap," "least," "channel") had negative contributions to the classification. The explanation helps interpret the model’s local reasoning for this instance. The IMDB dataset comprises 50,000 movie reviews, each categorized as positive or negative. Models like RNNs (e.g., LSTMs) and Transformers (e.g., BERT) deliver leading performance on this task. - The Opaque Nature of the Model: Architectures such as LSTMs or Transformers employ intricate sequential data processing and attention mechanisms on input word embeddings. Tracing the transformation of an input sentence’s semantic content through recurrent states or multiple self-attention layers to a final sentiment score is a complex endeavor. The learned representations are high-dimensional, and their precise semantic interpretations are not always readily apparent, making these models inherently uninterpretable for global understanding. - Attaining Local Explainability (Post-hoc): - Attention Visualization (Relevant for Transformers): When utilizing a Transformer, attention weights can reveal which words the model emphasized while processing others, or which words received the most attention from the final classification token. For a review classified as ’positive’, strong attention might fall on terms like “excellent,” “brilliant,” or “loved.” - LIME [5] or SHAP [6]: These techniques can be adapted for text models, for example, by omitting words from a review and observing the consequent shift in prediction probability. They can then allocate importance scores to each word in a specific review, signifying which words were most instrumental in the ’positive’ or ’negative’ classification. An example explanation might be: “This review was classified as negative primarily due to the substantial negative influence of the words ’awful’ and ’waste’.” - Takeaway from IMDB Analysis: Analogous to the MNIST scenario, XAI methods enable the generation of local explanations for the sentiment prediction of an individual review by identifying key influential textual elements. This yields valuable, localized comprehension. However, the comprehensive internal logic by which the LSTM or Transformer model processes language to discern sentiment remains multifaceted and largely uninterpretable from a global perspective. V Synthesizing Insights from Case Studies The analyses of MNIST and IMDB datasets furnish concrete evidence that: - Deep learning models, despite their power, frequently function as opaque systems, rendering their global internal processes challenging to interpret for holistic understanding. - Post-hoc XAI methodologies can deliver valuable local explanations for specific predictions generated by these opaque models. Pixel attribution maps for MNIST images and word-level importance scores for IMDB reviews serve as excellent examples. - These local explanations address the “why this particular prediction?” query, thereby enhancing localized trust and comprehension. - Nonetheless, the provision of such local explanations does not inherently render the entire model globally interpretable. The sophisticated underlying mechanisms often remain inscrutable. - Consequently, a model that is explainable (i.e., for which local justifications for outputs can be produced) is not axiomatically interpretable (i.e., one whose fundamental operations are transparent for global understanding). VI General Techniques in Deep Learning The field offers a spectrum of methods, extending beyond those highlighted in our case studies. VI-A Interpretability-focused Techniques (Often "Interpretable by Design" for Global Understanding) - Structured Attention Mechanisms: Designing models where attention is a constrained, core element can foster interpretability by forcing explicit focus indication. - Concept-based Architectures (e.g., TCAV): These aim to link model decisions to high-level, human-intelligible concepts. - Generalized Additive Models (GAMs) / Neural Additive Models (NAMs): These learn a sum of non-linear transformations of individual input features. - ProtoPNet (Prototypical Part Network): Predictions are made by comparing input segments to learned class-specific prototypes. - Rule Extraction from Neural Networks: This involves attempts to translate the learned knowledge of a neural network into a set of symbolic rules. VI-B Explainability-focused Techniques (Often Post-hoc for Local Explanations) - Saliency/Feature Attribution Maps (as in MNIST study): - Gradient-derived: Methods like Vanilla Gradients, Integrated Gradients, Grad-CAM. - Perturbation-based: Techniques such as occlusion sensitivity, or certain SHAP algorithm variants. - LIME (Local Interpretable Model-agnostic Explanations) (relevant to MNIST & IMDB studies): This approach locally trains a simpler, interpretable surrogate model. - SHAP (SHapley Additive exPlanations) (relevant to IMDB & MNIST studies): A game-theoretic method for assessing feature importance. - Counterfactual Explanations: These identify the minimal input modifications required to change a prediction. - Influence Functions: These assess the impact of individual training samples on model parameters and predictions. VII Importance and Applications Both interpretability (which targets global understanding) and explainability (which provides local explanations) are vital for endeavors such as building trust, facilitating model debugging, ensuring fairness, meeting regulatory compliance, enabling scientific discovery, and guaranteeing safety in critical systems. The case studies underscore that even when complete global interpretability is unattainable, instance-specific local explainability offers significant value. VIII Challenges and Limitations - Performance vs. Transparency Trade-off: High-accuracy DL models (as used for MNIST/IMDB) frequently lack global interpretability. Simpler, more interpretable alternatives might compromise predictive power. Post-hoc local explanations are often approximations. - Veracity of Explanations: Validating that local explanations genuinely mirror the model’s reasoning for a specific instance remains a significant hurdle. - Human-Centricity of Explanations: The definition of an effective or “understandable” local explanation is inherently subjective and varies with context and user. - Objective Evaluation: Quantifying the quality of global interpretability or a local explanation poses considerable difficulty. - Resource Intensiveness: Many XAI techniques demand substantial computational resources. - Breadth of Insight: Most XAI tools deliver local explanations. Achieving a reliable global understanding of entire DL models is a more elusive goal. IX The Symbiotic Relationship Although distinct, interpretability (global understanding) and explainability (local explanations) are not mutually exclusive concepts. - Incorporating inherently interpretable modules can enhance the coherence and reliability of post-hoc local explanations. - Local explanation methods can serve as auditing tools for models that are partially interpretable. The overarching aim is to achieve an adequate level of understanding pertinent to the specific application. This might be realized through an intrinsically globally interpretable model or through an opaque model supplemented by robust local explanations, as demonstrated by our MNIST/IMDB scenarios where local explanations furnish key insights despite the models’ overall complexity. X Future Directions - Development of Inherently Interpretable Deep Learning Systems: Pursuing novel DL architectures that combine high performance with inherent transparency for global understanding. - Advancement of Causal Explanations: Progressing from correlational feature importance in local explanations towards insights grounded in causal inference. - Creation of Interactive and Personalized Explanation Interfaces: Designing tools that provide tailored local explanations responsive to user queries. - Enhancing Robustness and Scrutiny of Explanations: Ensuring that local explanations are resilient to manipulation and are not deceptive. - Establishment of Standardization and Benchmarking Protocols: Developing common evaluative frameworks for both types of methodologies. XI Conclusion Within the deep learning landscape, interpretability is primarily associated with the inherent comprehensibility of a model’s internal operations, striving for a global understanding of its systemic behavior. This is commonly pursued via architectural choices that promote transparency. Conversely, explainability (XAI) generally entails the use of post-hoc methodologies to offer human-intelligible rationales for specific model outputs, thereby delivering local explanations concerning the behavior of intricate, often opaque, systems. Our investigations using the MNIST and IMDB datasets, coupled with the practical application of XAI utilities, serve to concretely delineate this difference. For a Convolutional Neural Network (CNN) tasked with MNIST digit classification, explainability techniques such as SHAP or other saliency approaches can produce pixel importance maps. These visualizations identify the pixels most critical to a particular digit’s classification (e.g., elucidating why a given image is identified as a ’7’ by highlighting its formative pixel configurations). In a similar vein, for models like LSTMs or Transformers engaged in IMDB movie review analysis, techniques including LIME or attention mechanisms can yield word importance scores. These scores clarify a sentiment assessment by pinpointing the most influential words or phrases (e.g., explaining why a review is classified as ’positive’ through specific textual cues). In both instances, while these post-hoc local explanations furnish valuable understanding of why an individual prediction was rendered, the overarching, sophisticated decision-making architecture of these high-performing deep learning models largely remains uninterpretable. The local explanations, by concentrating on input feature attributions for specific outcomes, supply justifications but do not provide a holistic, global grasp of the model’s complete learned logic or its internal operational transformations. Consequently, although a globally interpretable model is inherently explainable at both local and global scales, a model that is merely explainable—as illustrated by standard DL applications to MNIST and IMDB where local explanations for distinct predictions can be generated—is not necessarily globally interpretable. This distinction holds significant importance. Depending on post-hoc local explainability for complex models is frequently a pragmatic approach; however, recognizing its constraints concerning genuine global model comprehension is essential. Progress in artificial intelligence demands advancements on both fronts: the creation of more intrinsically interpretable models for achieving global understanding, and the refinement of robust explainability techniques for trustworthy local justifications. Such dual development is paramount for cultivating accountable, beneficial, and reliable AI systems. References - [1] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, et al., “Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, pp. 82-115, Jun. 2020. - [2] F. Doshi-Velez and B. Kim, “Towards a rigorous science of interpretable machine learning,” arXiv preprint arXiv:1702.08608, 2017. - [3] Z. C. Lipton, “The mythos of model interpretability,” Queue, vol. 16, no. 3, pp. 31-57, May-Jun. 2018. - [4] C. Molnar, Interpretable machine learning: a guide for making black box models explainable. Lulu.com, 2020. [Online]. Available: https://christophm.github.io/interpretable-ml-book/ - [5] M. T. Ribeiro, S. Singh, and C. Guestrin, “"Why should I trust you?": explaining the predictions of any classifier,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA, Aug. 2016, pp. 1135-1144. - [6] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, USA, Dec. 2017, pp. 4765-4774. - [7] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: visual explanations from deep networks via gradient-based localization,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, Oct. 2017, pp. 618-626.