# Clarifying Model Transparency: Interpretability versus Explainability in Deep Learning with MNIST and IMDB Examples
**Authors**: Mitali Raj
> School of Cyber Security and Digital ForensicsNational Forensic Sciences UniversityNew Delhi, Indiamitali.btmtcs22215@nfsu.ac.in
Abstract
The impressive capabilities of deep learning models are often counterbalanced by their inherent opacity, commonly termed the "black box" problem, which impedes their widespread acceptance in high-trust domains. In response, the intersecting disciplines of interpretability and explainability, collectively falling under the Explainable AI (XAI) umbrella, have become focal points of research. Although these terms are frequently used as synonyms, they carry distinct conceptual weights. This document offers a comparative exploration of interpretability and explainability within the deep learning paradigm, carefully outlining their respective definitions, objectives, prevalent methodologies, and inherent difficulties. Through illustrative examinations of the MNIST digit classification task and IMDB sentiment analysis, we substantiate a key argument: interpretability generally pertains to a model’s inherent capacity for human comprehension of its operational mechanisms (global understanding), whereas explainability is more commonly associated with post-hoc techniques designed to illuminate the basis for a model’s individual predictions or behaviors (local explanations). For example, feature attribution methods can reveal why a specific MNIST image is recognized as a ’7’, and word-level importance can clarify an IMDB sentiment outcome. However, these local insights do not render the complex underlying model globally transparent. A clear grasp of this differentiation, as demonstrated by these standard datasets, is vital for fostering dependable and sound artificial intelligence.
Index Terms: Deep Learning, Explainable AI (XAI), Model Interpretability, Prediction Explainability, Black Box Systems, Trustworthy AI, MNIST, IMDB, Case Study Analysis, Local Explanations, Global Understanding. publicationid: pubid: XXX-X-XXXX-XXXX-X/XX/$31.00 © 2026 IEEE
I Introduction
Deep learning (DL) has instigated transformative progress across numerous domains, such as computer vision and natural language processing, frequently demonstrating performance levels that surpass human benchmarks. Nevertheless, the sophisticated, non-linear architecture of DL models, which can encompass millions of adjustable parameters, frequently obscures their internal decision-making pathways. This "black box" attribute fuels apprehensions regarding reliability, accountability, equity, and security, particularly when deployed in critical applications.
To allay such concerns, significant research efforts have been directed towards enhancing the transparency of DL models. Central to this endeavor are two pivotal concepts: interpretability and explainability. Despite their common interchangeable usage, a more detailed examination uncovers crucial distinctions. This paper endeavors to analyze and contrast these notions, anchoring the discussion with tangible examples derived from established DL applications: image recognition on the MNIST dataset and text-based sentiment classification using the IMDB dataset. These case studies will effectively illustrate how specific prediction explanations can be achieved even when a model’s overall global interpretability remains limited.
II Defining Interpretability and Explainability
No universally endorsed definitions exist for interpretability or explainability, which has led to a degree of terminological fluidity in academic discourse. Nonetheless, practical definitions can be formulated based on prevalent usage and influential research.
II-A Interpretability
- Definition: The notion of interpretability centers on the extent to which a human observer can grasp the causal relationships or the underlying functional principles governing a model’s decision-making (Lipton, 2018 [3]; Doshi-Velez & Kim, 2017 [2]). The emphasis is on comprehending how the model functions at a systemic level.
- Focus: Intrinsic characteristics of the model, architectural transparency, global understanding.
- Goal: The construction of models that are fundamentally comprehensible to humans, whether in their complete form (global interpretability) or regarding their individual components.
- Example Question: “By what general process does this Convolutional Neural Network (CNN) convert raw pixel data, through its successive layers, into a final digit classification?”
II-B Explainability (within XAI)
- Definition: Explainability pertains to the capacity to furnish an interface or supplementary data that elucidates a model’s operational behavior, especially its individual predictions, in a manner that is intelligible to humans (Arrieta et al., 2020 [1]). This often entails the application of post-hoc methods to pre-trained models.
- Focus: Justification of model outputs after the fact, understanding at the level of individual predictions, local explanations.
- Goal: The generation of clear rationales for specific instances or for the model’s general behavioral patterns, frequently without needing to make the model’s entire internal architecture transparent.
- Example Question: “What factors led the model to classify this particular MNIST image as a ’3’ rather than an ’8’?” or “Which specific words in this IMDB review instance were most decisive for its ’positive’ sentiment label?”
III Comparative Analysis
The principal distinctions between interpretability and explainability are outlined in Table I.
TABLE I: Distinguishing Features of Interpretability and Explainability (XAI)
| Core Objective | Comprehend the model’s internal logic/system (Global Understanding). | Understand the rationale for a specific decision/prediction (Local Explanation). |
| --- | --- | --- |
| Methodological Timing | Primarily intrinsic to model design (ante-hoc). | Frequently extrinsic, applied post-training (post-hoc). |
| Preferred Model Type | Leans towards simpler, inherently transparent architectures. | Applicable to complex, opaque models (typical of DL). |
| Reach of Understanding | Can be global (entire model) or modular (components). | Often instance-specific (local predictions), though global approximations are possible. |
| Role of Human User | Human can directly understand the model’s functioning. | Human receives an explanation produced by an auxiliary process. |
| Exemplary Techniques | Simpler models (e.g., linear regression, decision trees), rule induction, concept-based systems (e.g., ProtoPNet). | Feature attribution maps (e.g., Grad-CAM), LIME, SHAP, counterfactual reasoning, attention visualization. |
| Interrelation | An interpretable model is by nature explainable. | An explainable model is not necessarily interpretable. |
IV Illustrative Case Studies: Anchoring Explainability Concepts
To provide a concrete illustration of this distinction—especially the point that explainability (yielding local explanations) can be attained for opaque models lacking comprehensive interpretability (global understanding)—we investigate two prevalent deep learning scenarios.
IV-A Case Study 1: MNIST Handwritten Digit Recognition
<details>
<summary>figure1.png Details</summary>
### Visual Description
\n
## Heatmap: SHAP Value Explanation for Digit '7'
### Overview
The image presents a visualization of SHAP (SHapley Additive exPlanations) values for a digit '7' image. It consists of the digit '7' itself displayed as a grayscale pixelated image, and a corresponding heatmap showing the contribution of each pixel to the model's prediction. A colorbar at the bottom indicates the SHAP value scale.
### Components/Axes
* **Left Panel:** Grayscale image of the digit '7' on a black background.
* **Right Panel:** Heatmap representing SHAP values for each pixel in the '7' image. The heatmap is a grid of colored squares, with color intensity indicating the magnitude and direction of the SHAP value.
* **Colorbar:** Located at the bottom of the image. It ranges from blue (-6e-9) to red (6e-9), with white representing a value of 0. The label on the colorbar is "SHAP value".
### Detailed Analysis
The heatmap shows a complex pattern of positive (red) and negative (blue) SHAP values. The '7' shape is clearly visible in the heatmap, with the areas corresponding to the stroke of the '7' exhibiting the strongest SHAP values.
* **Positive SHAP Values (Red):** The upper-left and lower-right portions of the '7' stroke show strong positive SHAP values, indicating that these pixels contribute positively to the model's prediction of '7'. The values appear to peak around +4e-9 to +6e-9.
* **Negative SHAP Values (Blue):** The background areas and some parts of the '7' stroke show negative SHAP values, indicating that these pixels contribute negatively to the model's prediction of '7'. The values appear to peak around -4e-9 to -6e-9.
* **Zero SHAP Values (White/Gray):** Areas with SHAP values close to zero have minimal impact on the prediction.
The pixel values are not directly quantifiable from the image, but the color intensity provides a relative measure of their contribution.
### Key Observations
* The heatmap clearly highlights the pixels that are most important for the model's classification of the image as a '7'.
* The areas of the '7' that are most visually salient (the stroke) have the largest SHAP values, both positive and negative.
* The background pixels generally have smaller SHAP values, indicating they are less important for the prediction.
* There is a complex interplay of positive and negative contributions within the '7' shape itself, suggesting that the model is not simply relying on the presence of pixels but also on their specific arrangement.
### Interpretation
This visualization demonstrates how a machine learning model arrives at its prediction for the digit '7'. The SHAP values quantify the contribution of each pixel to the final prediction. The heatmap reveals that the model focuses on the stroke of the '7', with certain pixels within the stroke increasing the confidence in the '7' classification (positive SHAP values) and others potentially decreasing it (negative SHAP values). This could be due to noise or variations in the digit's style.
The color distribution suggests that the model is sensitive to the specific shape and location of the pixels within the '7'. The SHAP values provide insights into the model's decision-making process, allowing us to understand which features (pixels) are most influential in its predictions. This is valuable for debugging the model, identifying potential biases, and improving its robustness. The values are very small (on the order of 1e-9), which is typical for SHAP values and indicates the incremental contribution of each feature.
</details>
Figure 1: SHAP-based local explanation of a CNN’s prediction for an MNIST digit classified as ’7’. Left: Original input image. Right: SHAP heatmap indicating pixel-level feature contributions; red areas contribute positively to the prediction ’7’, while blue areas contribute negatively.
The MNIST dataset consists of 70,000 grayscale images depicting handwritten digits (0-9). CNNs represent the standard, high-efficacy models for this recognition task.
- The Opaque Nature of the Model: A standard CNN for MNIST typically incorporates several convolutional and pooling layers, followed by fully-connected layers, amounting to potentially millions of parameters. Although the architectural blueprint is human-designed, deciphering precisely how a particular configuration of learned filter weights across all layers culminates in a digit classification presents a formidable challenge. The model’s global operational mechanics remain largely obscure, precluding an easy articulation of its learned comprehensive rule-set for achieving global understanding.
- Attaining Local Explainability (Post-hoc): Notwithstanding the CNN’s opacity, explanations for individual predictions are obtainable.
- Feature Attribution Maps (e.g., Grad-CAM [7], SHAP): For an image classified as a ’7’, such methods can produce a visual heatmap or assign scores that pinpoint the input image pixels most critical to this ’7’ determination (e.g., the horizontal bar and diagonal stroke characteristic of a ’7’). This offers a localized, post-hoc rationale: “The model designated this as ’7’ due to its focus on these ’7’-indicative pixel areas.” (See Fig. 1).
- LIME [5]: LIME can locally approximate the CNN’s decision boundary for a given image by introducing perturbations (like toggling superpixels) and training a simpler, understandable model (such as a sparse linear regressor) on these variations. This would identify which superpixels steered the decision towards ’7’.
- Takeaway from MNIST Analysis: XAI tools empower us to effectively generate a local explanation for why a specific image is assigned a particular digit label by highlighting crucial input features. Nevertheless, these explanations do not transform the entire CNN into a globally interpretable system. We gain insight into the ’why’ for an individual instance, but not necessarily the complete ’how’ of the model’s overarching learned function.
IV-B Case Study 2: IMDB Movie Review Sentiment Analysis
<details>
<summary>figure2.png Details</summary>
### Visual Description
\n
## Horizontal Bar Chart: LIME Explanation for 'positive' prediction (Instance 5)
### Overview
The image displays a horizontal bar chart generated by LIME (Local Interpretable Model-agnostic Explanations) to explain a 'positive' prediction for instance 5. The chart shows the contribution of different words to the prediction, with bars extending to the left (negative values) and right (positive values) of the y-axis. The length of each bar represents the weight or importance of the corresponding word in the prediction.
### Components/Axes
* **Title:** "LIME Explanation for 'positive' prediction (Instance 5)" - positioned at the top-center.
* **X-axis:** Labeled with numerical values ranging from approximately -0.010 to 0.000. The axis represents the contribution score of each word.
* **Y-axis:** Lists the following words (from top to bottom):
* movie
* cheap
* on
* this
* least
* channel
* and
* i
* doesn
* see
* **Bars:** Horizontal red bars representing the contribution score of each word.
### Detailed Analysis
The bars are all red and extend either to the left or right of the y-axis. The trend is overwhelmingly negative, with most words contributing negatively to the 'positive' prediction.
Here's a breakdown of approximate values, reading from top to bottom:
* **movie:** Approximately -0.0095
* **cheap:** Approximately -0.0085
* **on:** Approximately -0.0080
* **this:** Approximately -0.0075
* **least:** Approximately -0.0070
* **channel:** Approximately -0.0065
* **and:** Approximately -0.0060
* **i:** Approximately -0.0055
* **doesn:** Approximately -0.0030
* **see:** Approximately -0.0005
### Key Observations
* All words have a negative contribution score, meaning they are pushing the prediction *away* from 'positive'.
* The word "movie" has the most negative contribution, followed by "cheap" and "on".
* The word "see" has the smallest negative contribution, closest to zero.
* The chart suggests that the presence of these words, despite the overall 'positive' prediction, is somewhat counterintuitive.
### Interpretation
This LIME explanation reveals that the model associates the instance 5 prediction with words that, in isolation, might suggest a negative sentiment or context. The fact that all words contribute negatively suggests that the model is relying on subtle cues or combinations of these words to arrive at the 'positive' prediction. It's possible that the model is picking up on sarcasm, irony, or a specific context where these words are used in a positive way.
The chart highlights the importance of understanding *why* a model makes a prediction, not just *what* the prediction is. It also demonstrates the potential for LIME to uncover unexpected or counterintuitive relationships between features and predictions. The negative contributions of these words could indicate a need for further investigation into the training data or model behavior. It is also possible that the model is not fully capturing the nuances of language.
</details>
Figure 2: LIME explanation for a positive sentiment prediction on an IMDB movie review. Although the predicted label is positive, the words shown (e.g., "cheap," "least," "channel") had negative contributions to the classification. The explanation helps interpret the model’s local reasoning for this instance.
The IMDB dataset comprises 50,000 movie reviews, each categorized as positive or negative. Models like RNNs (e.g., LSTMs) and Transformers (e.g., BERT) deliver leading performance on this task.
- The Opaque Nature of the Model: Architectures such as LSTMs or Transformers employ intricate sequential data processing and attention mechanisms on input word embeddings. Tracing the transformation of an input sentence’s semantic content through recurrent states or multiple self-attention layers to a final sentiment score is a complex endeavor. The learned representations are high-dimensional, and their precise semantic interpretations are not always readily apparent, making these models inherently uninterpretable for global understanding.
- Attaining Local Explainability (Post-hoc):
- Attention Visualization (Relevant for Transformers): When utilizing a Transformer, attention weights can reveal which words the model emphasized while processing others, or which words received the most attention from the final classification token. For a review classified as ’positive’, strong attention might fall on terms like “excellent,” “brilliant,” or “loved.”
- LIME [5] or SHAP [6]: These techniques can be adapted for text models, for example, by omitting words from a review and observing the consequent shift in prediction probability. They can then allocate importance scores to each word in a specific review, signifying which words were most instrumental in the ’positive’ or ’negative’ classification. An example explanation might be: “This review was classified as negative primarily due to the substantial negative influence of the words ’awful’ and ’waste’.”
- Takeaway from IMDB Analysis: Analogous to the MNIST scenario, XAI methods enable the generation of local explanations for the sentiment prediction of an individual review by identifying key influential textual elements. This yields valuable, localized comprehension. However, the comprehensive internal logic by which the LSTM or Transformer model processes language to discern sentiment remains multifaceted and largely uninterpretable from a global perspective.
V Synthesizing Insights from Case Studies
The analyses of MNIST and IMDB datasets furnish concrete evidence that:
- Deep learning models, despite their power, frequently function as opaque systems, rendering their global internal processes challenging to interpret for holistic understanding.
- Post-hoc XAI methodologies can deliver valuable local explanations for specific predictions generated by these opaque models. Pixel attribution maps for MNIST images and word-level importance scores for IMDB reviews serve as excellent examples.
- These local explanations address the “why this particular prediction?” query, thereby enhancing localized trust and comprehension.
- Nonetheless, the provision of such local explanations does not inherently render the entire model globally interpretable. The sophisticated underlying mechanisms often remain inscrutable.
- Consequently, a model that is explainable (i.e., for which local justifications for outputs can be produced) is not axiomatically interpretable (i.e., one whose fundamental operations are transparent for global understanding).
VI General Techniques in Deep Learning
The field offers a spectrum of methods, extending beyond those highlighted in our case studies.
VI-A Interpretability-focused Techniques (Often "Interpretable by Design" for Global Understanding)
- Structured Attention Mechanisms: Designing models where attention is a constrained, core element can foster interpretability by forcing explicit focus indication.
- Concept-based Architectures (e.g., TCAV): These aim to link model decisions to high-level, human-intelligible concepts.
- Generalized Additive Models (GAMs) / Neural Additive Models (NAMs): These learn a sum of non-linear transformations of individual input features.
- ProtoPNet (Prototypical Part Network): Predictions are made by comparing input segments to learned class-specific prototypes.
- Rule Extraction from Neural Networks: This involves attempts to translate the learned knowledge of a neural network into a set of symbolic rules.
VI-B Explainability-focused Techniques (Often Post-hoc for Local Explanations)
- Saliency/Feature Attribution Maps (as in MNIST study):
- Gradient-derived: Methods like Vanilla Gradients, Integrated Gradients, Grad-CAM.
- Perturbation-based: Techniques such as occlusion sensitivity, or certain SHAP algorithm variants.
- LIME (Local Interpretable Model-agnostic Explanations) (relevant to MNIST & IMDB studies): This approach locally trains a simpler, interpretable surrogate model.
- SHAP (SHapley Additive exPlanations) (relevant to IMDB & MNIST studies): A game-theoretic method for assessing feature importance.
- Counterfactual Explanations: These identify the minimal input modifications required to change a prediction.
- Influence Functions: These assess the impact of individual training samples on model parameters and predictions.
VII Importance and Applications
Both interpretability (which targets global understanding) and explainability (which provides local explanations) are vital for endeavors such as building trust, facilitating model debugging, ensuring fairness, meeting regulatory compliance, enabling scientific discovery, and guaranteeing safety in critical systems. The case studies underscore that even when complete global interpretability is unattainable, instance-specific local explainability offers significant value.
VIII Challenges and Limitations
- Performance vs. Transparency Trade-off: High-accuracy DL models (as used for MNIST/IMDB) frequently lack global interpretability. Simpler, more interpretable alternatives might compromise predictive power. Post-hoc local explanations are often approximations.
- Veracity of Explanations: Validating that local explanations genuinely mirror the model’s reasoning for a specific instance remains a significant hurdle.
- Human-Centricity of Explanations: The definition of an effective or “understandable” local explanation is inherently subjective and varies with context and user.
- Objective Evaluation: Quantifying the quality of global interpretability or a local explanation poses considerable difficulty.
- Resource Intensiveness: Many XAI techniques demand substantial computational resources.
- Breadth of Insight: Most XAI tools deliver local explanations. Achieving a reliable global understanding of entire DL models is a more elusive goal.
IX The Symbiotic Relationship
Although distinct, interpretability (global understanding) and explainability (local explanations) are not mutually exclusive concepts.
- Incorporating inherently interpretable modules can enhance the coherence and reliability of post-hoc local explanations.
- Local explanation methods can serve as auditing tools for models that are partially interpretable.
The overarching aim is to achieve an adequate level of understanding pertinent to the specific application. This might be realized through an intrinsically globally interpretable model or through an opaque model supplemented by robust local explanations, as demonstrated by our MNIST/IMDB scenarios where local explanations furnish key insights despite the models’ overall complexity.
X Future Directions
- Development of Inherently Interpretable Deep Learning Systems: Pursuing novel DL architectures that combine high performance with inherent transparency for global understanding.
- Advancement of Causal Explanations: Progressing from correlational feature importance in local explanations towards insights grounded in causal inference.
- Creation of Interactive and Personalized Explanation Interfaces: Designing tools that provide tailored local explanations responsive to user queries.
- Enhancing Robustness and Scrutiny of Explanations: Ensuring that local explanations are resilient to manipulation and are not deceptive.
- Establishment of Standardization and Benchmarking Protocols: Developing common evaluative frameworks for both types of methodologies.
XI Conclusion
Within the deep learning landscape, interpretability is primarily associated with the inherent comprehensibility of a model’s internal operations, striving for a global understanding of its systemic behavior. This is commonly pursued via architectural choices that promote transparency. Conversely, explainability (XAI) generally entails the use of post-hoc methodologies to offer human-intelligible rationales for specific model outputs, thereby delivering local explanations concerning the behavior of intricate, often opaque, systems.
Our investigations using the MNIST and IMDB datasets, coupled with the practical application of XAI utilities, serve to concretely delineate this difference. For a Convolutional Neural Network (CNN) tasked with MNIST digit classification, explainability techniques such as SHAP or other saliency approaches can produce pixel importance maps. These visualizations identify the pixels most critical to a particular digit’s classification (e.g., elucidating why a given image is identified as a ’7’ by highlighting its formative pixel configurations). In a similar vein, for models like LSTMs or Transformers engaged in IMDB movie review analysis, techniques including LIME or attention mechanisms can yield word importance scores. These scores clarify a sentiment assessment by pinpointing the most influential words or phrases (e.g., explaining why a review is classified as ’positive’ through specific textual cues).
In both instances, while these post-hoc local explanations furnish valuable understanding of why an individual prediction was rendered, the overarching, sophisticated decision-making architecture of these high-performing deep learning models largely remains uninterpretable. The local explanations, by concentrating on input feature attributions for specific outcomes, supply justifications but do not provide a holistic, global grasp of the model’s complete learned logic or its internal operational transformations.
Consequently, although a globally interpretable model is inherently explainable at both local and global scales, a model that is merely explainable—as illustrated by standard DL applications to MNIST and IMDB where local explanations for distinct predictions can be generated—is not necessarily globally interpretable. This distinction holds significant importance. Depending on post-hoc local explainability for complex models is frequently a pragmatic approach; however, recognizing its constraints concerning genuine global model comprehension is essential. Progress in artificial intelligence demands advancements on both fronts: the creation of more intrinsically interpretable models for achieving global understanding, and the refinement of robust explainability techniques for trustworthy local justifications. Such dual development is paramount for cultivating accountable, beneficial, and reliable AI systems.
References
- [1] A. B. Arrieta, N. DĂaz-RodrĂguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, et al., “Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, pp. 82-115, Jun. 2020.
- [2] F. Doshi-Velez and B. Kim, “Towards a rigorous science of interpretable machine learning,” arXiv preprint arXiv:1702.08608, 2017.
- [3] Z. C. Lipton, “The mythos of model interpretability,” Queue, vol. 16, no. 3, pp. 31-57, May-Jun. 2018.
- [4] C. Molnar, Interpretable machine learning: a guide for making black box models explainable. Lulu.com, 2020. [Online]. Available: https://christophm.github.io/interpretable-ml-book/
- [5] M. T. Ribeiro, S. Singh, and C. Guestrin, “"Why should I trust you?": explaining the predictions of any classifier,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA, Aug. 2016, pp. 1135-1144.
- [6] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, USA, Dec. 2017, pp. 4765-4774.
- [7] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: visual explanations from deep networks via gradient-based localization,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, Oct. 2017, pp. 618-626.