# Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformersâ â thanks: This work was supported by US NSF Grant IIS 1910131, US DoD and industry grants. We thank the members of the ALPS lab at UT Dallas for insightful discussions.
**Authors**: ParthPadalkar, GopalGupta
> The University of Texas at Dallas
Cambridge Author
18 2021 10.1017/xxxxx
## Abstract
Recent neuro-symbolic approaches have successfully extracted symbolic rule-sets from CNN-based models to enhance interpretability. However, applying similar techniques to Vision Transformers (ViTs) remains challenging due to their lack of modular concept detectors and reliance on global self-attention mechanisms. We propose a framework for symbolic rule extraction from ViTs by introducing a sparse concept layer inspired by Sparse Autoencoders (SAEs). This linear layer operates on attention-weighted patch representations and learns a disentangled, binarized representation in which individual neurons activate for high-level visual concepts. To encourage interpretability, we apply a combination of L1 sparsity, entropy minimization, and supervised contrastive loss. These binarized concept activations are used as input to the FOLD-SE-M algorithm, which generates a rule-set in the form of logic programs. Our method achieves a 5.14% better classification accuracy than the standard ViT while enabling symbolic reasoning. Crucially, the extracted rule-set is not merely post-hoc but acts as a logic-based decision layer that operates directly on the sparse concept representations. The resulting programs are concise and semantically meaningful. This work is the first to extract executable logic programs from ViTs using sparse symbolic representations. It bridges the gap between transformer-based vision models and symbolic logic programming, providing a step forward in interpretable and verifiable neuro-symbolic AI.
keywords: Neuro-symbolic AI, Mechanistic Interpretability, Vision Transformers, Explainable AI (XAI), Rule-Based Machine Learning, Sparse Autoencoders
## 1 Introduction
Extracting logic-based rules from neural models has emerged as a central objective in neuro-symbolic AI, driven by the growing demand for interpretability and verifiability in machine learning. As deep learning models continue to scale, they are increasingly deployed in critical applications such as autonomous driving (Kanagaraj et al., (2021)), medical diagnosis (Sun et al., (2016)), and natural disaster prevention (Ko and Kwak, (2012)). In these sensitive domains, incorrect predictions can carry severe consequences, emphasizing the necessity for transparency in decision-making. Many of these applications depend significantly on accurate image classification models, particularly Convolutional Neural Networks (CNNs). Recent frameworks such as NeSyFOLD (Padalkar et al., 2024a ; Padalkar et al., 2024b ) and ERIC (Townsend et al., (2021)) have successfully demonstrated the extraction of human-interpretable symbolic rule sets from CNNs, providing insights into the underlying reasoning of predictions in vision tasks.
Most of this progress, however, has been concentrated on CNNs. CNNs are composed of filters, that are trainable matrices that learn to detect patterns in local regions of images (e.g., a bed in an image classified as a bedroom). Their modular architecture, where individual filters often correspond to distinct visual concepts, makes them particularly suitable for rule extraction. By binarizing the activations of the final layer and using them as input to rule-based machine learning algorithms such as decision trees or FOLD-SE-M (Wang and Gupta, (2024)), it is possible to extract the symbolic rule-sets.
In contrast, Vision Transformers (ViTs) (Dosovitskiy et al., (2021)) have remained largely inaccessible to symbolic extraction techniques. While ViTs now dominate the field of vision due to their superior performance and flexibility, their reliance on global self-attention and lack of explicit concept detectors pose a significant challenge for rule-based interpretability. ViTs encode information in a distributed and entangled manner, making it unclear how to localize or discretize concept-level representations.
In this work, we take a step toward bridging this gap by introducing a frameworkâNeSyViTâfor extracting logic-based rule sets from Vision Transformers (ViTs). We modify the standard ViT architecture by replacing the final fully connected classification head with a single linear layerâ sparse concept layer âtrained to produce binarized outputs. The goal is to encourage each neuron in this final layer to correspond to a few distinct high-level concepts. To achieve this, we draw inspiration from Sparse Autoencoders (SAEs) (Ng et al., (2011)), incorporating an L1 sparsity loss to ensure that only a small subset of neurons activates for any given image.
To encourage binarization, we apply a sigmoid activation to the linear outputs, constraining them to the $[0,1]$ range, and introduce an entropy-based loss that pushes activations toward binary extremes (0 or 1). This design enables us, after training, to extract a binary vector for each image that reflects concept-level activations. These vectors are then passed to the FOLD-SE-M algorithm to generate a symbolic rule-set in the form of a stratified Answer Set Program for classification. It is crucial that the representations corresponding to images from the same class are well-clustered in the latent space. This promotes the formation of clear decision boundaries that can be effectively exploited by FOLD-SE-M. To enforce this structure, we incorporate a supervised contrastive loss (SupCon) (Khosla et al., (2020)), which encourages representations of images with the same label to lie closer together while pushing apart those from different classes. This facilitates learning highly accurate rules.
Finally, each predicate in the extracted rule set is âsemantically labelledâ based on the concept(s) that its corresponding neuron represents. Padalkar et al. introduced an algorithm for automatic semantic labelling of the predicates in the rule-sets extracted from a CNN using FOLD-SE-M (Padalkar et al., 2024a ). We adopt the same algorithm for semantic labelling of the rule-sets we generate. Figure 1 illustrates the NeSyViT framework.
<details>
<summary>extracted/6428030/Images/NeSyViT2.png Details</summary>
### Visual Description
## Diagram: Vision Transformer (ViT) to Interpretable Rule Generation Pipeline
### Overview
The image is a technical flowchart illustrating a machine learning pipeline that transforms an input image into human-interpretable logical rules. The process uses a Vision Transformer (ViT) to extract features, binarizes neuron activations, generates raw rules via an algorithm called "FOLD-SE-M," and finally applies "Semantic Labelling" to produce a human-readable rule set. The diagram is divided into three main horizontal sections: the input and feature extraction (left), the binarization and rule generation (top-right), and the two resulting rule sets (bottom).
### Components/Axes
The diagram contains the following labeled components and text elements, organized by spatial region:
**1. Input & Feature Extraction (Left Side):**
* **Image:** A photograph of a bedroom interior (bed, nightstand, chandelier).
* **Label:** `INPUT TO ViT` (with an arrow pointing right).
* **Model Block:** An orange trapezoid labeled `ViT`.
* **Output Token:** A vertical stack of orange rectangles labeled `[CLS]`.
* **Layer Label:** `Sparse Concept Layer` (below the neuron circles).
* **Neuron Circles:** A vertical column of orange circles labeled `n1`, `n2`, `n3`, `n4`, `...`, `nd`.
* **Activation Values:** Arrows from each neuron to a numerical value: `n1 â 0.8`, `n2 â 0.1`, `n3 â 0.0`, `n4 â 0.9`, `...`, `nd â 0.2`.
**2. Binarization & Rule Generation (Top-Right):**
* **Process Label:** `BINARIZATION OF NEURON OUTPUTS` (with an arrow pointing right).
* **Binarization Table:** A grid with an orange header row (`n1`, `n2`, `n3`, `...`, `nd`) and five data rows containing binary values (0s and 1s).
* Row 1: `0`, `0`, `1`, `...`, `1`
* Row 2: `1`, `0`, `1`, `...`, `1`
* Row 3: `1`, `1`, `0`, `...`, `1`
* Row 4: `...`, `...`, `...`, `...`, `...`
* Row 5: `1`, `0`, `0`, `...`, `0`
* **Algorithm Label:** `FOLD-SE-M RULE GENERATION` (with a downward arrow pointing to the Raw Rule-Set).
**3. Rule Sets (Bottom):**
* **Raw Rule-Set Box (Right):** A beige rounded rectangle titled `RAW RULE-SET`. It contains three numbered logical rules in a Prolog-like syntax. Key elements are color-coded (pink for `n5`, `n2`, `n41`; blue for `n1`).
1. `target(X, 'bedroom') :- n5(X), n1(X), not ab1(X).`
2. `target(X, 'kitchen') :- n2(X), not ab2(X).`
3. `ab1(X) :- n41(X), not ab2(X).`
4. `...`
* **Semantic Labelling Arrow:** A leftward arrow labeled `SEMANTIC LABELLING` connecting the Raw Rule-Set to the Labelled Rule-Set.
* **Labelled Rule-Set Box (Left):** A beige rounded rectangle titled `LABELLED RULE-SET`. It contains the semantically labeled version of the rules. Key elements are color-coded (red for `bed1`, `cabinet1`; blue for `pillow1`; purple for `sink1`).
1. `target(X, 'bedroom') :- bed1(X), pillow1(X), not ab1(X).`
2. `target(X, 'kitchen') :- cabinet1(X), not ab2(X).`
3. `ab1(X) :- sink1(X), not ab2(X).`
4. `...`
### Content Details
The pipeline details are as follows:
1. **Input Processing:** An image of a bedroom is fed into a Vision Transformer (ViT). The ViT processes the image and outputs a `[CLS]` token representation.
2. **Sparse Concept Activation:** The `[CLS]` token activates a "Sparse Concept Layer" consisting of neurons `n1` through `nd`. The example shows specific activation strengths: `n1=0.8`, `n2=0.1`, `n3=0.0`, `n4=0.9`, `nd=0.2`.
3. **Binarization:** The continuous activation values are converted into a binary matrix. Each row in the table likely represents a different input sample or data point. The columns correspond to neurons `n1` to `nd`. A value of `1` indicates the neuron is active (above a threshold), and `0` indicates it is inactive. The first row shows neurons `n3` and `nd` are active for that sample.
4. **Rule Generation:** The "FOLD-SE-M" algorithm processes the binarized data to generate a `RAW RULE-SET`. The rules use the abstract neuron identifiers (`n5`, `n1`, `n2`, `n41`) as predicates. For example, Rule 1 states that for an input `X` to be classified as a 'bedroom', neurons `n5` and `n1` must be active, and the abnormality condition `ab1` must not hold.
5. **Semantic Labelling:** The abstract neuron identifiers in the raw rules are mapped to human-interpretable concept names (e.g., `n5` â `bed1`, `n1` â `pillow1`, `n2` â `cabinet1`, `n41` â `sink1`). This produces the final `LABELLED RULE-SET`. The rules now read as intuitive logical statements: e.g., a 'bedroom' is defined by the presence of a bed and a pillow, and the absence of a specific abnormality (`ab1`), which itself is defined by the presence of a sink.
### Key Observations
* **Color Coding:** Colors are used systematically to trace concepts. Orange represents the ViT and its neurons. In the rule sets, pink highlights raw neuron IDs, while red, blue, and purple highlight their corresponding semantic labels (`bed1`, `pillow1`, `sink1`).
* **Rule Structure:** The rules follow a Definite Clause Grammar (DCG) or Prolog-like syntax, using `:-` for implication and `not` for negation. They include both target classification rules and intermediate "abnormality" (`ab`) definition rules.
* **Sparsity:** The term "Sparse Concept Layer" and the binarization step suggest the system aims to identify a small, discrete set of meaningful features from the high-dimensional ViT output.
* **Transformation:** The core transformation is from subsymbolic, continuous neural activations (`0.8`, `0.1`) to symbolic, discrete logical rules (`bed1(X), pillow1(X)`).
### Interpretation
This diagram depicts a method for **extracting interpretable, logical knowledge from a black-box vision model (ViT)**. The pipeline bridges the gap between connectionist AI (neural networks) and symbolic AI (logic rules).
* **What it demonstrates:** It shows a concrete workflow for "opening the black box." Instead of just getting a classification ("bedroom"), the system produces an explicit, auditable reason: "because a bed and a pillow are present, and a sink is not present (which would indicate a kitchen)."
* **Relationship between elements:** The ViT acts as a powerful feature extractor. The Sparse Concept Layer and binarization act as a bottleneck to distill these features into discrete, switch-like concepts. The FOLD-SE-M algorithm then performs rule induction from these binary patterns. Finally, semantic labelling grounds these abstract patterns in human-understandable vocabulary, likely using an external knowledge base or embedding similarity.
* **Significance:** This approach addresses key criticisms of deep learningâlack of transparency and interpretability. The resulting rules can be inspected, verified, and potentially edited by humans. The inclusion of "abnormality" rules (`ab1`, `ab2`) suggests the system can also model and reason about exceptions or confounding factors. The pipeline is valuable for domains requiring explainable AI, such as medical diagnosis or autonomous systems, where understanding the "why" behind a decision is crucial.
</details>
Figure 1: The NeSyViT Framework
The final Ne uro- Sy mbolic (NeSy) model is a combination of the ViT based feature extractor that generates the binary vector for each image and the rule-set generated by FOLD-SE-M, which classifies the image into a particular class based on which neurons were activated/deactivated (1/0). We demonstrate through our experiments that the NeSy model produced by our framework outperforms the vanilla ViT in terms of classification accuracy, achieving an average improvement of 5.14%. Notably, this performance gain is achieved while also generating interpretable rule-setsâa significant result, as prior neuro-symbolic frameworks that extract rule-sets from neural models often suffer a drop in accuracy.
Our contributions are as follows:
1. We introduce a training method that combines supervised contrastive loss, L1 sparsity, and entropy loss to learn compact and binarized concept-level representations.
1. We propose an end-to-end neuro-symbolic framework, NeSyViT, for generating a rule-set from a modified ViT using FOLD-SE-M.
1. We show through experiments that our NeSyViT framework achieves classification accuracy better than the vanilla ViT, while also producing concise rule-sets
## 2 Background
### 2.1 Vision Transformers
The Vision Transformer (ViT) (Dosovitskiy et al., (2021)) is a deep learning architecture that applies the Transformer modelâoriginally developed for natural language processingâto image classification tasks. Unlike Convolutional Neural Networks (CNNs), which operate on local regions using convolutional filters, ViTs treat an image as a sequence of patches. An input image is divided into fixed-size non-overlapping patches (e.g., $16\times 16$ pixels), each of which is flattened and projected into a vector embedding. The vector embeddings of those patches then go through multiple self-attention layers followed by the final classification layer.
The Transformer uses self-attention layers to model relationships between all image patches simultaneously. At the core of this mechanism is the concept of an attention head, which determines how much attention each patch should pay to every other patch in the sequence. For each patch token, the model learns three distinct vectors: a query, a key, and a value. Attention scores are computed by taking the dot product between a patchâs query vector and the key vectors of all other patches, followed by a softmax normalization to produce attention weights. These weights are then used to compute a weighted sum of the value vectors, generating a context-aware representation for each token.
A special learnable token, known as the [CLS] token, is prepended to the patch sequence before being passed into the Transformer. Unlike the patch tokens that represent image regions, the [CLS] token acts as a summary placeholder. During self-attention, it aggregates information from all other patches by attending to them across multiple layers. After the final Transformer layer, the embedding corresponding to the [CLS] token is typically used for classification, as it encapsulates a global representation of the image.
While this architecture allows Vision Transformers to capture long-range dependencies and global structure, it also presents a major challenge for interpretability. In CNNs, individual filters often learn to detect localized visual patterns or objects (e.g., corners, textures, faces), enabling a degree of modularity and conceptual traceability. In contrast, attention heads are distributed, context-sensitive, and dynamically influenced by the full set of tokensâincluding the [CLS] token. This makes it difficult to associate specific heads or tokens with interpretable visual concepts. Consequently, extracting symbolic, modular representations from ViTs is significantly more challenging than from CNNs. Hence, in this work, we take the [CLS] vector and pass it through the sparse concept layer that is optimized to produce sparse outputs, allowing the information encoded in the [CLS] token to be disentangled into a small set of active neurons. Each neuron is encouraged to specialize and activate primarily for images of a specific class. This design is inspired by the core principles of Sparse Autoencoders (SAEs) (Ng et al., (2011)), which promote compact and interpretable representations through sparsity.
### 2.2 FOLD-SE-M
The FOLD-SE-M algorithm Wang and Gupta, (2024) that we employ in our framework, learns a rule-set from data as a default theory. Default logic is a non-monotonic logic used to formalize commonsense reasoning. A default $D$ is expressed as:
$$
D=\frac{A:\textbf{M}B}{\Gamma} \tag{1}
$$
Equation 1 states that the conclusion $\Gamma$ can be inferred if pre-requisite $A$ holds and $B$ is justified. $\textbf{M}B$ stands for âit is consistent to believe $B$ â. Normal logic programs can encode a default theory quite elegantly (Gelfond and Kahl, (2014)). A default of the form:
$$
\frac{\alpha_{1}\land\alpha_{2}\land\dots\land\alpha_{n}:\textbf{M}\lnot\beta_
{1},\textbf{M}\lnot\beta_{2}\dots\textbf{M}\lnot\beta_{m}}{\gamma}
$$
can be formalized as the normal logic programming rule:
$$
\gamma~{}\texttt{:-}~{}\alpha_{1},\alpha_{2},\dots,\alpha_{n},\texttt{not}~{}
\beta_{1},\texttt{not}~{}\beta_{2},\dots,\texttt{not}~{}\beta_{m}.
$$
where $\alpha$ âs and $\beta$ âs are positive predicates and not represents negation-as-failure. We call such rules default rules. Thus, the default
$$
\frac{bird(X):M\lnot penguin(X)}{flies(X)}
$$
will be represented as the following default rule in normal logic programming:
flies(X) :- bird(X), not penguin(X).
We call bird(X), the condition that allows us to jump to the default conclusion that X flies, the default part of the rule, and not penguin(X) the exception part of the rule.
FOLD-SE-M (Wang and Gupta, (2024)) is a Rule Based Machine Learning (RBML) algorithm. It generates a rule-set from tabular data, comprising rules in the form described above. The complete rule-set can be viewed as a stratified answer set program (a stratified ASP rule-set has no cycles through negation (Baral, (2003))). It uses special abx predicates to represent the exception part of a rule where x is a unique numerical identifier. FOLD-SE-M incrementally generates literals for default rules that cover positive examples while avoiding covering negative examples. It then swaps the positive and negative examples and calls itself recursively to learn exceptions to the default when there are still negative examples falsely covered.
There are $2$ tunable hyperparameters, $ratio$ , and $tail$ . The $ratio$ controls the upper bound on the number of false positives to the number of true positives implied by the default part of a rule. The $tail$ controls the limit of the minimum number of training examples a rule can cover.
## 3 Methodology
Our aim is to extract a symbolic rule-set from the ViT and use it for final classification. To this end, we first modify the architecture of a standard Vision Transformer by replacing the final classification headâplaced after the last self-attention layerâwith a single linear layer of dimension $D$ . The [CLS] tokenâs vector which has the aggregated global information feeds directly into this layer. This layer outputs a $D$ -dimensional vector, to which we apply a sigmoid function element-wise, constraining all values to lie in the range $[0,1]$ . Each value in this vector could then be interpreted as the activation of a distinct neuron.
There are three key objectives in designing this representation:
1. The model needs to produce similar binary vectors for images of the same class, and dissimilar ones for different classes, to allow the FOLD-SE-M to identify meaningful decision boundaries.
1. The output values had to be pushed as close as possible to either $0 0$ or $1$ , since after training, we binarize these vectors by rounding. Values near $0.5$ would introduce ambiguity and loss of information.
1. The output vectors need to be sparse, with most values being $0 0$ and only a few being $1$ . This sparsity ensures that rule-sets remain compact and interpretable, as fewer neurons would appear as predicates in the learned rules.
We enforce these three properties using a combination of supervised contrastive loss, entropy minimization, and L1 sparsity loss, described in detail below.
Supervised Contrastive Loss: To ensure that images from the same class produce similar sparse representations, we incorporate the Supervised Contrastive Loss (SupCon) (Khosla et al., (2020)). This loss encourages the $D$ -dimensional vectors corresponding to samples of the same class to cluster together in the latent space, while pushing apart those from different classes. Such class-wise clustering is essential for enabling a rule-based learner, such as FOLD-SE-M, to identify clean symbolic decision boundaries.
Let $\mathbf{z}_{i}\in\mathbb{R}^{D}$ denote the normalized sparse representation of sample $i$ in a batch $\mathcal{B}$ , and let $y_{i}$ be its corresponding class label. In contrastive learning, we refer to $\mathbf{z}_{i}$ as the anchor, and we compare it to the remaining samples in the batch. Those with the same label are treated as positives, while the rest are considered negatives.
The SupCon loss for a single anchor $i$ is defined as:
$$
\mathcal{L}_{\text{supcon}}^{i}=-\frac{1}{|P(i)|}\sum_{p\in P(i)}\log\frac{
\exp(\mathbf{z}_{i}\cdot\mathbf{z}_{p}/\tau)}{\sum_{a\in\mathcal{A}(i)}\exp(
\mathbf{z}_{i}\cdot\mathbf{z}_{a}/\tau)} \tag{2}
$$
where:
- $P(i)$ is the set of indices in the batch with the same label as $i$ (i.e., the positives for anchor $i$ ),
- $\mathcal{A}(i)$ is the set of all indices in the batch excluding $i$ itself,
- $\tau>0$ is a temperature parameter that controls the sharpness of the distribution.
The total loss is averaged over all anchors in the batch:
$$
\mathcal{L}_{\text{supcon}}=\frac{1}{N}\sum_{i\in\mathcal{B}}\mathcal{L}_{
\text{supcon}}^{i} \tag{3}
$$
where $N$ is the number of images in the batch $\mathcal{B}$ . By minimizing this loss, the model is encouraged to produce sparse vectors that are tightly clustered for images of the same class and distinct from those of other classes. This structure makes the representations well-suited for interpretable rule extraction.
Entropy Minimization: To encourage the sparse representations to become binarizedâi.e., close to either 0 or 1âwe apply an entropy minimization loss on the output of the sigmoid-activated linear layer. Since this linear layer uses a sigmoid function, each neuronâs activation lies in the range $[0,1]$ . Ideally, we want these activations to converge toward discrete values ( $0 0$ or $1$ ) to allow lossless binarization post-training.
We treat each neuronâs output as a Bernoulli random variable and compute its entropy using the standard binary entropy formula. Given a batch of $N$ images where $\mathbf{z}_{i}\in\mathbb{R}^{D}$ is the $D$ -dimensional output for image $i$ , the entropy loss is computed as:
$$
\mathcal{L}_{\text{entropy}}=-\frac{1}{ND}\sum_{i=1}^{N}\sum_{j=1}^{D}\left[z_
{i,j}\log(z_{i,j}+\epsilon)+(1-z_{i,j})\log(1-z_{i,j}+\epsilon)\right], \tag{4}
$$
where $z_{i,j}$ is the activation of neuron $j$ for image $i$ , and $\epsilon$ is a small constant added for numerical stability.
Minimizing this entropy term encourages each activation to move closer to either $0 0$ or $1$ , making the final binarization step more reliable. This is especially important for downstream symbolic rule extraction, where crisp binary features are essential for learning accurate and interpretable logic programs.
L1 Sparsity Loss: To promote interpretability and ensure that only a few neurons activate for each image, we incorporate an L1 sparsity loss on the output of the final linear layer. Sparse representations are crucial for generating concise rule-sets, as they limit the number of active predicates per image, reducing the complexity of the learned logic programs.
Formally, given a batch of $N$ vectors ${\mathbf{z}_{1},\dots,\mathbf{z}_{N}}$ , where $\mathbf{z}_{i}\in\mathbb{R}^{D}$ is the sigmoid-activated output for image $i$ , the L1 sparsity loss is defined as:
$$
\mathcal{L}_{\text{sparsity}}=\frac{1}{ND}\sum_{i=1}^{N}\sum_{j=1}^{D}|z_{i,j}| \tag{5}
$$
This loss penalizes large activation values and encourages most neurons to remain close to zero, allowing only a few dimensions to be active for any given input.
The total loss then becomes:
$$
\mathcal{L}_{Total}=\alpha\mathcal{L}_{supcon}+\beta\mathcal{L}_{entropy}+
\gamma\mathcal{L}_{sparsity} \tag{6}
$$
where $\alpha$ , $\beta$ and $\gamma$ are weights to control the impact of each loss term. Finally, after training the model using the combined loss $\mathcal{L}_{\text{Total}}$ , we collect the sparse representation vectors for all images in the training set. These vectors are binarizedâeach value thresholded at $0.5$ âto form a binary table, where each row corresponds to an image and each column to a neuron in the final linear layer. This binarization table is used as input to the FOLD-SE-M algorithm, which generates a symbolic rule-set in the form of a stratified Answer Set Program. Each predicate in the resulting rule-set corresponds directly to a neuron in the final layer, enabling symbolic reasoning over the learned representations.
Semantic Labelling Algorithm: To assign human-understandable meanings to these predicates, we employ the semantic labeling algorithm introduced by Padalkar et al., 2024a . For each neuron, we identify the top- $10$ images that activate it most strongly. We then compute attention-based heatmaps for these images to localize the spatial regions that the neuron focuses on. These heatmaps are intersected with the corresponding (pre-annotated) semantic segmentation masksâwhere image regions are labeled with semantic object categories. For each neuron, we compute the average Intersection-over-Union (IoU) between its heatmaps and each available concept label. The label with the highest average IoU is then assigned to the predicate corresponding to that neuron, thus grounding the symbolic rule-set in interpretable visual concepts.
After the rule-set is generated, new images can be classified by passing them through the modified ViT, which outputs a $D$ -dimensional vector. This vector is then binarized by thresholding each value at $0.5$ . The resulting binary vector serves as input to the FOLD-SE-M toolkitâs rule interpreter, which evaluates the symbolic rule-set in a top-down manner. For each rule, the truth values of the predicates in the body are determined based on the corresponding neuron activations in the binarized vector. The first rule that fires determines the predicted class of the image. Moreover, the s(CASP) (Arias et al., (2018)) goal-directed ASP interpreter can be used to obtain a justification of the prediction.
## 4 Experiments
We conducted experiments to address the following research questions:
Q1: How does the classification accuracy of the neuro-symbolic model produced by the NeSyViT framework compare to that of a standard vanilla Vision Transformer (ViT)?
Q2: What is the typical size of the rule-set generated by the NeSyViT framework, and how compact are the resulting rules?
Q3: How does our framework scale as the number of classes increases?
Q4: What do the neuron activation patterns look like for each class, and how well do they align with human-interpretable semantic concepts?
Q5: How well can the semantic labelling algorithm meant for CNN-based frameworks such as NeSyFOLD, be directly adopted for our NeSyViT framework?
[Q1, Q2, Q3] Classification Accuracy, Rule-set Size and Scalability: The central goal of the NeSyViT framework is to produce a neuro-symbolic model that maintains high classification accuracy while offering symbolic interpretability through a compact rule-set. Ideally, the accuracy of the interpretable model should match or exceed that of its purely neural counterpart (i.e., the vanilla ViT), and the resulting rule-set should remain as concise as possible. We use rule-set size as a proxy for interpretability, based on the findings of Lage et al. (Lage et al., (2019)), who demonstrated through human-subject evaluations that larger rule-sets are significantly more difficult to interpret. Thus, achieving high accuracy alongside a small rule-set is critical for balancing performance and human-understandability.
Setup: We use the ViT-Base architecture from the timm library, which processes input images of size $224\times 224$ using non-overlapping $16\times 16$ patches. The model consists of 12 Transformer blocks, each with 12 attention heads, and produces a [CLS] token embedding of dimension 768. As described earlier, we modify this architecture by removing the final classification head and replacing it with a single linear layer of output dimension 128. All experiments are conducted using this configuration with weights pretrained on ImageNet-1k to construct the NeSy model. The vanilla ViT refers to the same base architecture without any modifications. We used the AdamW optimizer along with a cosine annealing learning rate scheduler. The model was trained for $50$ epochs using the combined loss consisting of supervised contrastive loss, entropy minimization, and L1 sparsity, as previously described. For rule-set generation, we employed the FOLD-SE-M algorithm. The weights assigned to each loss component, as well as additional training and architectural hyperparameters, are provided in the Appendix.
We evaluated both our NeSy model and the vanilla ViT on subsets of two benchmark datasets (1): the Places dataset (Zhou and Others, (2018)), which contains images of various indoor and outdoor scenes, and the German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp and Others, (2012)), which consists of images of traffic signposts. From the Places dataset, we constructed multiple class subsets of increasing number of classes to gauge the scalability of NeSyViT: two classes (P2), three classes (P3.1), five classes (P5), and ten classes (P10). The P2 subset includes bathroom and bedroom images. P3.1 extends this by adding kitchen, while P5 further incorporates dining room and living room. P10 adds five additional classes: home office, office, waiting room, conference room, and hotel room. Additionally, we include two alternative three-class subsets: P3.2 with desert road, forest road, and street, and P3.3 with desert road, driveway, and highway. For each class, we sampled $5,000$ images, creating a $4$ k/ $1$ k train-test split, and used the official validation set without modification.
The GTSRB (GT43) dataset contains $43$ traffic sign classes. We used the official test set of $12.6k$ images and performed an $80/20$ train-validation split on the remaining data, resulting in approximately $21k$ training and $5k$ validation images.
We run each experiment $5$ times with random train-test splits and then report the average metrics in Table 1. We closely follow the experimental setup proposed by Padalkar et al.(Padalkar et al., 2024a ) for evaluating the NeSyFOLD framework, which uses FOLD-SE-M to extract rule-sets from CNNs. Specifically, we compare the drop (or gain) in accuracy observed while using NeSyFOLD as compared to the vanilla CNN, against the corresponding change in accuracy when using NeSyViT as compared to the Vanilla ViT. This relative comparison highlights the effectiveness of our approach and is summarized in Table 2.
| P2 Vanilla P3.1 | NeSyViT $99\pm 0.00$ NeSyViT | $\textbf{100}\pm\textbf{0.00}$ - $\textbf{99}\pm\textbf{0.00}$ | $2\pm 0.00$ - $3\pm 0.00$ | $2\pm 0.00$ - $3\pm 0.00$ | $2\pm 0.00$ $3\pm 0.00$ |
| --- | --- | --- | --- | --- | --- |
| Vanilla | $98\pm 0.00$ | - | - | - | |
| P3.2 | NeSyViT | $\textbf{99}\pm\textbf{0.00}$ | $3\pm 0.00$ | $3\pm 0.40$ | $3\pm 0.00$ |
| Vanilla | $97\pm 0.00$ | - | - | - | |
| P3.3 | NeSyViT | $\textbf{98}\pm\textbf{0.00}$ | $5\pm 0.40$ | $3\pm 0.49$ | $5\pm 0.40$ |
| Vanilla | $91\pm 0.00$ | - | - | - | |
| P5 | NeSyViT | $\textbf{98}\pm\textbf{0.00}$ | $5\pm 0.00$ | $5\pm 0.40$ | $5\pm 0.80$ |
| Vanilla | $90\pm 0.00$ | - | - | - | |
| P10 | NeSyViT | $\textbf{94}\pm\textbf{0.00}$ | $10\pm 0.49$ | $12\pm 1.47$ | $14\pm 1.67$ |
| Vanilla | $76\pm 0.00$ | - | - | - | |
| GT43 | NeSyViT | $98\pm 0.00$ | $43\pm 0.00$ | $44\pm 1.90$ | $51\pm 3.12$ |
| Vanilla | $\textbf{99}\pm\textbf{0.00}$ | - | - | - | |
| Mean Stats | NeSyViT | $\textbf{98}\pm\textbf{0.00}$ | $10.14\pm 0.13$ | $10.30\pm 0.67$ | $11.85\pm 0.86$ |
| Vanilla | $92.86\pm 0.00$ | - | - | - | |
Table 1: Comparison between the NeSyViT and the Vanilla ViT. Bold values are better.
| P2 | $-4$ | +1 | $12$ | 2 |
| --- | --- | --- | --- | --- |
| P3.1 | $-7$ | +1 | $16$ | 3 |
| P3.2 | $-4$ | +2 | $7$ | 3 |
| P3.3 | $-7$ | +7 | $23$ | 5 |
| P5 | $-15$ | +8 | $30$ | 5 |
| P10 | $-21$ | +18 | $65$ | 14 |
| GT43 | $-13$ | -1 | $99$ | 51 |
| Mean Stats | $-10.14$ | +5.14 | $36$ | 11.85 |
Table 2: Comparison of relative % accuracy change w.r.t. the vanilla model and rule-set size between NeSyFOLD and NeSyViT.
Result: Table 1 presents a comparison between the NeSy model generated using NeSyViT and the Vanilla ViT. All metrics are reported over five independent runs. Accuracy is computed on the test set for both models. The âRulesâ column reports the average and standard deviation of the number of rules generated. âUnique Predicatesâ indicates the number of distinct predicates used across the rule-set, and âSizeâ refers to the total number of predicate occurrences in the bodies of all rules. Recall that we use rule-set size as a proxy for interpretabilityâsmaller rule-sets are generally easier to understand.
The most significant observation is that our method achieves higher average accuracy than the Vanilla ViT. This is particularly noteworthy, as extracting interpretable rule-sets by binarizing internal representations typically results in some loss of information and a subsequent drop in performance. However, our approach mitigates this issue through the use of an entropy loss, which encourages the outputs of the sparse concept layer to be close to either $0 0$ or $1$ during training. As a result, thresholding these outputs at $0.5$ post-training introduces minimal distortion. Furthermore, the supervised contrastive loss ensures that the resulting binary vectors are well-clustered by class, enabling FOLD-SE-M to learn accurate and class-discriminative rule-sets. Finally, the L1 sparsity loss encourages only a small subset of neurons to activate for each image, resulting in compact and interpretable rule-sets. Notice that for the P10 dataset, the accuracy improvement over vanilla is 18%, which is a huge improvement and shows the merit of our approach when scaling to larger number of classes.
To provide context for the significance of our results, Table 2 presents the percentage change in accuracy of the NeSy model relative to its vanilla counterpart for both NeSyViT and NeSyFOLD (which uses a CNN instead of a ViT). We also report the average rule-set size generated by each framework. Notably, NeSyViT improves upon the accuracy of the Vanilla ViT by an average of 5.14% âa substantial gain, especially when contrasted with NeSyFOLD, which suffers an average accuracy drop of 10.14% compared to the vanilla CNN. Additionally, NeSyFOLD shows a consistent degradation in performance as the number of classes increases, largely due to the information loss introduced by post-hoc binarization. In contrast, our method consistently yields accuracy improvements across datasets, with the sole exception of GT43, where we observe a minor drop of 1 percentage point.
Furthermore, the rule-sets generated by NeSyViT are significantly more compact, with an average size of $11.85$ , which is 67% smaller than those produced by NeSyFOLDâfurther emphasizing the interpretability of our approach.
<details>
<summary>x1.png Details</summary>
### Visual Description
## [Composite Image]: Heatmap Grid with Rule Sets
### Overview
The image is a composite figure containing a grid of nine indoor scene photographs, each overlaid with a colored heatmap (likely representing attention, activation, or saliency from a computer vision model). Below the grid are two text boxes containing logical rule sets. The overall context appears to be from a technical paper or presentation on scene classification or visual reasoning, where heatmaps indicate model focus and rules define classification logic.
### Components/Axes
1. **Heatmap Grid (3x3):**
* **Structure:** Nine individual images arranged in three rows and three columns.
* **Image Identifiers:** To the left of each row, a numeric identifier followed by "(X)" is displayed:
* Top row: `106 (X)`
* Middle row: `43 (X)`
* Bottom row: `105 (X)`
* **Heatmap Overlay:** Each photograph has a semi-transparent color overlay ranging from blue (cool/low) to red (hot/high), indicating regions of interest or high activation.
* **Scene Content:** All images depict indoor domestic environments, primarily kitchens and what appear to be bedrooms or living areas.
2. **Text Boxes (Bottom):**
* **Left Box (b):** Labeled `RAW RULE-SET P3.1:`. Contains three numbered rules in a Prolog-like syntax.
* **Right Box:** Labeled `LABELLED RULE-SET P3.1`. Contains a partially visible, semantically enriched version of the rules from the left box.
### Detailed Analysis
#### Heatmap Grid Content
* **Row 1 (Identifier: 106 (X)):**
* **Image 1 (Top-Left):** Bedroom scene with a bed, pillows, and a small table. Heatmap shows high activation (red/yellow) on the bedspread and pillows.
* **Image 2 (Top-Center):** Kitchen scene with cabinets and a countertop. Heatmap shows strong, scattered activation across the countertop and lower cabinets.
* **Image 3 (Top-Right):** Kitchen/dining area with a table and chairs. Heatmap shows broad, high activation across the table surface and surrounding area.
* **Row 2 (Identifier: 43 (X)):**
* **Image 1 (Middle-Left):** Kitchen corner with a refrigerator. Heatmap shows a very concentrated, high-intensity (red) circular activation on the refrigerator door.
* **Image 2 (Middle-Center):** Kitchen with white cabinets and a dishwasher. Heatmap shows moderate activation on the upper cabinets and a bright spot on a small appliance (possibly a microwave or coffee maker).
* **Image 3 (Middle-Right):** Kitchen with a window and hanging pots. Heatmap shows activation along the countertop and lower cabinets.
* **Row 3 (Identifier: 105 (X)):**
* **Image 1 (Bottom-Left):** Kitchen with an island. Heatmap shows activation on the island countertop and upper cabinets.
* **Image 2 (Bottom-Center):** Kitchen sink area with a window. Heatmap shows activation on the countertop, sink, and a basket.
* **Image 3 (Bottom-Right):** Kitchen with a wine rack and countertop appliances. Heatmap shows a focused activation point on the countertop near a bowl of fruit.
#### Text Box Content (Transcription)
**Left Box (b): RAW RULE-SET P3.1:**
```
1. target(X,'bedroom') :- 106(X).
2. target(X,'bathroom') :- not 43(X).
3. target(X,'kitchen') :- 105(X).
```
**Right Box: LABELLED RULE-SET P3.1:**
```
1. target(X,'bedroom') :- ...
2. target(X,'bathroom') :- not water_cooler...
3. target(X,'kitchen') :- ...
```
*(Note: The right box is partially cropped. The transcribed text is what is visible. The ellipses (...) indicate where text is cut off.)*
### Key Observations
1. **Heatmap Specificity:** The heatmaps are not uniform. Some, like the refrigerator in `43(X)` row 1, show extremely localized activation. Others, like the tables in `106(X)` row 2 and 3, show broader activation across surfaces.
2. **Identifier-Scene Association:** The numeric identifiers (`106`, `43`, `105`) are associated with specific visual features highlighted by the heatmaps, which are then used in the rule set.
3. **Rule Set Logic:** The raw rule set uses the identifiers as predicates.
* Rule 1: If feature `106` is present in image `X`, classify it as a 'bedroom'.
* Rule 2: If feature `43` is *not* present in image `X`, classify it as a 'bathroom'.
* Rule 3: If feature `105` is present in image `X`, classify it as a 'kitchen'.
4. **Semantic Labeling:** The labelled rule set begins to replace the numeric identifiers with semantic concepts (e.g., `43(X)` becomes `not water_cooler...`). This suggests `43` might correspond to a "water cooler" feature, and its absence is indicative of a bathroom.
### Interpretation
This image illustrates a pipeline for **explainable scene classification**. The heatmaps provide visual evidence of what features a model uses (e.g., beds for bedrooms, countertops for kitchens, a specific appliance for non-bathrooms). The rule sets translate these visual features into human-readable, logical classification rules.
* **Data Suggestion:** The system demonstrates that complex scene recognition can be decomposed into detecting specific, localized visual concepts. The high activation on the refrigerator (`43`) is particularly notable as a strong negative indicator for a bathroom.
* **Element Relationships:** The heatmaps are the *grounding* for the symbolic rules. Each numeric predicate (`106`, `43`, `105`) in the raw rule set corresponds to a specific pattern of visual activation shown in the grid. The labelled rule set represents the next step: interpreting those patterns as named objects or concepts.
* **Anomalies/Patterns:** The rule for 'bathroom' is defined by the *absence* of a feature (`not 43(X)`), which is a common logical construct but interesting in a visual context. It implies the model identifies bathrooms partly by what they lack (e.g., a prominent water cooler or similar appliance found in kitchens). The heatmaps for the `105` (kitchen) identifier consistently highlight countertops and appliances, reinforcing the rule's validity.
</details>
Figure 2: a) The top-5 images overlayed with activation heat-maps for neurons 106, 43 and 105 when rules are extracted for the P3.1 dataset containing classes âbathroomâ, âbedroomâ and âkitchenâ. b) The raw rule-set and the labelled rule-set when NeSyViT is employed on the P3.1 dataset.
[Q4, Q5] Neuron Activation Patterns and Automatic Semantic Labelling Efficacy: A key component of interpretability in neuro-symbolic frameworks is the alignment of individual neurons with semantically meaningful concepts relevant to specific classes.
Setup: To evaluate this alignment, we apply the previously described, semantic labelling algorithm to assign human-interpretable concept names to each predicate in the rule-set, where each predicate corresponds to a neuron in the final sparse concept layer. Initially, the raw rule-set produced by the FOLD-SE-M algorithm uses only neuron indices as predicate names. The semantic labelling process replaces these with descriptive concept labels derived by analyzing the top-10 most activating images per neuron. These images are overlaid with attention-based heatmaps and compared with pre-annotated semantic segmentation masks to identify the dominant concept for each neuron.
In Figure 2 a), we visualize the top- $5$ most activating images for each predicate in the labelled rule-set for the P3.1 dataset, which contains the classes âbathroomâ, âbedroomâ, and âkitchen.â Figure 2 b) shows a side-by-side comparison of the raw and semantically labelled rule-sets, illustrating how the abstract neuron indices are transformed into interpretable concept-based rules.
Result: Examining the labelled rule-set in Figure 2 b), we observe that just three rules are sufficient to achieve a classification accuracy of $99\$ . Consider Rule 2: target(X, âbathroomâ) :- not water_cooler1_tray1_refrigerator1_range1(X). At first glance, this rule may appear counterintuitiveâwhy would the absence of several kitchen-related concepts imply that the image depicts a bathroom rather than a bedroom? The explanation lies in the execution strategy of FOLD-SE-M, which evaluates rules in a top-down fashion. Rule 2 only activates if Rule 1 fails to fire. Rule 1 is: target(X, âbedroomâ) :- bed1(X). Thus, if the image lacks the âbedâ concept, Rule 1 is bypassed, and Rule 2 checks for the absence of kitchen concepts; if none are present, the image is classified as a âbathroom.â Rule 3: target(X, âkitchenâ) :- refrigerator2(X) fires only if the previous two rules do not apply and the ârefrigeratorâ concept is present, thereby classifying the image as kitchen.â
While the logical flow of the rules is valid, the quality of semantic labelling is less reliable. The labelling pipeline we adapted from NeSyFOLD was designed for CNNs, where modular filters make concept isolation more straightforward. In ViTs, however, neurons often attend to multiple regions and concepts simultaneously, making the learned representations less disentangled. As a result, some neuron labels can be misleading. This limitation is evident in Figure 2 a): neuron $106$ consistently activates for the âbedâ concept in bedroom images, but neurons $43$ and $105$ do not display clear concept selectivity and instead respond to mixed features. This highlights the challenge of isolating concepts in self-attention-based architectures.
While the automatic labelling approach may have limitations, the visualization of neuron activation heatmaps remains a powerful tool. These heatmaps allow for manual inspection of concept associations and provide valuable interpretability cues. We believe this combination of automated rule generation and visual validation offers a promising direction. Also, since the extracted rule-set is an Answer Set Program we can get justification of any prediction using the s(CASP) (Arias et al., (2018)) ASP interpreter. For completeness, we provide the labelled rule-sets for all datasets used in our experiments in Table 1, in the Appendix.
## 5 Related Works
The integration of symbolic reasoning with deep learning has led to various approaches for extracting interpretable rules from neural networks. While initial efforts focused on convolutional neural networks (CNNs), recent studies have begun exploring Vision Transformers (ViTs). VisionLogic (Geng et al., (2025)) introduces a framework that transforms neurons in the final fully connected layer of deep vision models into predicates, grounding them into visual concepts through causal validation. ViT-NeT (Kim et al., (2022)) presents a neural tree decoder that interprets the decision-making process of ViTs. By routing images hierarchically through a tree structure, it provides transparent and interpretable classifications, addressing the trade-off between model complexity and interoperability. These methods primarily focus on post-hoc interpretability or rely on additional structures for explanation. In contrast, our approach integrates a sparse linear layer directly into the ViT architecture, enabling the extraction of executable logic programs without auxiliary components.
Enhancing the interpretability of ViTs has been a subject of extensive research, leading to various methodologies. INTR (Paul et al., (2024)) proposes a proactive approach, asking each class to search for itself in an image. This idea is realized via a Transformer encoder-decoder where âclass-specificâ queries (one for each class) are learnt as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. Each class is attended to very distinctly hence the cross-attention weights provide a interpretation of the prediction. LeGrad (Bousselham et al., (2024)) introduces an explainability method specifically designed for ViTs. By computing gradients with respect to attention maps and aggregating signals across layers, LeGrad produces explainability maps that offer insights into the modelâs focus areas during decision-making. While these approaches enhance the transparency of ViTs, they do not produce symbolic or executable explanations. Our method differs by generating logic programs that serve as the final decision layer, enabling symbolic reasoning directly from the modelâs internal representations.
The pursuit of sparse and disentangled representations in ViTs has led to innovative methodologies. Recent work integrates sparse autoencoders (SAEs) with Vision Transformers (ViTs) to improve interpretability. Joseph et al., (2025) trained SAEs on CLIPâs ViT and found that manipulating a small set of steerable features can enhance performance and robustness. PatchSAE (Lim et al., (2025)) introduces method to extract granular visual concepts and study how these influence predictions, showing that most adaptation gains stem from existing features in the pre-trained model. Our approach builds upon these concepts by integrating a sparse linear layer into the ViT architecture and using it for extracting a symbolic rule-set which gives a global explanation of the model and a justification can be obtained for each prediction.
## 6 Conclusion and Future Work
In this work, we proposed a novel neuro-symbolic framework for interpretable image classification using Vision Transformers (ViTs). By replacing the final classification layer with a linear layer (sparse concept layer) producing sigmoid outputs, and introducing a unified loss functionâcomprising supervised contrastive loss for better class separation in latent space, entropy loss for sharper binarization, and sparsity loss for concept selectivityâwe generated sparse binary vectors that represent images in a disentangled manner. These vectors serve as inputs to the FOLD-SE-M algorithm, which produces stratified Answer Set Programs in the form of concise rule-sets. The resulting neuro-symbolic (NeSy) model, composed of the modified ViT and the learned rule-set, outperforms the vanilla ViT by an average of 5.14% in accuracy.
We compared our framework, NeSyViT, with NeSyFOLD, a similar approach for CNNs, and found that NeSyViT produces rule-sets that are 67% smaller on average while improving upon the baseline ViTâs accuracyâwhereas NeSyFOLD suffers a 10.14% drop in performance compared to its CNN counterpart. These results demonstrate the potential of attention-based architectures for interpretable, logic-driven classification.
Finally, we investigated the semantic interpretability of neurons in our framework. We observed that the automatic semantic labelling algorithm used in NeSyFOLDâbased on overlap with segmentation masksâstruggles in ViTs due to limited neuron monosemanticity. However, we argue that this challenge can be partially mitigated by visualizing attention-guided heatmaps for the small set of neurons that appear in the rule-set, enabling manual inspection and insight into the learned concepts.
Future work will focus on improving neuron disentanglement and enhancing monosemanticity using architectural or training refinements. Another promising direction explored in CNN-based interpretability methods is bias correction using extracted rule-sets Padalkar et al., 2024c which could be the natural next step for this work. We also plan to explore using multimodal LLMs like GPT-4o for concept labeling, enabling an automatic semantic annotation pipeline that does not rely on pixel-level segmentation masks.
## References
- Arias et al., (2018) Arias, J., Carro, M., Salazar, E., Marple, K., and Gupta, G. Constraint answer set programming without grounding. In Theory and Practice of Logic Programming 2018, volume 18, pp. 337â354.
- Baral, (2003) Baral, C. 2003. Knowledge representation, reasoning and declarative problem solving. Cambridge University Press.
- Bousselham et al., (2024) Bousselham, W., Boggust, A., Chaybouti, S., Strobelt, H., and Kuehne, H. 2024. Legrad: An explainability method for vision transformers via feature formation sensitivity.
- Dosovitskiy et al., (2021) Dosovitskiy, A., Beyer, L., and others. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021 2021.
- Gelfond and Kahl, (2014) Gelfond, M. and Kahl, Y. 2014. Knowledge representation, reasoning, and the design of intelligent agents: The answer-set programming approach. Cambridge University Press.
- Geng et al., (2025) Geng, C., Jiang, Y., Zhao, Z., Ye, H., Wang, Z., and Si, X. Learning interpretable logic rules from deep vision models. In arXiv 2503.10547 2025.
- Joseph et al., (2025) Joseph, S., Suresh, P., and Others. Steering clipâs vision transformer with sparse autoencoders. In arXiv 2504.08729 2025.
- Kanagaraj et al., (2021) Kanagaraj, N., Hicks, D., and Others. Deep learning using computer vision in self driving cars for lane and traffic sign detection. In Inter. Journal of Sys. Assurance Engg. and Management 2021, volume 12, pp. 1011â1025.
- Khosla et al., (2020) Khosla, P., Teterwak, P., and others. Supervised contrastive learning. In NeurIPS 2020.
- Kim et al., (2022) Kim, S., Nam, J., and Ko, B. C. Vit-net: Interpretable vision transformers with neural tree decoder. In International conference on machine learning 2022, pp. 11162â11172. PMLR.
- Ko and Kwak, (2012) Ko, B. and Kwak, S. 2012. Survey of computer vision-based natural disaster warning systems.
- Lage et al., (2019) Lage, I., Chen, E., and others. Human evaluation of models built for interpretability. In Proc. HCOMP 2019, volume 7, pp. 59â67.
- Lim et al., (2025) Lim, H., Choi, J., and Others. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In ICLR 2025.
- Ng et al., (2011) Ng, A. and others 2011. Sparse autoencoder.
- (15) Padalkar, P., Wang, H., and Gupta, G. Nesyfold: A framework for interpretable image classification. In Proc. AAAI 2024a, pp. 4378â4387. AAAI Press.
- (16) Padalkar, P., Wang, H., and Gupta, G. Using logic programming and kernel-grouping for improving interpretability of convolutional neural networks. In Proc. PADL 2024b, volume 14512 of LNCS, pp. 134â150. Springer.
- (17) Padalkar, P., Ćlusarz, N., Komendantskaya, E., and Gupta, G. 2024c. A neurosymbolic framework for bias correction in convolutional neural networks, 24 c, 4, 644â662.
- Paul et al., (2024) Paul, D., Chowdhury, A., and Others 2024. A simple interpretable transformer for fine-grained image classification and analysis.
- Stallkamp and Others, (2012) Stallkamp, J. and Others. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. In Neural Networks 2012, volume 32, pp. 323â332.
- Sun et al., (2016) Sun, W., Zheng, B., and Qian, W. Computer aided lung cancer diagnosis with deep learning algorithms. In SPIE Medical Imaging 2016.
- Townsend et al., (2021) Townsend, J., Kasioumis, T., and Inakoshi, H. Eric: Extracting relations inferred from convolutions. In Computer Vision â ACCV 2021, pp. 206â222, Cham. Springer International.
- Wang and Gupta, (2024) Wang, H. and Gupta, G. FOLD-SE: an efficient rule-based machine learning algorithm with scalable explainability. In Proc. PADL 2024, volume 14512 of LNCS, pp. 37â53. Springer.
- Zhou and Others, (2018) Zhou, B. and Others. Places: A 10 million image database for scene recognition. In IEEE Trans. Pattern Anal. Mach. Intell. 2018, volume 40, pp. 1452â1464.
Appendix
Our paper above shows the labeled rule-set for only one dataset i.e. P3.1 so here we show the labelled rule-sets for all the other datasets as well. Also we show the hyperparameters used for training and the rule generation using FOLD-SE-M.
## Appendix A Labelled rule-sets
P2 {âbathroomâ, âbedroomâ}: target(X,âbathroomâ) :- bathroom_tiles1_shower_screen1(X). target(X,âbedroomâ) :- bed1_duvet1(X).
P3.1 {âbathroomâ, âbedroomâ, âkitchenâ}: target(X,âbedroomâ) :- not shower_screen1(X). target(X,âbathroomâ) :- not refrigerator1_kitchen_island1_range1_countertop1_person1_wall1_air_conditioning1(X). target(X,âkitchenâ) :- not bathtub1_mirror1(X).
P3.2 {âdesert roadâ, âforest roadâ, âstreetâ}: target(X,âdesert_roadâ) :- field1_grass1_desert1(X). target(X,âstreetâ) :- food_cart1(X). target(X,âforest_roadâ) :- not building1(X).
P3.3 {âdesert roadâ, âdrivewayâ, âhighwayâ}: target(X,âhighwayâ) :- not buildings1(X). target(X,âdrivewayâ) :- not hill1_ground3(X). target(X,âdesert_roadâ) :- field1_ground1(X). target(X,âhighwayâ) :- buildings1(X). target(X,âdesert_roadâ) :- ground2(X).
P5 {âbathroomâ, âbedroomâ, âkitchenâ, âdining roomâ, âliving roomâ}:
target(X,âbathroomâ) :- toilet1(X). target(X,âliving_roomâ) :- fireplace1(X). target(X,âdining_roomâ) :- not bed2_railing1(X). target(X,âbedroomâ) :- bed1(X). target(X,âkitchenâ) :- top1(X).
P10 {âbathroomâ, âbedroomâ, âkitchenâ, âdining roomâ, âliving roomâ, âhome officeâ, âofficeâ, âwaiting roomâ, âconference roomâ, âhotel roomâ}: target(X,âbedroomâ) :- bed1(X), not bed2(X). target(X,âbathroomâ) :- shower_screen1_mirror1(X). target(X,âdining_roomâ) :- tablecloth1_bar1(X), not work_surface1_front1_trash_can1(X). target(X,âofficeâ) :- tablecloth1_bar1(X). target(X,âliving_roomâ) :- shelves1(X), not table1_lectern1_floor2_top1(X). target(X,âconference_roomâ) :- shelves1(X). target(X,âkitchenâ) :- floor1(X). target(X,âhotel_roomâ) :- sofa1_seat1_armchair1_door1_wall1(X). target(X,âhome_officeâ) :- bed1(X). target(X,âwaiting_roomâ) :- bed2(X), not bathtub1(X).
## Appendix B Neuron Attention-Maps for P10
label(X,âbedroomâ) :- 14(X), not 83(X). label(X,âbathroomâ) :- 81(X). label(X,âdining_roomâ) :- 122(X), not 97(X). label(X,âofficeâ) :- 122(X). label(X,âliving_roomâ) :- 43(X), not 45(X). label(X,âconference_roomâ) :- 43(X). label(X,âkitchenâ) :- 41(X). label(X,âhotel_roomâ) :- 50(X). label(X,âhome_officeâ) :- 14(X). label(X,âwaiting_roomâ) :- 83(X), not 5(X).
## Appendix C Hyperparameters
NeSyViT: For the subsets of the Places dataset i.e. P2, P3.1, P3.2, P3.3, P5, P10 We used a learning rate of $5\times 10^{-6}$ and l2 weight decay of $5e^{-3}$ with the AdamW optimizer, batch size of $32$ , patience of $10$ epochs for the LR-Scheduler, LR decay factor of $0.5$ .
The weights for the supcon loss, sparsity loss and entropy loss were $2,1$ and $1$ respectively.
For the GT43 dataset we changed the learning rate to $5\times 10^{-5}$ and the weights for the supcon loss, sparsity loss and entropy loss were $4,0.001$ and $0.001$ respectively. Every other hyperparameter was the same.
For all datasets we used a ratio of $0.8$ and a tail of $5e^{-3}$ for the FOLD-SE-M algorithm.
Vanilla ViT: For all datasets we used a learning rate of $5\times 10^{-6}$ , batch size of $32$ , patience of $10$ epochs for the LR-Scheduler, LR decay factor of $0.5$ and l2 weight decay of $5e^{-3}$ . The loss used was standard crossentropy loss.