# Symbolic Rule Extraction from Attention-Guided Sparse Representations in Vision Transformersâ â thanks: This work was supported by US NSF Grant IIS 1910131, US DoD and industry grants. We thank the members of the ALPS lab at UT Dallas for insightful discussions.
**Authors**: \snParth\gnPadalkar, \snGopal\gnGupta
> The University of Texas at Dallas
\lefttitle
Cambridge Author
\jnlPage
18 \jnlDoiYr 2021 \doival 10.1017/xxxxx
{authgrp}
Abstract
Recent neuro-symbolic approaches have successfully extracted symbolic rule-sets from CNN-based models to enhance interpretability. However, applying similar techniques to Vision Transformers (ViTs) remains challenging due to their lack of modular concept detectors and reliance on global self-attention mechanisms. We propose a framework for symbolic rule extraction from ViTs by introducing a sparse concept layer inspired by Sparse Autoencoders (SAEs). This linear layer operates on attention-weighted patch representations and learns a disentangled, binarized representation in which individual neurons activate for high-level visual concepts. To encourage interpretability, we apply a combination of L1 sparsity, entropy minimization, and supervised contrastive loss. These binarized concept activations are used as input to the FOLD-SE-M algorithm, which generates a rule-set in the form of logic programs. Our method achieves a 5.14% better classification accuracy than the standard ViT while enabling symbolic reasoning. Crucially, the extracted rule-set is not merely post-hoc but acts as a logic-based decision layer that operates directly on the sparse concept representations. The resulting programs are concise and semantically meaningful. This work is the first to extract executable logic programs from ViTs using sparse symbolic representations. It bridges the gap between transformer-based vision models and symbolic logic programming, providing a step forward in interpretable and verifiable neuro-symbolic AI.
keywords: Neuro-symbolic AI, Mechanistic Interpretability, Vision Transformers, Explainable AI (XAI), Rule-Based Machine Learning, Sparse Autoencoders
1 Introduction
Extracting logic-based rules from neural models has emerged as a central objective in neuro-symbolic AI, driven by the growing demand for interpretability and verifiability in machine learning. As deep learning models continue to scale, they are increasingly deployed in critical applications such as autonomous driving (Kanagaraj et al., (2021)), medical diagnosis (Sun et al., (2016)), and natural disaster prevention (Ko and Kwak, (2012)). In these sensitive domains, incorrect predictions can carry severe consequences, emphasizing the necessity for transparency in decision-making. Many of these applications depend significantly on accurate image classification models, particularly Convolutional Neural Networks (CNNs). Recent frameworks such as NeSyFOLD (Padalkar et al., 2024a ; Padalkar et al., 2024b ) and ERIC (Townsend et al., (2021)) have successfully demonstrated the extraction of human-interpretable symbolic rule sets from CNNs, providing insights into the underlying reasoning of predictions in vision tasks.
Most of this progress, however, has been concentrated on CNNs. CNNs are composed of filters, that are trainable matrices that learn to detect patterns in local regions of images (e.g., a bed in an image classified as a bedroom). Their modular architecture, where individual filters often correspond to distinct visual concepts, makes them particularly suitable for rule extraction. By binarizing the activations of the final layer and using them as input to rule-based machine learning algorithms such as decision trees or FOLD-SE-M (Wang and Gupta, (2024)), it is possible to extract the symbolic rule-sets.
In contrast, Vision Transformers (ViTs) (Dosovitskiy et al., (2021)) have remained largely inaccessible to symbolic extraction techniques. While ViTs now dominate the field of vision due to their superior performance and flexibility, their reliance on global self-attention and lack of explicit concept detectors pose a significant challenge for rule-based interpretability. ViTs encode information in a distributed and entangled manner, making it unclear how to localize or discretize concept-level representations.
In this work, we take a step toward bridging this gap by introducing a frameworkâNeSyViTâfor extracting logic-based rule sets from Vision Transformers (ViTs). We modify the standard ViT architecture by replacing the final fully connected classification head with a single linear layerâ sparse concept layer âtrained to produce binarized outputs. The goal is to encourage each neuron in this final layer to correspond to a few distinct high-level concepts. To achieve this, we draw inspiration from Sparse Autoencoders (SAEs) (Ng et al., (2011)), incorporating an L1 sparsity loss to ensure that only a small subset of neurons activates for any given image.
To encourage binarization, we apply a sigmoid activation to the linear outputs, constraining them to the $[0,1]$ range, and introduce an entropy-based loss that pushes activations toward binary extremes (0 or 1). This design enables us, after training, to extract a binary vector for each image that reflects concept-level activations. These vectors are then passed to the FOLD-SE-M algorithm to generate a symbolic rule-set in the form of a stratified Answer Set Program for classification. It is crucial that the representations corresponding to images from the same class are well-clustered in the latent space. This promotes the formation of clear decision boundaries that can be effectively exploited by FOLD-SE-M. To enforce this structure, we incorporate a supervised contrastive loss (SupCon) (Khosla et al., (2020)), which encourages representations of images with the same label to lie closer together while pushing apart those from different classes. This facilitates learning highly accurate rules.
Finally, each predicate in the extracted rule set is âsemantically labelledâ based on the concept(s) that its corresponding neuron represents. Padalkar et al. introduced an algorithm for automatic semantic labelling of the predicates in the rule-sets extracted from a CNN using FOLD-SE-M (Padalkar et al., 2024a ). We adopt the same algorithm for semantic labelling of the rule-sets we generate. Figure 1 illustrates the NeSyViT framework.
<details>
<summary>extracted/6428030/Images/NeSyViT2.png Details</summary>
### Visual Description
\n
## Diagram: Visualizing Rule Generation from Image Input
### Overview
This diagram illustrates a process for generating rules from image input using a Vision Transformer (ViT) model, a sparse concept layer, and a Fold-SE-M rule generation component. The process involves inputting an image, extracting features with ViT, binarizing neuron outputs, and then generating labeled and raw rule sets through semantic labeling.
### Components/Axes
The diagram consists of the following key components:
* **Input Image:** A photograph of a bedroom scene.
* **ViT (Vision Transformer):** A red rectangular block representing the image feature extractor.
* **Sparse Concept Layer:** A series of circles labeled `n1`, `n2`, `n3`, `n4`, ..., `nd`, representing neurons with associated confidence scores.
* **Binarization of Neuron Outputs:** A rectangular table showing the binary (0 or 1) outputs of the neurons.
* **Fold-SE-M Rule Generation:** A rectangular block indicating the rule generation process.
* **Labelled Rule-Set:** A block of text listing rules with semantic labels.
* **Raw Rule-Set:** A block of text listing rules without semantic labels.
* **Arrows:** Indicate the flow of information between components.
* **[CLS]**: A label indicating the classification token.
### Detailed Analysis or Content Details
**1. Input to ViT:**
* The input is an image of a bedroom. The image shows a bed, a window, and other furniture.
**2. ViT Output & Sparse Concept Layer:**
* The ViT model processes the image and outputs features to the Sparse Concept Layer.
* The Sparse Concept Layer consists of neurons `n1` through `nd`.
* The confidence scores associated with each neuron are as follows (approximate values):
* `n1`: 0.8
* `n2`: 0.1
* `n3`: 0.0
* `n4`: 0.9
* `nd`: 0.2
**3. Binarization of Neuron Outputs:**
* The neuron outputs are binarized (converted to 0 or 1). The table shows the binary outputs for neurons `n1` through `nd`.
* The table has columns `n1`, `n2`, `n3`, `n4`, and `nd` (and more, indicated by "...").
* The table has rows of binary values (0 or 1).
* Example values:
* `n1`: 0, 1, 1, 1, ... , 1
* `n2`: 0, 0, 1, 1, ... , 1
* `n3`: 1, 0, 1, 0, ... , 0
* `n4`: 1, 1, 0, 1, ... , 1
* `nd`: 1, 0, 0, 0, ... , 0
**4. Labelled Rule-Set:**
* The labelled rule-set contains rules with semantic labels.
* Example rules:
1. `target(X, 'bedroom') :- bed1(X), pillow1(X), not ab1(X).`
2. `target(X, 'kitchen') :- cabinet1(X), not ab2(X).`
3. `ab1(X) :- sink1(X), not ab2(X).`
**5. Raw Rule-Set:**
* The raw rule-set contains rules without semantic labels.
* Example rules:
1. `target(X, 'bedroom') :- n5(X), n1(X), not ab1(X).`
2. `target(X, 'kitchen') :- n2(X), not ab2(X).`
3. `ab1(X) :- n41(X), not ab2(X).`
**6. Semantic Labelling:**
* An arrow labeled "SEMANTIC LABELLING" connects the Raw Rule-Set to the Labelled Rule-Set, indicating the process of adding semantic labels to the rules.
**7. Fold-SE-M Rule Generation:**
* An arrow labeled "FOLD-SE-M RULE GENERATION" points from the Binarization of Neuron Outputs to the Fold-SE-M Rule Generation block.
### Key Observations
* The confidence scores of the neurons vary significantly, suggesting that some neurons are more strongly activated by the input image than others.
* The binarization process converts the continuous confidence scores into discrete binary values, which may result in information loss.
* The labelled rule-set provides more interpretable rules by associating semantic labels with the concepts.
* The raw rule-set uses neuron identifiers (e.g., `n5(X)`) instead of semantic labels.
### Interpretation
This diagram illustrates a pipeline for converting visual information into symbolic rules. The ViT model extracts features from the image, and the sparse concept layer represents these features as a set of neurons. The binarization step creates a discrete representation of the neuron activations, which is then used to generate rules. The semantic labelling process adds meaning to the rules, making them more interpretable. The overall goal is to create a system that can reason about images and generate logical rules based on their content. The difference between the "Labelled Rule-Set" and the "Raw Rule-Set" suggests a process of mapping neuron activations to human-understandable concepts. The varying confidence scores of the neurons indicate that the system is able to identify different concepts with varying degrees of certainty. The use of "not" in the rules suggests that the system is also able to identify the absence of certain concepts. This system could be used for tasks such as visual question answering, image captioning, or robot navigation.
</details>
Figure 1: The NeSyViT Framework
The final Ne uro- Sy mbolic (NeSy) model is a combination of the ViT based feature extractor that generates the binary vector for each image and the rule-set generated by FOLD-SE-M, which classifies the image into a particular class based on which neurons were activated/deactivated (1/0). We demonstrate through our experiments that the NeSy model produced by our framework outperforms the vanilla ViT in terms of classification accuracy, achieving an average improvement of 5.14%. Notably, this performance gain is achieved while also generating interpretable rule-setsâa significant result, as prior neuro-symbolic frameworks that extract rule-sets from neural models often suffer a drop in accuracy.
Our contributions are as follows:
1. We introduce a training method that combines supervised contrastive loss, L1 sparsity, and entropy loss to learn compact and binarized concept-level representations.
1. We propose an end-to-end neuro-symbolic framework, NeSyViT, for generating a rule-set from a modified ViT using FOLD-SE-M.
1. We show through experiments that our NeSyViT framework achieves classification accuracy better than the vanilla ViT, while also producing concise rule-sets
2 Background
2.1 Vision Transformers
The Vision Transformer (ViT) (Dosovitskiy et al., (2021)) is a deep learning architecture that applies the Transformer modelâoriginally developed for natural language processingâto image classification tasks. Unlike Convolutional Neural Networks (CNNs), which operate on local regions using convolutional filters, ViTs treat an image as a sequence of patches. An input image is divided into fixed-size non-overlapping patches (e.g., $16Ă 16$ pixels), each of which is flattened and projected into a vector embedding. The vector embeddings of those patches then go through multiple self-attention layers followed by the final classification layer.
The Transformer uses self-attention layers to model relationships between all image patches simultaneously. At the core of this mechanism is the concept of an attention head, which determines how much attention each patch should pay to every other patch in the sequence. For each patch token, the model learns three distinct vectors: a query, a key, and a value. Attention scores are computed by taking the dot product between a patchâs query vector and the key vectors of all other patches, followed by a softmax normalization to produce attention weights. These weights are then used to compute a weighted sum of the value vectors, generating a context-aware representation for each token.
A special learnable token, known as the [CLS] token, is prepended to the patch sequence before being passed into the Transformer. Unlike the patch tokens that represent image regions, the [CLS] token acts as a summary placeholder. During self-attention, it aggregates information from all other patches by attending to them across multiple layers. After the final Transformer layer, the embedding corresponding to the [CLS] token is typically used for classification, as it encapsulates a global representation of the image.
While this architecture allows Vision Transformers to capture long-range dependencies and global structure, it also presents a major challenge for interpretability. In CNNs, individual filters often learn to detect localized visual patterns or objects (e.g., corners, textures, faces), enabling a degree of modularity and conceptual traceability. In contrast, attention heads are distributed, context-sensitive, and dynamically influenced by the full set of tokensâincluding the [CLS] token. This makes it difficult to associate specific heads or tokens with interpretable visual concepts. Consequently, extracting symbolic, modular representations from ViTs is significantly more challenging than from CNNs. Hence, in this work, we take the [CLS] vector and pass it through the sparse concept layer that is optimized to produce sparse outputs, allowing the information encoded in the [CLS] token to be disentangled into a small set of active neurons. Each neuron is encouraged to specialize and activate primarily for images of a specific class. This design is inspired by the core principles of Sparse Autoencoders (SAEs) (Ng et al., (2011)), which promote compact and interpretable representations through sparsity.
2.2 FOLD-SE-M
The FOLD-SE-M algorithm Wang and Gupta, (2024) that we employ in our framework, learns a rule-set from data as a default theory. Default logic is a non-monotonic logic used to formalize commonsense reasoning. A default $D$ is expressed as:
$$
D=\frac{A:\textbf{M}B}{\Gamma} \tag{1}
$$
Equation 1 states that the conclusion $\Gamma$ can be inferred if pre-requisite $A$ holds and $B$ is justified. $\textbf{M}B$ stands for âit is consistent to believe $B$ â. Normal logic programs can encode a default theory quite elegantly (Gelfond and Kahl, (2014)). A default of the form:
$$
\frac{\alpha_{1}\land\alpha_{2}\land\dots\land\alpha_{n}:\textbf{M}\lnot\beta_%
{1},\textbf{M}\lnot\beta_{2}\dots\textbf{M}\lnot\beta_{m}}{\gamma}
$$
can be formalized as the normal logic programming rule:
$$
\gamma~{}\texttt{:-}~{}\alpha_{1},\alpha_{2},\dots,\alpha_{n},\texttt{not}~{}%
\beta_{1},\texttt{not}~{}\beta_{2},\dots,\texttt{not}~{}\beta_{m}.
$$
where $\alpha$ âs and $\beta$ âs are positive predicates and not represents negation-as-failure. We call such rules default rules. Thus, the default
$$
\frac{bird(X):M\lnot penguin(X)}{flies(X)}
$$
will be represented as the following default rule in normal logic programming:
flies(X) :- bird(X), not penguin(X).
We call bird(X), the condition that allows us to jump to the default conclusion that X flies, the default part of the rule, and not penguin(X) the exception part of the rule.
FOLD-SE-M (Wang and Gupta, (2024)) is a Rule Based Machine Learning (RBML) algorithm. It generates a rule-set from tabular data, comprising rules in the form described above. The complete rule-set can be viewed as a stratified answer set program (a stratified ASP rule-set has no cycles through negation (Baral, (2003))). It uses special abx predicates to represent the exception part of a rule where x is a unique numerical identifier. FOLD-SE-M incrementally generates literals for default rules that cover positive examples while avoiding covering negative examples. It then swaps the positive and negative examples and calls itself recursively to learn exceptions to the default when there are still negative examples falsely covered.
There are $2$ tunable hyperparameters, $ratio$ , and $tail$ . The $ratio$ controls the upper bound on the number of false positives to the number of true positives implied by the default part of a rule. The $tail$ controls the limit of the minimum number of training examples a rule can cover.
3 Methodology
Our aim is to extract a symbolic rule-set from the ViT and use it for final classification. To this end, we first modify the architecture of a standard Vision Transformer by replacing the final classification headâplaced after the last self-attention layerâwith a single linear layer of dimension $D$ . The [CLS] tokenâs vector which has the aggregated global information feeds directly into this layer. This layer outputs a $D$ -dimensional vector, to which we apply a sigmoid function element-wise, constraining all values to lie in the range $[0,1]$ . Each value in this vector could then be interpreted as the activation of a distinct neuron.
There are three key objectives in designing this representation:
1. The model needs to produce similar binary vectors for images of the same class, and dissimilar ones for different classes, to allow the FOLD-SE-M to identify meaningful decision boundaries.
1. The output values had to be pushed as close as possible to either $0 0$ or $1$ , since after training, we binarize these vectors by rounding. Values near $0.5$ would introduce ambiguity and loss of information.
1. The output vectors need to be sparse, with most values being $0 0$ and only a few being $1$ . This sparsity ensures that rule-sets remain compact and interpretable, as fewer neurons would appear as predicates in the learned rules.
We enforce these three properties using a combination of supervised contrastive loss, entropy minimization, and L1 sparsity loss, described in detail below.
Supervised Contrastive Loss: To ensure that images from the same class produce similar sparse representations, we incorporate the Supervised Contrastive Loss (SupCon) (Khosla et al., (2020)). This loss encourages the $D$ -dimensional vectors corresponding to samples of the same class to cluster together in the latent space, while pushing apart those from different classes. Such class-wise clustering is essential for enabling a rule-based learner, such as FOLD-SE-M, to identify clean symbolic decision boundaries.
Let $\mathbf{z}_{i}â\mathbb{R}^{D}$ denote the normalized sparse representation of sample $i$ in a batch $\mathcal{B}$ , and let $y_{i}$ be its corresponding class label. In contrastive learning, we refer to $\mathbf{z}_{i}$ as the anchor, and we compare it to the remaining samples in the batch. Those with the same label are treated as positives, while the rest are considered negatives.
The SupCon loss for a single anchor $i$ is defined as:
$$
\mathcal{L}_{\text{supcon}}^{i}=-\frac{1}{|P(i)|}\sum_{p\in P(i)}\log\frac{%
\exp(\mathbf{z}_{i}\cdot\mathbf{z}_{p}/\tau)}{\sum_{a\in\mathcal{A}(i)}\exp(%
\mathbf{z}_{i}\cdot\mathbf{z}_{a}/\tau)} \tag{2}
$$
where:
- $P(i)$ is the set of indices in the batch with the same label as $i$ (i.e., the positives for anchor $i$ ),
- $\mathcal{A}(i)$ is the set of all indices in the batch excluding $i$ itself,
- $\tau>0$ is a temperature parameter that controls the sharpness of the distribution.
The total loss is averaged over all anchors in the batch:
$$
\mathcal{L}_{\text{supcon}}=\frac{1}{N}\sum_{i\in\mathcal{B}}\mathcal{L}_{%
\text{supcon}}^{i} \tag{3}
$$
where $N$ is the number of images in the batch $\mathcal{B}$ . By minimizing this loss, the model is encouraged to produce sparse vectors that are tightly clustered for images of the same class and distinct from those of other classes. This structure makes the representations well-suited for interpretable rule extraction.
Entropy Minimization: To encourage the sparse representations to become binarizedâi.e., close to either 0 or 1âwe apply an entropy minimization loss on the output of the sigmoid-activated linear layer. Since this linear layer uses a sigmoid function, each neuronâs activation lies in the range $[0,1]$ . Ideally, we want these activations to converge toward discrete values ( $0 0$ or $1$ ) to allow lossless binarization post-training.
We treat each neuronâs output as a Bernoulli random variable and compute its entropy using the standard binary entropy formula. Given a batch of $N$ images where $\mathbf{z}_{i}â\mathbb{R}^{D}$ is the $D$ -dimensional output for image $i$ , the entropy loss is computed as:
$$
\mathcal{L}_{\text{entropy}}=-\frac{1}{ND}\sum_{i=1}^{N}\sum_{j=1}^{D}\left[z_%
{i,j}\log(z_{i,j}+\epsilon)+(1-z_{i,j})\log(1-z_{i,j}+\epsilon)\right], \tag{4}
$$
where $z_{i,j}$ is the activation of neuron $j$ for image $i$ , and $\epsilon$ is a small constant added for numerical stability.
Minimizing this entropy term encourages each activation to move closer to either $0 0$ or $1$ , making the final binarization step more reliable. This is especially important for downstream symbolic rule extraction, where crisp binary features are essential for learning accurate and interpretable logic programs.
L1 Sparsity Loss: To promote interpretability and ensure that only a few neurons activate for each image, we incorporate an L1 sparsity loss on the output of the final linear layer. Sparse representations are crucial for generating concise rule-sets, as they limit the number of active predicates per image, reducing the complexity of the learned logic programs.
Formally, given a batch of $N$ vectors ${\mathbf{z}_{1},...,\mathbf{z}_{N}}$ , where $\mathbf{z}_{i}â\mathbb{R}^{D}$ is the sigmoid-activated output for image $i$ , the L1 sparsity loss is defined as:
$$
\mathcal{L}_{\text{sparsity}}=\frac{1}{ND}\sum_{i=1}^{N}\sum_{j=1}^{D}|z_{i,j}| \tag{5}
$$
This loss penalizes large activation values and encourages most neurons to remain close to zero, allowing only a few dimensions to be active for any given input.
The total loss then becomes:
$$
\mathcal{L}_{Total}=\alpha\mathcal{L}_{supcon}+\beta\mathcal{L}_{entropy}+%
\gamma\mathcal{L}_{sparsity} \tag{6}
$$
where $\alpha$ , $\beta$ and $\gamma$ are weights to control the impact of each loss term. Finally, after training the model using the combined loss $\mathcal{L}_{\text{Total}}$ , we collect the sparse representation vectors for all images in the training set. These vectors are binarizedâeach value thresholded at $0.5$ âto form a binary table, where each row corresponds to an image and each column to a neuron in the final linear layer. This binarization table is used as input to the FOLD-SE-M algorithm, which generates a symbolic rule-set in the form of a stratified Answer Set Program. Each predicate in the resulting rule-set corresponds directly to a neuron in the final layer, enabling symbolic reasoning over the learned representations.
Semantic Labelling Algorithm: To assign human-understandable meanings to these predicates, we employ the semantic labeling algorithm introduced by Padalkar et al., 2024a . For each neuron, we identify the top- $10$ images that activate it most strongly. We then compute attention-based heatmaps for these images to localize the spatial regions that the neuron focuses on. These heatmaps are intersected with the corresponding (pre-annotated) semantic segmentation masksâwhere image regions are labeled with semantic object categories. For each neuron, we compute the average Intersection-over-Union (IoU) between its heatmaps and each available concept label. The label with the highest average IoU is then assigned to the predicate corresponding to that neuron, thus grounding the symbolic rule-set in interpretable visual concepts.
After the rule-set is generated, new images can be classified by passing them through the modified ViT, which outputs a $D$ -dimensional vector. This vector is then binarized by thresholding each value at $0.5$ . The resulting binary vector serves as input to the FOLD-SE-M toolkitâs rule interpreter, which evaluates the symbolic rule-set in a top-down manner. For each rule, the truth values of the predicates in the body are determined based on the corresponding neuron activations in the binarized vector. The first rule that fires determines the predicted class of the image. Moreover, the s(CASP) (Arias et al., (2018)) goal-directed ASP interpreter can be used to obtain a justification of the prediction.
4 Experiments
We conducted experiments to address the following research questions:
Q1: How does the classification accuracy of the neuro-symbolic model produced by the NeSyViT framework compare to that of a standard vanilla Vision Transformer (ViT)?
Q2: What is the typical size of the rule-set generated by the NeSyViT framework, and how compact are the resulting rules?
Q3: How does our framework scale as the number of classes increases?
Q4: What do the neuron activation patterns look like for each class, and how well do they align with human-interpretable semantic concepts?
Q5: How well can the semantic labelling algorithm meant for CNN-based frameworks such as NeSyFOLD, be directly adopted for our NeSyViT framework?
[Q1, Q2, Q3] Classification Accuracy, Rule-set Size and Scalability: The central goal of the NeSyViT framework is to produce a neuro-symbolic model that maintains high classification accuracy while offering symbolic interpretability through a compact rule-set. Ideally, the accuracy of the interpretable model should match or exceed that of its purely neural counterpart (i.e., the vanilla ViT), and the resulting rule-set should remain as concise as possible. We use rule-set size as a proxy for interpretability, based on the findings of Lage et al. (Lage et al., (2019)), who demonstrated through human-subject evaluations that larger rule-sets are significantly more difficult to interpret. Thus, achieving high accuracy alongside a small rule-set is critical for balancing performance and human-understandability.
Setup: We use the ViT-Base architecture from the timm library, which processes input images of size $224Ă 224$ using non-overlapping $16Ă 16$ patches. The model consists of 12 Transformer blocks, each with 12 attention heads, and produces a [CLS] token embedding of dimension 768. As described earlier, we modify this architecture by removing the final classification head and replacing it with a single linear layer of output dimension 128. All experiments are conducted using this configuration with weights pretrained on ImageNet-1k to construct the NeSy model. The vanilla ViT refers to the same base architecture without any modifications. We used the AdamW optimizer along with a cosine annealing learning rate scheduler. The model was trained for $50$ epochs using the combined loss consisting of supervised contrastive loss, entropy minimization, and L1 sparsity, as previously described. For rule-set generation, we employed the FOLD-SE-M algorithm. The weights assigned to each loss component, as well as additional training and architectural hyperparameters, are provided in the Appendix.
We evaluated both our NeSy model and the vanilla ViT on subsets of two benchmark datasets (1): the Places dataset (Zhou and Others, (2018)), which contains images of various indoor and outdoor scenes, and the German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp and Others, (2012)), which consists of images of traffic signposts. From the Places dataset, we constructed multiple class subsets of increasing number of classes to gauge the scalability of NeSyViT: two classes (P2), three classes (P3.1), five classes (P5), and ten classes (P10). The P2 subset includes bathroom and bedroom images. P3.1 extends this by adding kitchen, while P5 further incorporates dining room and living room. P10 adds five additional classes: home office, office, waiting room, conference room, and hotel room. Additionally, we include two alternative three-class subsets: P3.2 with desert road, forest road, and street, and P3.3 with desert road, driveway, and highway. For each class, we sampled $5,000$ images, creating a $4$ k/ $1$ k train-test split, and used the official validation set without modification.
The GTSRB (GT43) dataset contains $43$ traffic sign classes. We used the official test set of $12.6k$ images and performed an $80/20$ train-validation split on the remaining data, resulting in approximately $21k$ training and $5k$ validation images.
We run each experiment $5$ times with random train-test splits and then report the average metrics in Table 1. We closely follow the experimental setup proposed by Padalkar et al.(Padalkar et al., 2024a ) for evaluating the NeSyFOLD framework, which uses FOLD-SE-M to extract rule-sets from CNNs. Specifically, we compare the drop (or gain) in accuracy observed while using NeSyFOLD as compared to the vanilla CNN, against the corresponding change in accuracy when using NeSyViT as compared to the Vanilla ViT. This relative comparison highlights the effectiveness of our approach and is summarized in Table 2.
| P2 Vanilla P3.1 | NeSyViT $99± 0.00$ NeSyViT | $\textbf{100}±\textbf{0.00}$ - $\textbf{99}±\textbf{0.00}$ | $2± 0.00$ - $3± 0.00$ | $2± 0.00$ - $3± 0.00$ | $2± 0.00$ $3± 0.00$ |
| --- | --- | --- | --- | --- | --- |
| Vanilla | $98± 0.00$ | - | - | - | |
| P3.2 | NeSyViT | $\textbf{99}±\textbf{0.00}$ | $3± 0.00$ | $3± 0.40$ | $3± 0.00$ |
| Vanilla | $97± 0.00$ | - | - | - | |
| P3.3 | NeSyViT | $\textbf{98}±\textbf{0.00}$ | $5± 0.40$ | $3± 0.49$ | $5± 0.40$ |
| Vanilla | $91± 0.00$ | - | - | - | |
| P5 | NeSyViT | $\textbf{98}±\textbf{0.00}$ | $5± 0.00$ | $5± 0.40$ | $5± 0.80$ |
| Vanilla | $90± 0.00$ | - | - | - | |
| P10 | NeSyViT | $\textbf{94}±\textbf{0.00}$ | $10± 0.49$ | $12± 1.47$ | $14± 1.67$ |
| Vanilla | $76± 0.00$ | - | - | - | |
| GT43 | NeSyViT | $98± 0.00$ | $43± 0.00$ | $44± 1.90$ | $51± 3.12$ |
| Vanilla | $\textbf{99}±\textbf{0.00}$ | - | - | - | |
| Mean Stats | NeSyViT | $\textbf{98}±\textbf{0.00}$ | $10.14± 0.13$ | $10.30± 0.67$ | $11.85± 0.86$ |
| Vanilla | $92.86± 0.00$ | - | - | - | |
Table 1: Comparison between the NeSyViT and the Vanilla ViT. Bold values are better.
| P2 | $-4$ | +1 | $12$ | 2 |
| --- | --- | --- | --- | --- |
| P3.1 | $-7$ | +1 | $16$ | 3 |
| P3.2 | $-4$ | +2 | $7$ | 3 |
| P3.3 | $-7$ | +7 | $23$ | 5 |
| P5 | $-15$ | +8 | $30$ | 5 |
| P10 | $-21$ | +18 | $65$ | 14 |
| GT43 | $-13$ | -1 | $99$ | 51 |
| Mean Stats | $-10.14$ | +5.14 | $36$ | 11.85 |
Table 2: Comparison of relative % accuracy change w.r.t. the vanilla model and rule-set size between NeSyFOLD and NeSyViT.
Result: Table 1 presents a comparison between the NeSy model generated using NeSyViT and the Vanilla ViT. All metrics are reported over five independent runs. Accuracy is computed on the test set for both models. The âRulesâ column reports the average and standard deviation of the number of rules generated. âUnique Predicatesâ indicates the number of distinct predicates used across the rule-set, and âSizeâ refers to the total number of predicate occurrences in the bodies of all rules. Recall that we use rule-set size as a proxy for interpretabilityâsmaller rule-sets are generally easier to understand.
The most significant observation is that our method achieves higher average accuracy than the Vanilla ViT. This is particularly noteworthy, as extracting interpretable rule-sets by binarizing internal representations typically results in some loss of information and a subsequent drop in performance. However, our approach mitigates this issue through the use of an entropy loss, which encourages the outputs of the sparse concept layer to be close to either $0 0$ or $1$ during training. As a result, thresholding these outputs at $0.5$ post-training introduces minimal distortion. Furthermore, the supervised contrastive loss ensures that the resulting binary vectors are well-clustered by class, enabling FOLD-SE-M to learn accurate and class-discriminative rule-sets. Finally, the L1 sparsity loss encourages only a small subset of neurons to activate for each image, resulting in compact and interpretable rule-sets. Notice that for the P10 dataset, the accuracy improvement over vanilla is 18%, which is a huge improvement and shows the merit of our approach when scaling to larger number of classes.
To provide context for the significance of our results, Table 2 presents the percentage change in accuracy of the NeSy model relative to its vanilla counterpart for both NeSyViT and NeSyFOLD (which uses a CNN instead of a ViT). We also report the average rule-set size generated by each framework. Notably, NeSyViT improves upon the accuracy of the Vanilla ViT by an average of 5.14% âa substantial gain, especially when contrasted with NeSyFOLD, which suffers an average accuracy drop of 10.14% compared to the vanilla CNN. Additionally, NeSyFOLD shows a consistent degradation in performance as the number of classes increases, largely due to the information loss introduced by post-hoc binarization. In contrast, our method consistently yields accuracy improvements across datasets, with the sole exception of GT43, where we observe a minor drop of 1 percentage point.
Furthermore, the rule-sets generated by NeSyViT are significantly more compact, with an average size of $11.85$ , which is 67% smaller than those produced by NeSyFOLDâfurther emphasizing the interpretability of our approach.
<details>
<summary>x1.png Details</summary>
### Visual Description
\n
## Heatmap Grid: Room Classification Rules
### Overview
The image presents a 3x3 grid of heatmaps overlaid on interior room scenes, accompanied by two blocks of text detailing rule sets for classifying these rooms. The heatmaps visually represent areas of focus or "attention" within each image, likely indicating features used for classification. The rule sets provide a logical framework for identifying rooms as 'bedroom', 'bathroom', or 'kitchen'.
### Components/Axes
The image consists of:
* **Nine Heatmap Images:** Each image depicts a different room scene with a heatmap overlay.
* **Numerical Labels:** Each heatmap image has a numerical label in the top-left corner (106 (X), 43 (X), 105 (X)).
* **Two Text Blocks:** Located at the bottom of the image, these blocks contain the "RAW RULE-SET P3.1" and "LABELLED RULE-SET P3.1".
* **Rule Set Structure:** Each rule set consists of numbered rules with a specific format: `target(X, 'room_type') :- value (X).`
### Detailed Analysis or Content Details
**Heatmap Images:**
The heatmaps use a color gradient, ranging from blue (low attention) to red/yellow (high attention). The intensity and location of the heatmaps vary across the images, suggesting different features are important for classifying each room.
* **Top Row:**
* Image 1 (106 (X)): Heatmap focuses on the bed and surrounding area.
* Image 2: Heatmap focuses on the sink and counter area.
* Image 3: Heatmap focuses on the window and plants.
* **Middle Row:**
* Image 4 (43 (X)): Heatmap focuses on the toilet and surrounding area.
* Image 5: Heatmap focuses on the sink, counter, and mirror.
* Image 6: Heatmap focuses on the plants and window.
* **Bottom Row:**
* Image 7 (105 (X)): Heatmap focuses on the counter and stove area.
* Image 8: Heatmap focuses on the sink and counter area.
* Image 9: Heatmap focuses on the counter and dining table.
**RAW RULE-SET P3.1:**
1. `target(X, 'bedroom') :- 106(X).` - If X is 106, then the target is 'bedroom'.
2. `target(X, 'bathroom') :- not 43(X).` - If X is not 43, then the target is 'bathroom'.
3. `target(X, 'kitchen') :- 105(X).` - If X is 105, then the target is 'kitchen'.
**LABELLED RULE-SET P3.1:**
1. `target(X, 'bedroom') :- 106(X).` - If X is 106, then the target is 'bedroom'.
2. `target(X, 'bathroom') :- not water_cooler(X).` - If X is not a water cooler, then the target is 'bathroom'.
3. `target(X, 'kitchen') :- 105(X).` - If X is 105, then the target is 'kitchen'.
### Key Observations
* The numerical labels (106, 43, 105) appear to be identifiers for specific room scenes.
* The RAW rule set uses these numerical identifiers directly for classification.
* The LABELLED rule set introduces a new feature, 'water\_cooler', suggesting a more sophisticated classification approach.
* The rule `target(X, 'bathroom') :- not 43(X).` in the RAW rule set is potentially problematic, as it classifies anything *not* 43 as a bathroom, which is overly broad.
* The LABELLED rule set attempts to refine this by using a more specific feature ('water\_cooler').
### Interpretation
The image demonstrates a rule-based approach to room classification using visual features represented by heatmaps. The RAW rule set provides a simple, albeit potentially inaccurate, classification scheme based on numerical identifiers. The LABELLED rule set attempts to improve accuracy by incorporating more meaningful features like the presence or absence of a 'water\_cooler'.
The heatmaps themselves suggest that the system is focusing on key objects and areas within each room to make its classification decisions. For example, the heatmap in the bedroom image focuses on the bed, while the heatmap in the kitchen image focuses on the stove.
The difference between the RAW and LABELLED rule sets highlights the importance of feature engineering in machine learning. By moving from simple numerical identifiers to more descriptive features, the system can potentially achieve higher accuracy and robustness. The introduction of 'water\_cooler' suggests an attempt to address the limitations of the initial rule set and improve the classification of bathrooms. The use of "not" in the rules indicates a negative constraint, which can be useful but also prone to errors if not carefully considered.
</details>
Figure 2: a) The top-5 images overlayed with activation heat-maps for neurons 106, 43 and 105 when rules are extracted for the P3.1 dataset containing classes âbathroomâ, âbedroomâ and âkitchenâ. b) The raw rule-set and the labelled rule-set when NeSyViT is employed on the P3.1 dataset.
[Q4, Q5] Neuron Activation Patterns and Automatic Semantic Labelling Efficacy: A key component of interpretability in neuro-symbolic frameworks is the alignment of individual neurons with semantically meaningful concepts relevant to specific classes.
Setup: To evaluate this alignment, we apply the previously described, semantic labelling algorithm to assign human-interpretable concept names to each predicate in the rule-set, where each predicate corresponds to a neuron in the final sparse concept layer. Initially, the raw rule-set produced by the FOLD-SE-M algorithm uses only neuron indices as predicate names. The semantic labelling process replaces these with descriptive concept labels derived by analyzing the top-10 most activating images per neuron. These images are overlaid with attention-based heatmaps and compared with pre-annotated semantic segmentation masks to identify the dominant concept for each neuron.
In Figure 2 a), we visualize the top- $5$ most activating images for each predicate in the labelled rule-set for the P3.1 dataset, which contains the classes âbathroomâ, âbedroomâ, and âkitchen.â Figure 2 b) shows a side-by-side comparison of the raw and semantically labelled rule-sets, illustrating how the abstract neuron indices are transformed into interpretable concept-based rules.
Result: Examining the labelled rule-set in Figure 2 b), we observe that just three rules are sufficient to achieve a classification accuracy of $99\%$ . Consider Rule 2: target(X, âbathroomâ) :- not water_cooler1_tray1_refrigerator1_range1(X). At first glance, this rule may appear counterintuitiveâwhy would the absence of several kitchen-related concepts imply that the image depicts a bathroom rather than a bedroom? The explanation lies in the execution strategy of FOLD-SE-M, which evaluates rules in a top-down fashion. Rule 2 only activates if Rule 1 fails to fire. Rule 1 is: target(X, âbedroomâ) :- bed1(X). Thus, if the image lacks the âbedâ concept, Rule 1 is bypassed, and Rule 2 checks for the absence of kitchen concepts; if none are present, the image is classified as a âbathroom.â Rule 3: target(X, âkitchenâ) :- refrigerator2(X) fires only if the previous two rules do not apply and the ârefrigeratorâ concept is present, thereby classifying the image as kitchen.â
While the logical flow of the rules is valid, the quality of semantic labelling is less reliable. The labelling pipeline we adapted from NeSyFOLD was designed for CNNs, where modular filters make concept isolation more straightforward. In ViTs, however, neurons often attend to multiple regions and concepts simultaneously, making the learned representations less disentangled. As a result, some neuron labels can be misleading. This limitation is evident in Figure 2 a): neuron $106$ consistently activates for the âbedâ concept in bedroom images, but neurons $43$ and $105$ do not display clear concept selectivity and instead respond to mixed features. This highlights the challenge of isolating concepts in self-attention-based architectures.
While the automatic labelling approach may have limitations, the visualization of neuron activation heatmaps remains a powerful tool. These heatmaps allow for manual inspection of concept associations and provide valuable interpretability cues. We believe this combination of automated rule generation and visual validation offers a promising direction. Also, since the extracted rule-set is an Answer Set Program we can get justification of any prediction using the s(CASP) (Arias et al., (2018)) ASP interpreter. For completeness, we provide the labelled rule-sets for all datasets used in our experiments in Table 1, in the Appendix.
5 Related Works
The integration of symbolic reasoning with deep learning has led to various approaches for extracting interpretable rules from neural networks. While initial efforts focused on convolutional neural networks (CNNs), recent studies have begun exploring Vision Transformers (ViTs). VisionLogic (Geng et al., (2025)) introduces a framework that transforms neurons in the final fully connected layer of deep vision models into predicates, grounding them into visual concepts through causal validation. ViT-NeT (Kim et al., (2022)) presents a neural tree decoder that interprets the decision-making process of ViTs. By routing images hierarchically through a tree structure, it provides transparent and interpretable classifications, addressing the trade-off between model complexity and interoperability. These methods primarily focus on post-hoc interpretability or rely on additional structures for explanation. In contrast, our approach integrates a sparse linear layer directly into the ViT architecture, enabling the extraction of executable logic programs without auxiliary components.
Enhancing the interpretability of ViTs has been a subject of extensive research, leading to various methodologies. INTR (Paul et al., (2024)) proposes a proactive approach, asking each class to search for itself in an image. This idea is realized via a Transformer encoder-decoder where âclass-specificâ queries (one for each class) are learnt as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. Each class is attended to very distinctly hence the cross-attention weights provide a interpretation of the prediction. LeGrad (Bousselham et al., (2024)) introduces an explainability method specifically designed for ViTs. By computing gradients with respect to attention maps and aggregating signals across layers, LeGrad produces explainability maps that offer insights into the modelâs focus areas during decision-making. While these approaches enhance the transparency of ViTs, they do not produce symbolic or executable explanations. Our method differs by generating logic programs that serve as the final decision layer, enabling symbolic reasoning directly from the modelâs internal representations.
The pursuit of sparse and disentangled representations in ViTs has led to innovative methodologies. Recent work integrates sparse autoencoders (SAEs) with Vision Transformers (ViTs) to improve interpretability. Joseph et al., (2025) trained SAEs on CLIPâs ViT and found that manipulating a small set of steerable features can enhance performance and robustness. PatchSAE (Lim et al., (2025)) introduces method to extract granular visual concepts and study how these influence predictions, showing that most adaptation gains stem from existing features in the pre-trained model. Our approach builds upon these concepts by integrating a sparse linear layer into the ViT architecture and using it for extracting a symbolic rule-set which gives a global explanation of the model and a justification can be obtained for each prediction.
6 Conclusion and Future Work
In this work, we proposed a novel neuro-symbolic framework for interpretable image classification using Vision Transformers (ViTs). By replacing the final classification layer with a linear layer (sparse concept layer) producing sigmoid outputs, and introducing a unified loss functionâcomprising supervised contrastive loss for better class separation in latent space, entropy loss for sharper binarization, and sparsity loss for concept selectivityâwe generated sparse binary vectors that represent images in a disentangled manner. These vectors serve as inputs to the FOLD-SE-M algorithm, which produces stratified Answer Set Programs in the form of concise rule-sets. The resulting neuro-symbolic (NeSy) model, composed of the modified ViT and the learned rule-set, outperforms the vanilla ViT by an average of 5.14% in accuracy.
We compared our framework, NeSyViT, with NeSyFOLD, a similar approach for CNNs, and found that NeSyViT produces rule-sets that are 67% smaller on average while improving upon the baseline ViTâs accuracyâwhereas NeSyFOLD suffers a 10.14% drop in performance compared to its CNN counterpart. These results demonstrate the potential of attention-based architectures for interpretable, logic-driven classification.
Finally, we investigated the semantic interpretability of neurons in our framework. We observed that the automatic semantic labelling algorithm used in NeSyFOLDâbased on overlap with segmentation masksâstruggles in ViTs due to limited neuron monosemanticity. However, we argue that this challenge can be partially mitigated by visualizing attention-guided heatmaps for the small set of neurons that appear in the rule-set, enabling manual inspection and insight into the learned concepts.
Future work will focus on improving neuron disentanglement and enhancing monosemanticity using architectural or training refinements. Another promising direction explored in CNN-based interpretability methods is bias correction using extracted rule-sets Padalkar et al., 2024c which could be the natural next step for this work. We also plan to explore using multimodal LLMs like GPT-4o for concept labeling, enabling an automatic semantic annotation pipeline that does not rely on pixel-level segmentation masks.
References
- Arias et al., (2018) Arias, J., Carro, M., Salazar, E., Marple, K., and Gupta, G. Constraint answer set programming without grounding. In Theory and Practice of Logic Programming 2018, volume 18, pp. 337â354.
- Baral, (2003) Baral, C. 2003. Knowledge representation, reasoning and declarative problem solving. Cambridge University Press.
- Bousselham et al., (2024) Bousselham, W., Boggust, A., Chaybouti, S., Strobelt, H., and Kuehne, H. 2024. Legrad: An explainability method for vision transformers via feature formation sensitivity.
- Dosovitskiy et al., (2021) Dosovitskiy, A., Beyer, L., and others. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR 2021 2021.
- Gelfond and Kahl, (2014) Gelfond, M. and Kahl, Y. 2014. Knowledge representation, reasoning, and the design of intelligent agents: The answer-set programming approach. Cambridge University Press.
- Geng et al., (2025) Geng, C., Jiang, Y., Zhao, Z., Ye, H., Wang, Z., and Si, X. Learning interpretable logic rules from deep vision models. In arXiv 2503.10547 2025.
- Joseph et al., (2025) Joseph, S., Suresh, P., and Others. Steering clipâs vision transformer with sparse autoencoders. In arXiv 2504.08729 2025.
- Kanagaraj et al., (2021) Kanagaraj, N., Hicks, D., and Others. Deep learning using computer vision in self driving cars for lane and traffic sign detection. In Inter. Journal of Sys. Assurance Engg. and Management 2021, volume 12, pp. 1011â1025.
- Khosla et al., (2020) Khosla, P., Teterwak, P., and others. Supervised contrastive learning. In NeurIPS 2020.
- Kim et al., (2022) Kim, S., Nam, J., and Ko, B. C. Vit-net: Interpretable vision transformers with neural tree decoder. In International conference on machine learning 2022, pp. 11162â11172. PMLR.
- Ko and Kwak, (2012) Ko, B. and Kwak, S. 2012. Survey of computer vision-based natural disaster warning systems.
- Lage et al., (2019) Lage, I., Chen, E., and others. Human evaluation of models built for interpretability. In Proc. HCOMP 2019, volume 7, pp. 59â67.
- Lim et al., (2025) Lim, H., Choi, J., and Others. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In ICLR 2025.
- Ng et al., (2011) Ng, A. and others 2011. Sparse autoencoder.
- (15) Padalkar, P., Wang, H., and Gupta, G. Nesyfold: A framework for interpretable image classification. In Proc. AAAI 2024a, pp. 4378â4387. AAAI Press.
- (16) Padalkar, P., Wang, H., and Gupta, G. Using logic programming and kernel-grouping for improving interpretability of convolutional neural networks. In Proc. PADL 2024b, volume 14512 of LNCS, pp. 134â150. Springer.
- (17) Padalkar, P., Ćlusarz, N., Komendantskaya, E., and Gupta, G. 2024c. A neurosymbolic framework for bias correction in convolutional neural networks, 24 c, 4, 644â662.
- Paul et al., (2024) Paul, D., Chowdhury, A., and Others 2024. A simple interpretable transformer for fine-grained image classification and analysis.
- Stallkamp and Others, (2012) Stallkamp, J. and Others. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. In Neural Networks 2012, volume 32, pp. 323â332.
- Sun et al., (2016) Sun, W., Zheng, B., and Qian, W. Computer aided lung cancer diagnosis with deep learning algorithms. In SPIE Medical Imaging 2016.
- Townsend et al., (2021) Townsend, J., Kasioumis, T., and Inakoshi, H. Eric: Extracting relations inferred from convolutions. In Computer Vision â ACCV 2021, pp. 206â222, Cham. Springer International.
- Wang and Gupta, (2024) Wang, H. and Gupta, G. FOLD-SE: an efficient rule-based machine learning algorithm with scalable explainability. In Proc. PADL 2024, volume 14512 of LNCS, pp. 37â53. Springer.
- Zhou and Others, (2018) Zhou, B. and Others. Places: A 10 million image database for scene recognition. In IEEE Trans. Pattern Anal. Mach. Intell. 2018, volume 40, pp. 1452â1464.
Appendix
Our paper above shows the labeled rule-set for only one dataset i.e. P3.1 so here we show the labelled rule-sets for all the other datasets as well. Also we show the hyperparameters used for training and the rule generation using FOLD-SE-M.
Appendix A Labelled rule-sets
P2 {âbathroomâ, âbedroomâ}: target(X,âbathroomâ) :- bathroom_tiles1_shower_screen1(X). target(X,âbedroomâ) :- bed1_duvet1(X).
P3.1 {âbathroomâ, âbedroomâ, âkitchenâ}: target(X,âbedroomâ) :- not shower_screen1(X). target(X,âbathroomâ) :- not refrigerator1_kitchen_island1_range1_countertop1_person1_wall1_air_conditioning1(X). target(X,âkitchenâ) :- not bathtub1_mirror1(X).
P3.2 {âdesert roadâ, âforest roadâ, âstreetâ}: target(X,âdesert_roadâ) :- field1_grass1_desert1(X). target(X,âstreetâ) :- food_cart1(X). target(X,âforest_roadâ) :- not building1(X).
P3.3 {âdesert roadâ, âdrivewayâ, âhighwayâ}: target(X,âhighwayâ) :- not buildings1(X). target(X,âdrivewayâ) :- not hill1_ground3(X). target(X,âdesert_roadâ) :- field1_ground1(X). target(X,âhighwayâ) :- buildings1(X). target(X,âdesert_roadâ) :- ground2(X).
P5 {âbathroomâ, âbedroomâ, âkitchenâ, âdining roomâ, âliving roomâ}:
target(X,âbathroomâ) :- toilet1(X). target(X,âliving_roomâ) :- fireplace1(X). target(X,âdining_roomâ) :- not bed2_railing1(X). target(X,âbedroomâ) :- bed1(X). target(X,âkitchenâ) :- top1(X).
P10 {âbathroomâ, âbedroomâ, âkitchenâ, âdining roomâ, âliving roomâ, âhome officeâ, âofficeâ, âwaiting roomâ, âconference roomâ, âhotel roomâ}: target(X,âbedroomâ) :- bed1(X), not bed2(X). target(X,âbathroomâ) :- shower_screen1_mirror1(X). target(X,âdining_roomâ) :- tablecloth1_bar1(X), not work_surface1_front1_trash_can1(X). target(X,âofficeâ) :- tablecloth1_bar1(X). target(X,âliving_roomâ) :- shelves1(X), not table1_lectern1_floor2_top1(X). target(X,âconference_roomâ) :- shelves1(X). target(X,âkitchenâ) :- floor1(X). target(X,âhotel_roomâ) :- sofa1_seat1_armchair1_door1_wall1(X). target(X,âhome_officeâ) :- bed1(X). target(X,âwaiting_roomâ) :- bed2(X), not bathtub1(X).
Appendix B Neuron Attention-Maps for P10
label(X,âbedroomâ) :- 14(X), not 83(X). label(X,âbathroomâ) :- 81(X). label(X,âdining_roomâ) :- 122(X), not 97(X). label(X,âofficeâ) :- 122(X). label(X,âliving_roomâ) :- 43(X), not 45(X). label(X,âconference_roomâ) :- 43(X). label(X,âkitchenâ) :- 41(X). label(X,âhotel_roomâ) :- 50(X). label(X,âhome_officeâ) :- 14(X). label(X,âwaiting_roomâ) :- 83(X), not 5(X).
Appendix C Hyperparameters
NeSyViT: For the subsets of the Places dataset i.e. P2, P3.1, P3.2, P3.3, P5, P10 We used a learning rate of $5Ă 10^{-6}$ and l2 weight decay of $5e^{-3}$ with the AdamW optimizer, batch size of $32$ , patience of $10$ epochs for the LR-Scheduler, LR decay factor of $0.5$ .
The weights for the supcon loss, sparsity loss and entropy loss were $2,1$ and $1$ respectively.
For the GT43 dataset we changed the learning rate to $5Ă 10^{-5}$ and the weights for the supcon loss, sparsity loss and entropy loss were $4,0.001$ and $0.001$ respectively. Every other hyperparameter was the same.
For all datasets we used a ratio of $0.8$ and a tail of $5e^{-3}$ for the FOLD-SE-M algorithm.
Vanilla ViT: For all datasets we used a learning rate of $5Ă 10^{-6}$ , batch size of $32$ , patience of $10$ epochs for the LR-Scheduler, LR decay factor of $0.5$ and l2 weight decay of $5e^{-3}$ . The loss used was standard crossentropy loss.