2510.14538

# Symbol Grounding in Neuro-Symbolic AI: A Gentle Introduction to Reasoning Shortcuts **Authors**: - Emanuele Marconato (University of Trento) - &Samuele Bortolotti (University of Trento) - &Emile van Krieken - Vrije Universiteit Amsterdam - The Netherlands - Paolo Morettin (University of Trento) - &Elena Umili (Sapienza University of Rome) - &Antonio Vergari (University of Edinburgh) - United Kingdom - Efthymia Tsamoura (Huawei Labs) - Cambridge United Kingdom - &Andrea Passerini (University of Trento) - &Stefano Teso (University of Trento) > ∗ Corresponding author. † \dagger Equal contribution. ‡ \ddagger Work started before Efthymia Tsamoura joined Huawei Labs. ## Abstract Neuro-symbolic (NeSy) AI aims to develop deep neural networks whose predictions comply with prior knowledge encoding, e.g., safety or structural constraints. As such, it represents one of the most promising avenues for reliable and trustworthy AI. The core idea behind NeSy AI is to combine neural and symbolic steps: neural networks are typically responsible for mapping low-level inputs into high-level symbolic concepts, while symbolic reasoning infers predictions compatible with the extracted concepts and the prior knowledge. Despite their promise, it was recently shown that – whenever the concepts are not supervised directly – NeSy models can be affected by Reasoning Shortcuts (RSs). That is, they can achieve high label accuracy by grounding the concepts incorrectly. RSs can compromise the interpretability of the model’s explanations, performance in out-of-distribution scenarios, and therefore reliability. At the same time, RSs are difficult to detect and prevent unless concept supervision is available, which is typically not the case. However, the literature on RSs is scattered, making it difficult for researchers and practitioners to understand and tackle this challenging problem. This overview addresses this issue by providing a gentle introduction to RSs, discussing their causes and consequences in intuitive terms. It also reviews and elucidates existing theoretical characterizations of this phenomenon. Finally, it details methods for dealing with RSs, including mitigation and awareness strategies, and maps their benefits and limitations. By reformulating advanced material in a digestible form, this overview aims to provide a unifying perspective on RSs to lower the bar to entry for tackling them. Ultimately, we hope this overview contributes to the development of reliable NeSy and trustworthy AI models. Contents 1. 1 Introduction 1. 2 Preliminaries 1. 2.1 From Neuro-Symbolic AI to Neuro-Symbolic Predictors 1. 2.2 The Variety of NeSy Predictor Architectures 1. 2.3 NeSy Predictor Architectures: Differences and Similarities 1. 2.4 NeSy Predictors: Benefits 1. 3 A Gentle Introduction to Reasoning Shortcuts 1. 3.1 Causes, Frequency and Impact 1. 4 Theory of Reasoning Shortcuts 1. 4.1 Setting and Assumptions 1. 4.2 Perspective: Identification 1. 4.2.1 Model class of NeSy predictors 1. 4.2.2 Non-identifiability of the concept extractor 1. 4.2.3 Concept remapping distributions. 1. 4.2.4 Reasoning shortcuts as unintended, optimal concept remapping distributions 1. 4.2.5 Characterization of deterministic reasoning shortcuts. 1. 4.3 Perspective: Statistical Learning 1. 4.3.1 Knowledge complexity 1. 4.3.2 Impossibility result in bounding the reasoning shortcut risk 1. 4.3.3 Setting 1. 4.3.4 PAC learnability with $k$ -unambiguous inference layers 1. 4.4 Relationship between Theories 1. 5 Handling Reasoning Shortcuts 1. 5.1 Root Causes 1. 5.1.1 The knowledge 1. 5.1.2 The training distribution 1. 5.1.3 The optimality condition 1. 5.1.4 The family of learnable maps 1. 5.2 How to Diagnose Reasoning Shortcuts 1. 5.2.1 Task-level diagnosis 1. 5.2.2 Model metrics 1. 5.3 A Tour of Mitigation Strategies 1. 5.3.1 Concept supervision 1. 5.3.2 Multi-task learning 1. 5.3.3 Abductive weak supervision 1. 5.3.4 Entropy maximization 1. 5.3.5 Smoothing 1. 5.3.6 Reconstruction 1. 5.3.7 Contrastive learning 1. 5.3.8 Architectural disentanglement 1. 5.3.9 Which mitigation strategy is best? 1. 5.3.10 Can one simply recover learned concepts? 1. 5.4 Awareness Strategies 1. 5.4.1 RS-awareness as mixtures of deterministic RSs 1. 5.4.2 Building RS-aware NeSy predictors 1. 5.4.3 RS-Awareness via Ensembles 1. 5.4.4 RS-Awareness via Diffusion 1. 5.5 Awareness Helps Mitigation 1. 6 Extensions and Open Problems 1. 6.1 Reasoning Shortcuts in NeSy AI Beyond Predictors 1. 6.2 Reasoning Shortcuts in Concept-based Models 1. 6.3 Foundation and Large Language Models 1. 6.4 Reasoning Shortcuts in Reinforcement Learning 1. 6.5 Imbalanced Learning 1. 6.6 Additional open problems 1. 7 Related Work 1. 8 Conclusion ## 1 Introduction In cognitive science and psychology, symbols —or more generally, concepts —are the compositional building blocks of human thought (Mandler, 2004; Spelke and Kinzler, 2007). They enable the abstraction and structuring of perception, supporting high-level cognitive functions (Whitehead, 1927; Johnson-Laird, 1994) such as language, reasoning, and planning. In recent years, there has been growing interest in using concepts in neural networks (Smolensky, 1987; Sun, 1992; Greff et al., 2020), which bring not only improved generalization but also greater interpretability to their decision-making processes. While the emergence of abstract concepts in the human brain is well-documented (Mandler, 2004), how—and even whether—such entities arise in neural networks remains fundamentally unclear (Jo and Bengio, 2017; Lake and Baroni, 2018; Park et al., 2024). Neuro-symbolic (NeSy) models aim to bridge this gap by integrating neural learning with symbolic reasoning (Harnad, 1990; McMillan et al., 1991; Sun, 1992), thereby enabling explicit manipulation of concepts within the inference process. Some approaches embed prior symbolic structure—such as logical rules (De Raedt et al., 2021) —while others attempt to learn such structure through specialized, trainable components (Lake and Baroni, 2018; Ellis et al., 2021). A key promise of this integration is reliability and trustworthiness: by making all decisions traceable to interpretable, well-defined concepts, decisions become more transparent (Kambhampati et al., 2022). Moreover, thanks to the modularity of symbolic components, learned concepts can be reused in novel scenarios without significant performance degradation. Yet, the effectiveness of this integration hinges on solving a core challenge in NeSy AI: the problem of symbol grounding (Harnad, 1990) —that is, how to connect low-level perceptual data with high-level, abstract concepts. **Example 1.1** *Consider a model trained on autonomous driving scenarios, where the car must decide whether to stop or go based on the visual input, as per Fig. 1. The model leverages prior knowledge $K_1$ which imposes that whenever pedestrian or red_light are detected, the car must stop. Attaining accurate action predictions requires assigning valid binary values to the concepts pedestrian and red_light (among others) from the visual input. Therefore, concepts are grounded by the model through learning to give correct driving predictions. Because pedestrian and red_light are treated symmetrically by the knowledge (due to the disjunction) We will precisely define the symmetries of the knowledge in later sections. the self-driving car may confuse the two while still achieving correct predictions (e.g., in Fig. 1 (Left), red_light fires when either “pedestrians” or “red lights” are detected). This example shows that model concepts can be grounded differently from their intended meaning: training the autonomous car to give correct predictions does not guarantee that concepts activate when they should; e.g., the autonomous car’s red_light may not solely contain information about the presence of “red lights” in the raw sensory data.* Footnote 1 illustrates the central challenge of symbol grounding in Neuro-Symbolic AI: ensuring that abstract predicted concepts inside the model maintain a consistent and semantically correct link to the real-world entities they are meant to represent. Unfortunately, in many learning contexts NeSy models— even when they perform nearly optimally on a task—may fail to assign the right meaning to learned concepts, leading to concepts with unintended semantics (Marconato et al., 2023a, b). This issue originates from reasoning shortcuts (RSs): unwanted concept assignments that allow models to reach correct label predictions. These have been shown to affect many NeSy models (Marconato et al., 2023b; Yang et al., 2024). While the decision-making appears correct on the surface, the underlying concepts are flawed, undermining the reliability and trustworthiness of the NeSy model. This directly impacts the model’s interpretability, as well as generalization when concepts are reused in out-of-distribution scenarios and continual learning. For example, in the context of autonomous driving of Footnote 1 and of Fig. 1, the correct “ stop ” prediction can be achieved by mistaking “ pedestrians ” with “ red lights ”, as both concepts induce the same decision. While this misalignment does not induce a drop in prediction on data in-distribution, it becomes critical in situations where such distinctions matter. In fact, poorly grounded concepts do not transfer to out-of-distribution scenarios, as displayed in Fig. 1 (Right), where the wrong activation of model concepts leads to a faulty prediction. RSs were observed in early NeSy work (Manhaeve et al., 2018; Chang et al., 2020; Topan et al., 2021), but only recently received a formal treatment, revealing that RSs are hard to tackle (Marconato et al., 2023b; Umili et al., 2024a; Yang et al., 2024; DeLong et al., 2024), despite arising naturally from certain symmetries in the symbolic component of NeSy systems. In turn, theoretical studies (Marconato et al., 2023b; Wang et al., 2023; Yang et al., 2024; Bortolotti et al., 2025) have begun to identify under which conditions RSs can be provably avoided. This often requires pairing standard NeSy training methods with mitigation strategies that encourage better concept grounding. However, designing effective mitigation strategies that do not require a significant annotation cost remains an open problem. In this paper, we aim to consolidate the scattered literature on RSs in NeSy AI and offer a comprehensive overview of different approaches to mitigating them. Contributions This article provides a gentle introduction to reasoning shortcuts, unifies the existing literature on the topic, and connects it to the well-known symbol grounding problem. We provide a general perspective on RSs, highlighting that they cannot be avoided simply by designing different neuro-symbolic architectures. To this end, we compile all relevant theory on RSs through the perspectives of identifiability and statistical learning. We then present known mitigation strategies to prevent RSs using a taxonomy, and explain which strategies can effectively reduce the likelihood of RSs. Finally, we identify open problems and future directions, showing that RSs — and thus correct symbol grounding — extend beyond the NeSy models studied so far. <details> <summary>figures/nesy-road-kill.png Details</summary>  ### Visual Description \n ## Diagram: Logical Rules for Traffic Decision-Making ### Overview The image is a technical diagram illustrating two sequential tasks (Task 1 and Task 2) that apply logical rules to autonomous driving or traffic control scenarios. It uses a combination of visual scenes, logical formulas, and decision outputs to demonstrate how a system should behave under different conditions involving traffic lights, pedestrians, and emergency vehicles. The diagram is divided into two main sections separated by a vertical dotted line. ### Components/Axes 1. **Task Labels:** * **Left Side:** Vertical text "Task 1" runs along the left edge. * **Right Side:** Vertical text "Task 2" runs along the right edge. 2. **Logical Formulas (Top):** * **Task 1 Formula:** `K₁ = (pedestrian ∨ red ⇒ stop)` * **Task 2 Formula:** `K₂ = (emergency ∧ ¬pedestrian ⇒ go) ∧ K₁` * **Notation:** Uses logical symbols: `∨` (OR), `∧` (AND), `¬` (NOT), `⇒` (implies). 3. **Visual Scenes (Panels):** * **Task 1 - Left Panel:** A road scene with mountains in the background. A traffic light is highlighted with a magenta bounding box and labeled "red" in magenta text. The red light is illuminated. * **Task 1 - Right Panel:** A similar road scene. A traffic light is highlighted and labeled "green" in magenta text; the green light is illuminated. A pedestrian (stick figure) is highlighted with a magenta bounding box and labeled "red" in magenta text. * **Task 2 - Single Panel:** A road scene with snow-capped mountains. A pedestrian is highlighted and labeled "red" in magenta text. On the left side of the road, an emergency vehicle (red, with flashing lights depicted by red lines) is present. 4. **Decision Outputs (Bottom of each panel):** * **Task 1, Left Panel:** `y = stop` and `ŷ = stop` followed by a green checkmark (✓). * **Task 1, Right Panel:** `y = stop` and `ŷ = stop` followed by a green checkmark (✓). * **Task 2, Panel:** `ŷ = go` followed by a red cross (✗). 5. **Spatial Layout:** * The two panels for Task 1 are positioned side-by-side on the left half of the image. * The single panel for Task 2 occupies the right half. * Magenta bounding boxes and labels are used consistently to highlight key elements (traffic lights, pedestrians) within the scenes. * The logical formulas are centered above their respective task sections. ### Detailed Analysis * **Task 1 Analysis:** * **Rule (K₁):** The system must stop if there is a pedestrian OR if the traffic light is red. * **Left Panel Scenario:** Traffic light is red, no pedestrian is present. The condition `red` is true, so the rule dictates `stop`. The output shows the ground truth `y = stop` and the system's prediction `ŷ = stop`, marked as correct (✓). * **Right Panel Scenario:** Traffic light is green, but a pedestrian is present. The condition `pedestrian` is true, so the rule dictates `stop`. The output again shows `y = stop` and `ŷ = stop`, marked as correct (✓). * **Trend/Flow:** Task 1 establishes a baseline safety rule. The system correctly applies the "stop" decision in both scenarios where at least one of the triggering conditions (red light or pedestrian) is met. * **Task 2 Analysis:** * **Rule (K₂):** This rule builds on K₁. It adds a new condition: the system should `go` if there is an emergency vehicle AND there is NO pedestrian. This rule is conjoined (`∧`) with the original rule K₁. * **Scenario:** An emergency vehicle is present, and a pedestrian is also present (labeled "red"). * **Logical Evaluation:** According to the new clause `(emergency ∧ ¬pedestrian ⇒ go)`, the condition `¬pedestrian` (no pedestrian) is false because a pedestrian is present. Therefore, this specific "go" rule does not apply. The system should fall back to the original rule K₁. Since a pedestrian is present, K₁ dictates `stop`. * **Output:** The diagram shows the system's prediction as `ŷ = go`, which is marked with a red cross (✗), indicating an error. This suggests the system incorrectly prioritized the emergency vehicle condition over the pedestrian presence, violating the conjoined logic of K₂. ### Key Observations 1. **Consistent Labeling:** Magenta text and bounding boxes are used to draw attention to the critical decision-making elements in each scene: the state of the traffic light and the presence of a pedestrian. 2. **Symbolic Logic:** The diagram explicitly defines the decision-making rules using formal logic notation, making the expected behavior unambiguous. 3. **Error Demonstration:** Task 2 is specifically designed to show a failure case. The system's incorrect "go" prediction (`ŷ = go`) contradicts the logical rule K₂, which should have resulted in a "stop" due to the presence of the pedestrian. 4. **Visual Confirmation:** The green checkmarks (✓) and red cross (✗) provide immediate visual feedback on the correctness of the system's prediction relative to the defined rules. ### Interpretation This diagram serves as a technical specification and a test case for a rule-based decision system, likely for an autonomous vehicle. It demonstrates the process of **incremental rule definition and validation**. * **What it demonstrates:** It shows how complex behavior can be built by conjoining simpler logical rules (K₂ includes K₁). It highlights the critical importance of **rule precedence and conjunction**. The error in Task 2 reveals a potential flaw in the system's implementation: it may be treating the emergency vehicle rule as an overriding exception that ignores other safety conditions (like pedestrians), rather than as a rule that must be evaluated in conjunction with all other active rules. * **Why it matters:** In safety-critical systems like autonomous driving, formal verification against logical specifications is essential. This diagram illustrates a method for defining expected behavior and testing for violations. The depicted error is a serious one, as it could lead to a vehicle moving when it should stop for a pedestrian, even in the presence of an emergency vehicle. The diagram argues for the necessity of systems that can correctly evaluate compound logical conditions in real-time. * **Underlying Message:** The core message is about the **hierarchy and integration of rules**. Safety rules involving vulnerable road users (pedestrians) are typically fundamental and should not be overridden by situational rules (emergency vehicle priority) unless explicitly and safely designed to do so (e.g., after ensuring the pedestrian is clear). The conjoined logic `∧ K₁` in the formula for K₂ is meant to enforce this, but the system's failure shows it was not properly implemented. </details> Figure 1: Reasoning shortcuts are failures of correct symbol grounding. Consider an autonomous driving car that must drive in compliance with traffic rules. (Left) The car learns to make correct predictions by leveraging background knowledge $K_1$ in Task $1$ , i.e., “if there is a pedestrian or a red light the car has to stop”. However, in the process, it may incorrectly associate the concept of red_light with the presence of either pedestrians or red lights in the dashcam, resulting in what is known as a reasoning shortcut. (Right) Consequently, the learned concepts can lead to incorrect (and potentially catastrophic) predictions when the background knowledge changes. For example, $K_2$ in Task $2$ introduces an exception in the presence of an emergency situation, i.e., “if there is an emergency and no pedestrians are on the street, the car may proceed”. Outline The remainder of the article is organized as follows. Section 2 introduces the preliminaries necessary for understanding the theoretical material. Section 3 presents the problem of reasoning shortcuts at a high level, providing examples and discussing their causes and impacts. Section 4 delves into the details by introducing the mathematical tools required to analyze reasoning shortcuts, exploring the issue from two perspectives: identification and statistical learning. Section 5 adopts a more applied perspective by examining the root causes of RSs, together with practical strategies for diagnosing their presence and for mitigating or addressing their effects, and discusses the benefits of each strategy. Section 6 explores various extensions of the reasoning shortcut problem across different domains and discusses open problems and future directions. Finally, Section 7 reviews relevant related research, and Section 8 provides concluding remarks. ## 2 Preliminaries Table 1: Glossary of used symbols. | $x$ , $y$ , $z$ , $X$ , $Y$ , $Z$ ${\bm{x}}$ , ${\bm{X}}$ | Scalar constants Scalar variables Vectors | | --- | --- | | $X,Y,C$ | Sets | | $f({\bm{x}})$ , $p({\bm{c}}\mid{\bm{x}})$ | Concept extractor | | $β({\bm{c}})$ , $p({\bm{y}}\mid{\bm{c}};K)$ | Inference layer | | $\models$ | Logical entailment | | $K$ | Prior knowledge | | $θ$ | Network parameters | | $D$ | Dataset | Notation. Throughout, we denote scalar constants by lowercase letters $x$ , scalar (random) variables by uppercase letters $X$ , vectors of constants ${\bm{x}}$ and (random) variables ${\bm{X}}$ in bold typeface, and sets (e.g., $X$ ) by calligraphic letters. If $φ$ is a logical formula over variables ${\bm{X}}$ , we say that the constants ${\bm{x}}$ entail the formula $φ$ ( ${\bm{x}}\modelsφ$ ) if and only if replacing the variables ${\bm{X}}$ in the formula $φ$ with ${\bm{x}}$ makes the formula true. In this case, we say that ${\bm{x}}$ satisfies or is consistent with the formula; otherwise, it violates or is inconsistent with the formula. See Table 1 for a glossary. ### 2.1 From Neuro-Symbolic AI to Neuro-Symbolic Predictors Neuro-symbolic models aim to solve the long-standing problem of integrating learning and reasoning. Over time, many radically different NeSy architectures have emerged (Garcez et al., 2022; De Raedt et al., 2021; Feldstein et al., 2024) that differ not only in what kind of logic they implement reasoning with (e.g., propositional (Hoernle et al., 2022; Ahmed et al., 2022; Buffelli and Tsamoura, 2023) vs. first-order (Lippi and Frasconi, 2009; Diligenti et al., 2012; Manhaeve et al., 2018) and classical (Zhou, 2019) vs. fuzzy (Donadello et al., 2017; van Krieken et al., 2020) vs. probabilistic (Manhaeve et al., 2018; Ahmed et al., 2022; Feldstein et al., 2023)), but also in how they interpret logic reasoning itself (e.g., model vs. proof-based inference (De Raedt et al., 2021)) and in how they implement the computations (e.g., fully neural (Rocktäschel and Riedel, 2016) vs. hybrid neural-symbolic (Manhaeve et al., 2018)). This variety is reflected by the diversity of tasks that NeSy architectures can tackle, which range from hierarchical classification (Giunchiglia and Lukasiewicz, 2020; Hoernle et al., 2022; Ahmed et al., 2022) and knowledge base completion (Rocktäschel and Riedel, 2016), to open-ended reasoning (Manhaeve et al., 2018; Badreddine et al., 2022) and learning knowledge graph embeddings (Maene and Tsamoura, 2025). Reasoning shortcuts have been studied mainly in the context of NeSy predictors, a class of NeSy architectures specialized for constrained prediction tasks (Giunchiglia et al., 2022; Dash et al., 2022; Marconato et al., 2023a), denoted here as NeSy tasks; a discussion of RS in other NeSy architectures is left to Section 6.1. A NeSy task requires predicting labels that are consistent with known constraints. Each such task is defined by four elements: a description of the input ${\bm{X}}$ with domain $X$ , a description of the $k$ concepts ${\bm{C}}=(C_1,…,C_k)$ with domain $C=C_1×…×C_k$ , a description of $m$ discrete labels ${\bm{Y}}=(Y_1,…,Y_m)$ with domain $Y=Y_1×…×Y_m$ , and prior knowledge $K$ . Here, $X$ is the input domain (e.g., a vector space), $C$ is a finite set of concept vectors ${\bm{c}}$ , and so are the label sets $Y_i$ . The input ${\bm{X}}$ is usually high-dimensional and sub-symbolic (e.g., an image), while the concepts ${\bm{C}}$ are high-level and possibly human readable properties of the input (e.g., the objects appearing in the image). The prior knowledge $K$ encodes the constraints that we wish the model to satisfy— e.g., safety constraints or relevant regulations—as a single propositional logical formula. Although some NeSy predictor architectures support first-order logical formulas, we restrict our description to propositional formulas, for ease of exposition. We postpone a discussion of how RSs transfer to the first-order case to Section 6.1. Both the concepts ${\bm{C}}$ and the labels ${\bm{Y}}$ appear as logic variables in $K$ . We ground these notions using a running example: **Example 2.1** *Consider the simplified BDD-OIA autonomous driving task (Xu et al., 2020). Here, the inputs ${\bm{x}}$ are dashcam images and the (binary) label $y$ indicates whether the vehicle is allowed to proceed. The task involves three binary concepts $C_\texttt{grn}$ , $C_\texttt{red}$ , $C_\texttt{ped}$ encoding the presence of green lights, red lights, and pedestrians in the input image, respectively, and the prior knowledge $K$ encodes the constraint that if either a pedestrian or a red light is visible in the image, the vehicle must stop: $K=(C_\texttt{ped}∨ C_\texttt{red}⇔\lnot Y)$ . In this scenario, a NeSy predictor has to predict vehicle actions that are both accurate and comply with traffic regulations.* Given a (noiseless) training set $D=\{({\bm{x}},{\bm{y}})\}$ , a NeSy predictor is tasked with learning a mapping from inputs ${\bm{x}}$ to labels ${\bm{y}}$ that comply with the knowledge, that is, $({\bm{c}},{\bm{y}})\modelsK$ . Here, $({\bm{c}},{\bm{y}})$ is to be read as the concatenation of ${\bm{c}}$ and ${\bm{y}}$ . The key challenge is that, just like in Example 2.1, the prior knowledge $K$ is not specified at the input level, but rather over the concepts. These, in turn, are complex functions of the input that, in practice, cannot be manually specified. For this reason, NeSy predictors adopt modular architectures that comprise a learnable concept extractor responsible for predicting the concepts ${\bm{C}}$ from the inputs ${\bm{X}}$ , and an inference layer responsible for inferring the labels ${\bm{Y}}$ from ${\bm{C}}$ compatibly with the prior knowledge $K$ . The former is typically implemented as a feed-forward neural network, Although other architectures can be used (Diligenti et al., 2012). and the latter using some form of (differentiable) symbolic reasoning. Given an input ${\bm{x}}$ , a NeSy predictor first applies the concept extractor to obtain a distribution $p({\bm{C}}\mid{\bm{x}})$ over concepts, and then applies the inference layer to obtain a distribution $p({\bm{Y}}\mid{\bm{C}};K)$ over outputs ${\bm{y}}$ , given the concepts. Overall, the predictor defines a predictive distribution $p({\bm{Y}}\mid{\bm{X}};K)$ (see Section 2.2 for details). **Example 2.2** *Consider the BDD-OIA (Xu et al., 2020) dataset illustrated in Example 2.1. Here, the input ${\bm{x}}∈X$ is a dashcam image. The concept extractor is a neural network that takes ${\bm{x}}$ and predicts the probabilities of three binary concepts $(C_\texttt{grn},C_\texttt{red},C_\texttt{ped})$ . For example, given an image ${\bm{x}}$ , the extractor may output $p(C_\texttt{red}=1\mid{\bm{x}})=0.9, p(C_\texttt{ped}=1\mid{\bm{x}})=0.2, p(C_\texttt{grn}=1\mid{\bm{x}})=0.1$ . These concept predictions are then passed to an inference layer, which combines them with prior knowledge $K=(C_\texttt{ped}∨ C_\texttt{red})⇔\texttt{stop}$ to obtain the final decision $y$ . Depending on the chosen implementation, this reasoning step may enforce the rule strictly, approximate it through fuzzy logic, or learn it implicitly during training. In this example, since the concept extractor assigns high probability to $C_\texttt{red}=1$ , the system will likely infer $Y=\texttt{stop}$ , indicating that the vehicle should stop.* In most cases, NeSy predictors can be, and in fact are, trained using label supervision only. This is not surprising if we consider that per-concept annotations can be expensive to collect in practice. However – and this is critical for our discussion of RSs – this also means that the concepts themselves are often not supervised, and that therefore they act as latent variables. ### 2.2 The Variety of NeSy Predictor Architectures Different NeSy predictor architectures differ in how they implement and use the concept extractor and the reasoning layer. Next, we introduce the four architectures that have been most extensively studied in the RS literature, namely probabilistic NeSy predictors, the Semantic Loss, Logic Tensor Networks, and Abductive Learning. These are illustrated in Fig. 2. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Four Neural-Symbolic Integration Frameworks ### Overview The image displays a comparative diagram of four distinct neural-symbolic integration frameworks, presented as four vertical panels separated by dotted lines. Each panel illustrates a different architectural approach for combining a neural network (NN) with a symbolic knowledge component (K). The diagram uses a consistent visual language: dark blue rounded rectangles for the Neural Network (NN), colored rounded rectangles for the Knowledge component (K), and arrows to denote data flow. Red lightning bolt symbols appear in three of the four panels, likely indicating a loss or error signal. ### Components/Axes The diagram is structured into four labeled panels: * **(a) PNSP** (leftmost panel) * **(b) SL** (second from left) * **(c) ABL** (third from left) * **(d) LTN** (rightmost panel) **Common Components per Panel:** * **NN**: A dark blue block representing a Neural Network. * **K**: A colored block representing a Knowledge component. The color varies by panel. * **x**: Input data (at the bottom of each flow). * **c**: An intermediate representation or concept (between NN and K). * **y**: Output or prediction (at the top of each flow). * **Red Lightning Bolt**: A symbol appearing near an output (`y`), suggesting a loss calculation or error signal. * **Dotted Line with Label**: Connects the `K` block to a text label describing the underlying logic paradigm. ### Detailed Analysis **Panel (a) PNSP (Probabilistic Neural-Symbolic Programming):** * **Flow:** `x` → `NN` → `c` → `K` → `y`. * **Knowledge Component (K):** Colored **red**. * **Logic Label:** "Probabilistic Logic" (in red text), connected to `K` via a red dotted line. * **Loss Symbol:** A red lightning bolt is positioned to the upper right of the final output `y`. **Panel (b) SL (Supervised Learning with Logic):** * **Flow:** `x` → `NN`. The NN has two outputs: one arrow goes to `c` and then to `K`, and a second arrow goes directly to an output `y`. The `K` block also outputs to the same `y`. * **Knowledge Component (K):** Colored **red**. * **Logic Label:** "Probabilistic Logic" (in red text), connected to `K` via a red dotted line. * **Loss Symbol:** A red lightning bolt is positioned to the right of the direct `y` output from the NN. **Panel (c) ABL (Abductive Learning):** * **Flow:** `x` → `NN` → `c` → `K`. The `K` block also receives an input `y` from above. * **Knowledge Component (K):** Colored **orange**. * **Logic Label:** "Abduction" (in orange text), connected to `K` via an orange dotted line. * **Loss Symbol:** A red lightning bolt is positioned to the upper right of the input `y`. **Panel (d) LTN (Logic Tensor Networks):** * **Flow:** `x` → `NN` → `c` → `K` → `y`. * **Knowledge Component (K):** Colored **teal/dark cyan**. * **Logic Label:** "Fuzzy Logic" (in teal text), connected to `K` via a teal dotted line. * **Loss Symbol:** A red lightning bolt is positioned to the upper right of the final output `y`. ### Key Observations 1. **Architectural Variation:** The core difference lies in the connection topology between the NN, the symbolic K component, and the output `y`. PNSP and LTN show a strict sequential flow (NN→K), while SL has a parallel structure where the NN contributes directly to the output alongside K. ABL is unique in having `y` as an *input* to the K component. 2. **Logic Paradigm Mapping:** Each framework is explicitly associated with a different logic-based reasoning method: Probabilistic Logic (PNSP, SL), Abduction (ABL), and Fuzzy Logic (LTN). 3. **Loss Signal Placement:** The red lightning bolt (loss) is consistently placed near the `y` variable, but its context changes: it's on the final output in PNSP/LTN, on a direct NN output in SL, and on an input to the symbolic reasoner in ABL. 4. **Color Coding:** The color of the `K` block and its associated logic label text/dotted line is consistent within each panel (red for Probabilistic, orange for Abduction, teal for Fuzzy). ### Interpretation This diagram serves as a high-level taxonomy for integrating deep learning (NN) with symbolic AI (K). It visually argues that the primary design choice in such systems is **how the neural and symbolic components interact** and **what form of logical reasoning** bridges them. * **PNSP & LTN** represent a **"neural-first, symbolic-refinement"** pattern. The neural network extracts concepts (`c`) from raw data (`x`), which are then processed by a symbolic reasoner (`K`) using probabilistic or fuzzy logic to produce a final, logically constrained output (`y`). The loss is applied to this final output. * **SL** represents a **"hybrid supervision"** pattern. The neural network's output is regularized or guided by a symbolic knowledge base (`K`), but it also contributes directly to the prediction. The loss appears to be applied to the NN's direct output, suggesting the symbolic component acts as a regularizer during training. * **ABL** represents an **"abductive reasoning"** pattern. This is distinct: the system starts with an observation or hypothesis (`y`) and uses the symbolic knowledge (`K`) to find the best explanation (`c`), which is then used to train or refine the neural network (`x`→`NN`). The loss is associated with the quality of the hypothesis (`y`). The lightning bolt symbols are critical. Their placement suggests where the primary training signal (loss) is applied in each framework, which fundamentally changes what is being optimized: the final output (PNSP, LTN), an intermediate neural representation (SL), or the plausibility of an input hypothesis (ABL). The diagram efficiently communicates that while all four methods combine neural and symbolic elements, their operational logic and training objectives are fundamentally different. </details> Figure 2: Schematic illustration of NeSy predictors. All of them employ a neural network for extracting concepts and prior knowledge $K$ . PNSPs (a) map inputs ${\bm{x}}$ to (a distribution over) concepts ${\bm{c}}$ and then uses probabilistic logic to infer labels ${\bm{y}}$ consistent with $K$ . The SL (b) maps inputs to both concepts ${\bm{c}}$ and labels ${\bm{y}}$ and uses $K$ to define a loss term penalizing inconsistent predictions. During training, ABL (c) abduces concepts from the ground-truth label ${\bm{y}}$ using $K$ and uses them as pseudo-labels to train the concept extractor. LTN (d) works similarly to PNSPs but uses fuzzy logic to make $K$ differentiable. Lightning strikes indicate supervision. Probabilistic NeSy Predictors A common method for designing NeSy architectures is the family of Probabilistic NeSy Predictors (PNSPs) (Manhaeve et al., 2018, 2021a; Yang et al., 2020; Huang et al., 2021a). PNSPs can be viewed as combining a neural concept extractor with a probabilistic-logic reasoning layer (De Raedt and Kimmig, 2015), as shown in Fig. 2 (a). Perhaps the best-known member of this family is DeepProbLog (DPL) (Manhaeve et al., 2018, 2021a), a fully-fledged neuro-symbolic programming language based on Prolog and its probabilistic extension, ProbLog (De Raedt et al., 2007; Kimmig et al., 2011). From a simplified point of view, PNSPs define the following predictive distribution for any ${\bm{x}}$ and ${\bm{y}}$ : $$ p({\bm{y}}\mid{\bm{x}};K)=∑_{\bm{c}}p({\bm{y}},{\bm{c}}\mid{\bm{x}};K)=∑_{\bm{c}}p({\bm{y}}\mid{\bm{c}};K) p({\bm{c}}\mid{\bm{x}})=\frac{1}{Z_{\bm{x}}}∑_{\bm{c}}1≤ft\{({\bm{c}},{\bm{y}})\modelsK\right\} p({\bm{c}}\mid{\bm{x}}) \tag{1} $$ Here, $p({\bm{C}}\mid{\bm{x}})$ is the predictive distribution over concepts computed by the concept extractor, $1≤ft\{·\right\}$ is the indicator function, and $Z_{\bm{x}}$ is a normalization constant ensuring that the result is indeed a conditional distribution. In words, the probability of a label ${\bm{y}}$ is the sum of the probabilities of all concept vectors ${\bm{c}}$ that are consistent with that label ${\bm{y}}$ according to the knowledge $K$ . Inference in PNSPs amounts to computing a most likely label, which usually requires solving the MAP problem $\operatorname*{argmax}_{\bm{y}} p({\bm{y}}\mid{\bm{x}};K)$ (Koller and Friedman, 2009). Importantly, if a label ${\bm{y}}$ , together with the predicted concepts, would violate the knowledge $K$ , then its probability is exactly zero and it will not be predicted. Learning in PNSPs is implemented via maximum likelihood estimation. That is, given a training set $D=\{({\bm{x}},{\bm{y}})\}$ , learning amounts to maximizing the log-likelihood of the data, namely $$ L(p,D,K)=\frac{1}{|D|}∑_({\bm{x},{\bm{y}})∈D} \log{p({\bm{y}}\mid{\bm{x}};K)}. \tag{2} $$ by tuning the parameters of the concept extractor. In practice, this is done via gradient descent, as the predictive distribution $p({\bm{y}}\mid{\bm{x}};K)$ is differentiable (Manhaeve et al., 2018). PNSPs require evaluating the predictive probability $p({\bm{y}}\mid{\bm{x}};K)$ , which involves summing over a potentially exponential number (in the cardinality of the concept vector $k$ ) of vectors ${\bm{c}}$ , as per Eq. 1. Hence, both inference and learning are worst-case intractable. DPL works around this issue by exploiting knowledge compilation techniques (Darwiche and Marquis, 2002) that leverage symmetries in the summation to rewrite it into a data structure – a probabilistic circuit (Choi et al., 2020; Vergari et al., 2021; Maene et al., 2025; Derkinderen et al., 2025a) – that is potentially much more compact and supports efficient evaluation. This allows DPL to scale to practical NeSy tasks. Other PNSPs refine and approximate these ideas (specifically, Eq. 1) to further improve scalability (Manhaeve et al., 2021b; Huang et al., 2021a; Winters et al., 2022; De Smet et al., 2023a; van Krieken et al., 2023; Choi et al., 2025; Chen et al., 2025). Semantic Loss The Semantic Loss (SL) (Xu et al., 2018) also relies on probabilistic logic and knowledge compilation, but rather than defining an inference layer proper, it uses them to convert the prior knowledge $K$ into a differentiable penalty term. This can be used to steer any neural network classifier toward allocating low or even zero probability to inconsistent outputs. While the SL is a general purpose strategy, here we discuss how it can be exploited for defining a NeSy predictor (Marconato et al., 2023a). The idea is to apply the SL to a neural network that acts both as concept extractor and as classifier, i.e., that outputs both $p({\bm{C}}\mid{\bm{X}})$ and $p({\bm{Y}}\mid{\bm{X}})$ . This encourages the concepts and labels predicted by the network to be logically consistent with each other according to $K$ . Whether this is a single neural network or two specialized networks makes no difference from a modeling perspective. The SL grows proportionally to how much probability mass the concept extractor associates to invalid configurations, or more precisely: $$ \texttt{SL}(p_θ,({\bm{x}},{\bm{y}}),K)=-\log∑_\bm{c}1≤ft\{({\bm{c}},{\bm{y}})\modelsK\right\}p_θ({\bm{c}}\mid{\bm{x}}) \tag{3} $$ As in DPL, efficient computation of this sum relies on knowledge compilation. During training, the SL is paired with a standard supervised loss $\ell$ (e.g., the cross-entropy loss), resulting in a joint objective of the form: $$ L(p_θ,D,K)=\frac{1}{|D|}∑_({\bm{x},{\bm{y}})∈D}\ell(p_θ,({\bm{x}},{\bm{y}}))+μ \texttt{SL}(p_θ,({\bm{x}},{\bm{y}}),K) \tag{4} $$ with $μ>0$ a hyperparameter. Learning amounts to minimizing this joint objective on training data. At test time, predictions are computed directly by the neural network classifier $p_θ({\bm{y}}\mid{\bm{x}})$ rather than by a symbolic layer as in PNSPs. Logic Tensor Networks Another well-known approach is Logic Tensor Networks (LTN s) (Donadello et al., 2017; Badreddine et al., 2022), which share elements with both PNSPs and SL. As with the SL, they transform the symbolic knowledge $K$ into a differentiable real-valued function $T_K$ that evaluates how well predictions conform to the logical constraints using fuzzy logic (van Krieken et al., 2020). This transformation is a proper relaxation. Specifically, it converts all the original Boolean variables in $K$ (namely, the concepts and the labels) into real-valued variables in the range $[0,1]$ encoding degrees of truth. At the same time, it converts all logic connectives (conjunctions, disjunctions, and negations) into real-valued operators capable of handling soft degrees of truth. The translation yields a function $T_K$ that takes as input a distribution over the concepts ${\bm{C}}$ and a distribution over the labels ${\bm{Y}}$ , and outputs the degree (in $[0,1]$ ) to which these satisfy the knowledge $K$ , the higher the better. By default, LTN s employ a transformation based on the product real logic (Donadello et al., 2017) T-norm to ensure that $T_K$ is fully differentiable, although other options are available (van Krieken et al., 2020). During training, LTN s penalize the concept extractor $p({\bm{C}}\mid{\bm{X}})$ for violating the prior knowledge by minimizing the following: $$ L(p,D,K)=1-\frac{1}{|D|}∑_({\bm{x},{\bm{y}})∈D}T_K(p({\bm{C}}\mid{\bm{x}}),1≤ft\{{\bm{Y}}={\bm{y}}\right\}) \tag{5} $$ Here, $1≤ft\{{\bm{Y}}={\bm{y}}\right\}$ is a (deterministic) distribution that assigns all probability mass to the ground-truth label ${\bm{y}}$ . The purpose of this objective is to encourage $p({\bm{C}}\mid{\bm{X}})$ to concentrate its mass on concepts that satisfy $K$ . Similarly to PNSPs, LTN s employ the prior knowledge at inference time. They proceed in two steps: they first compute the most probable concept vector $\hat{\bm{c}}=\operatorname*{argmax}_{\bm{c}}p_θ({\bm{c}}\mid{\bm{x}})$ in a forward pass, then select the label $\hat{\bm{y}}$ that maximizes the satisfaction of the knowledge $T_K(\hat{\bm{c}},1≤ft\{{\bm{Y}}=\hat{\bm{y}}\right\})$ . Abductive Learning Another well-studied approach is Abductive Learning (ABL) (Zhou, 2019). Compared to the above NeSy predictors, it works backwards: rather than inferring a prediction from the concepts, it constrains the predicted concepts using the ground-truth label. To this end, during training, ABL finds the concepts vectors $\hat{\bm{c}}$ that are both close to those predicted by the concept extractor (according to some pre-defined distance metric) and that according to $K$ entail the ground-truth label ${\bm{y}}$ . It then uses them as pseudo-labels to supervise the concept extractor. The pseudo-labels are obtained by solving the following optimization problem: $$ \hat{\bm{c}}=\operatorname*{argmin}_{\bm{c}^\prime∈C}d(\bar{\bm{c}},{\bm{c}}^\prime) s.t. ({\bm{c}}^\prime,{\bm{y}})\modelsK \tag{6} $$ where $\bar{\bm{c}}=\operatorname*{argmax}_{\bm{c}∈C}p({\bm{c}}\mid{\bm{x}})$ , and $d$ is a suitable distance metric. The constraint ensures that $\hat{\bm{c}}$ entail the ground-truth label ${\bm{y}}$ . The choice of distance metric influences the type of weak supervision obtained, thereby intuitively biasing learning toward certain solutions. The training objective of the concept extractor is to maximize the (log)-likelihood of the pseudo-labels using $\hat{\bm{c}}$ from Eq. 6: $$ L(p,D,K)=\cfrac{1}{|D|}∑_({\bm{x},\hat{\bm{c}})}\log p(\hat{\bm{c}}\mid{\bm{x}}) \tag{7} $$ At inference time, ABL first predicts the concepts and then uses the symbolic knowledge to obtain the final prediction. ### 2.3 NeSy Predictor Architectures: Differences and Similarities These different NeSy predictor architectures all share a key property: if the concept extractor allocates all probability mass to a single concept vector, then all architectures will output a label compatible with the rules of propositional logic. Consider MNIST-Add: if the concepts $C_1=2$ and $C_2=3$ are predicted with certainty, then all architectures will predict the label $Y=5$ with certainty—thus matching what any any logical encoding of the arithmetic sum would do. However, for concept predictions that are not certain, different architectures can output different label distributions. This fact will become relevant in Section 4. At the same time, these four NeSy predictor architectures differ in several subtle but significant ways. The first one concerns efficiency. LTN and ABL can have an advantage over probabilistic logic approaches (like PNSPs and the SL) in that inference—and therefore training–does not require to sum over all possible concept configurations, making it potentially more efficient, although knowledge compilation and approximation strategies help bridge the gap (Huang et al., 2021a). This, however, often comes at a cost, in that LTN and ABL tend to be more susceptible to local minima (Badreddine et al., 2022). On the other hand, fuzzy relaxations (as used by LTN) may not be entirely accurate to the original prior knowledge, in the sense that, for certain problems, the optima of the satisfaction function $T_K$ may not correspond to models (that is, $0$ – $1$ solutions) of $K$ (Giannini et al., 2018; van Krieken et al., 2022). Moreover, the fuzzy transformation may lead to mathematically different relaxations for logically equivalent constraints (Di Liello et al., 2020; van Krieken et al., 2022). Probabilistic logic is not affected by these issues (Xu et al., 2018). ABL is special in that it does not require the reasoning step to be differentiable, and as such it does not need to relax or extend the semantics of the prior knowledge at all. A second difference concerns validity guarantees, which set apart layer-based approaches from penalty-based ones. The SL “bakes” the prior knowledge directly into the neural network, meaning that finding a high-quality output amounts to a simple forward pass over the network: $K$ plays no role during inference. This also results in a lower memory footprint compared to PNSPs, as the probabilistic circuit can be dropped after training (Di Liello et al., 2020). The downside is that, while PNSPs and ABL ensure invalid outputs will not be predicted, this is not the case for the SL, unless the neural network attains exactly zero Semantic Loss during training, and even so this property might not carry over to test and out-of-distribution samples. Regardless, it has been shown that PNSPs and the SL have the same effect on the underlying neural network, that is, when learned to optimality both models yield concept extractors that comply with the prior knowledge $K$ (Marconato et al., 2023a). LTN sits in-between these alternatives, as it uses the prior knowledge at inference time, but may output inconsistent predictions if the most likely inputs violate the knowledge. To resolve this issue, fuzzy logic layers such as CCN+ (Giunchiglia et al., 2024) ensure all outputs comply with the knowledge. ### 2.4 NeSy Predictors: Benefits Despite these differences, all NeSy predictors offer a number of key benefits: - Performance: Just like regular neural networks, they are fully-fledged deep learning architectures that can handle complex low-level inputs via end-to-end training and latent representations. - Validity: Unlike regular neural networks, their predictions comply with the prior knowledge $K$ , possibly also with guarantees and out-of-distribution, cf. Section 2.3. This is essential in high-stakes applications where predictions have to comply with safety or structural constraints. - Reusability: It is straightforward to reuse the learned concept extractors in downstream tasks, e.g., out-of-distribution tasks (Marconato et al., 2023b), continual learning scenarios (Marconato et al., 2023a), model verification (Xie et al., 2022; Zaid et al., 2023; Morettin et al., 2024), and shielding (Yang et al., 2023a). - Interpretability: Users can not only inspect the concept-level predictions and the symbolic inference steps to make sense of the model’s predictions, they can also trace back the model’s prediction to the underlying concepts using gradient-based (Sundararajan et al., 2017) or formal (Huang et al., 2021b) explainability techniques. This supports stakeholders in assessing the reliability of the predictor and potentially enables them to supply corrective feedback (Teso et al., 2023). Compared to regular neural networks, which primarily target B1, benefits B2 – B4 make NeSy predictors an ideal choice for high-stakes applications (Di Liello et al., 2020; Hoernle et al., 2022; Marconato et al., 2023b; Yang et al., 2023a). However, reusability (B3) and interpretability (B4) hinge on the concepts being grounded appropriately. In fact, unless the learned concepts possess reasonable semantics, their meaning might be opaque to stakeholders, making it difficult to properly explain the model’s predictions with them. Furthermore, as exemplified by Fig. 1, reusing poorly grounded concepts in downstream applications may lead to unintended consequences. This is precisely where reasoning shortcuts enter the picture, as discussed next. ## 3 A Gentle Introduction to Reasoning Shortcuts When does a NeSy predictor ground the concepts incorrectly? It may be tempting to assume that – as long as the training data $D=\{({\bm{x}},{\bm{y}})\}$ is abundant and noiseless, and the prior knowledge is complete and correct – NeSy predictors that minimize the training loss will predict the concepts correctly, just like in standard supervised learning. However, this is not the case. This is because of reasoning shortcuts (RSs), which we informally define as follows: Definition 3.1 (Reasoning Shortcut, Informal). A reasoning shortcut is a situation in which a NeSy predictor attains accurate label predictions that comply with the prior knowledge by grounding concepts incorrectly. We illustrate two prototypical RSs in the following examples. **Example 3.2 (RSs inMNIST-Add(Manhaeve et al.,2018))** *In MNIST-Add, the task is to predict the sum (e.g., $y=9$ ) of the digits appearing in two MNIST (LeCun, 1998) images (e.g., ${\bm{x}}=\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}$ ). The concepts ${\bm{C}}=(C_1,C_2)$ encode the individual digits (e.g., ${\bm{c}}=(4,5)$ ). Here and elsewhere, we simplify the presentation by using numerical variables. It is always possible to encode these and the corresponding constraints into propositional logic. The prior knowledge enforces the prediction to be their arithmetic sum: $$ K=(Y=C_1+C_2), \tag{8} $$ Consider a toy scenario where the training set consists of just two examples: $$ (\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}})↦ 9 and (\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}})↦ 5. \tag{9} $$ Assume that the concept extractor processes the MNIST images separately. In this case, it may learn the intended image-concept mapping – that is, $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 2,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 3,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 4,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 5\}$ – as it adheres to the constraints and attains high label accuracy. But it can alternatively learn a different input-concept mapping $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 4,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 1,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 3,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 6\}$ , which also complies with the constraints and achieves high label accuracy, but it does so by grounding the concepts incorrectly.* While this is not a realistic application, the same issue affects also high-stakes tasks: **Example 3.3 (RSs inBDD-OIA)** *Consider Example 2.1. A NeSy predictor that grounds the concepts of green light, red light, and pedestrian correctly achieves high accuracy, as it will also correctly infer that it has to stop whenever a pedestrian or a red light appear in the input image. However, a different NeSy predictor that systematically confuses pedestrians with red lights achieves the same accuracy, since, according to $K$ , both lead to the correct ${\tt stop}$ action (Marconato et al., 2023b), as shown in Fig. 3.* | 0 | 0 | 1 (go) | | --- | --- | --- | | 0 | 1 | 0 (stop) | | 1 | 0 | 0 (stop) | | 1 | 1 | 0 (stop) | <details> <summary>x2.png Details</summary>  ### Visual Description ## Comparative Diagram: Traffic Light Classification Scenarios ### Overview The image displays two side-by-side schematic diagrams illustrating traffic light and pedestrian scenarios, likely for evaluating a computer vision or machine learning model's classification performance. Each panel depicts a simplified intersection with a traffic light, a pedestrian crossing, and mathematical annotations indicating ground truth (`y`), model prediction (`ŷ`), and confidence or indicator variables (`c_red`, `c_ped`). ### Components/Axes The image is divided into two distinct panels (left and right), each containing: 1. **Visual Scene:** A simplified top-down view of a road intersection with green grass areas, gray road surfaces, and a white pedestrian crossing line. 2. **Traffic Light:** A black pole-mounted traffic light. 3. **Pedestrian:** A black stick-figure icon (present only in the right panel). 4. **Highlight Box:** An orange rectangular bounding box around the primary object of interest (traffic light in left panel, pedestrian in right panel). 5. **Text Label:** The word "red" in orange text, positioned near the highlighted object. 6. **Mathematical Annotations:** * **Top of each panel:** `y = 0, ŷ = 0` followed by a green checkmark (✓). * **Bottom of each panel:** `c_red = 1, c_ped = 0` followed by a symbol (✓ or ✗). ### Detailed Analysis **Left Panel (Correct Classification):** * **Scene:** Traffic light shows a **red** illuminated circle. No pedestrian is present on the crossing. * **Highlight & Label:** The orange box surrounds the traffic light. The orange text "red" is placed to its left. * **Annotations:** * Top: `y = 0, ŷ = 0` ✓. This indicates the ground truth label (`y`) and the model's prediction (`ŷ`) are both 0, which is correct (✓). * Bottom: `c_red = 1, c_ped = 0` ✓. This suggests a confidence or indicator variable for "red light" (`c_red`) is active (1), and for "pedestrian" (`c_ped`) is inactive (0). This combination is marked as correct (✓). **Right Panel (Incorrect Classification):** * **Scene:** Traffic light shows a **green** illuminated circle. A pedestrian icon is present on the crossing. * **Highlight & Label:** The orange box surrounds the **pedestrian**. The orange text "red" is placed to the left of the pedestrian, not the traffic light. * **Annotations:** * Top: `y = 0, ŷ = 0` ✓. The ground truth and prediction are again both 0, marked as correct. * Bottom: `c_red = 1, c_ped = 0` ✗. The same variable state as the left panel (`c_red=1`, `c_ped=0`) is now marked as **incorrect** (✗). ### Key Observations 1. **Spatial Grounding Discrepancy:** In the right panel, the label "red" and the highlight box are associated with the pedestrian, not the traffic light which is actually green. This is a critical misalignment. 2. **Variable State Contradiction:** The variable state `c_red = 1, c_ped = 0` is consistent across both panels but is evaluated as correct (✓) in the left and incorrect (✗) in the right. This implies the *meaning* or *context* of these variables changes based on the visual scene, not just their values. 3. **Top-Level vs. Bottom-Level Evaluation:** The top annotation (`y=0, ŷ=0`) is marked correct in both cases, suggesting the model's final output (perhaps a binary "violation" or "stop" decision) is correct. The bottom annotation reveals an error in the model's *internal reasoning* or *intermediate feature detection* in the right panel. ### Interpretation This diagram appears to be a **failure case analysis** for an AI system, likely one designed for traffic scene understanding. It demonstrates a scenario where the model arrives at the correct final prediction (`ŷ=0`) but for the wrong reasons. * **Left Panel (Success):** The model correctly identifies the red light (`c_red=1`) and the absence of a pedestrian (`c_ped=0`), leading to a correct final output. * **Right Panel (Failure):** The model incorrectly associates the "red" attribute with the pedestrian instead of the traffic light. It still outputs `c_red=1` (perhaps triggered by the pedestrian's presence or clothing) and `c_ped=0` (failing to detect the pedestrian as a distinct entity). While the final prediction `ŷ=0` might still be correct by chance (e.g., if a green light with a pedestrian also means "do not proceed"), the internal representation is flawed. The red "X" (✗) highlights this reasoning error. The core message is that **accuracy in final output (`ŷ`) does not guarantee correctness in intermediate understanding (`c_red`, `c_ped`)**. The diagram emphasizes the importance of evaluating not just *what* a model decides, but *why* it decides it, using spatial grounding and attribute binding as key diagnostic tools. </details> Figure 3: Left: truth table of the simplified prior knowledge $K$ used in Example 2.1: the two concepts are interchangeable, in that as soon as one of them fires, the predictor infers the same (correct) action. Right: illustration of this RS: the model can confuse pedestrians and red lights with no drop in training loss. These examples show that correct and faulty NeSy predictors cannot be distinguished by label accuracy alone, and therefore there is no reason why, during training, NeSy methods should favor one concept mapping over the other. They also indicate that RSs can occur even when the prior knowledge is accurate and complete, and the training data is noiseless. Moreover, RSs can persist even if the training data is exhaustive (e.g., if the BDD-OIA training set encompasses examples containing all possible combinations of green lights, red lights and pedestrians), as the correct and faulty models yield the same predictions on all examples. ### 3.1 Causes, Frequency and Impact Why do RSs arise? Roughly speaking, RSs stem from two related issues: on the one hand, the prior knowledge $K$ may allow inferring the correct labels ${\bm{y}}$ from improperly grounded concept vectors ${\bm{c}}$ ; on the other, the concept extractor is expressive enough to learn the improperly grounded concepts. Our two examples satisfy both conditions. Their combination introduces ambiguity in the learning problem, meaning that NeSy predictors are free to learn incorrect concept mappings and still achieve high accuracy. Properly understanding the root causes of RSs, however, requires more technical background, which we provide in Section 4, hence we postpone a more comprehensive discussion to Section 5.1. Importantly, clarifying the causes of RSs allows us to discriminate between risky and safe NeSy tasks, and to design RS mitigation strategies. We explore these topics in Section 5. Do RSs occur in practice? The above discussion entails that RSs can occur whenever the prior knowledge is not “strict” enough, but it does not prove that they must occur. Arguably, NeSy predictors can learn the intended concepts even in this case. The question is then whether RSs occur in practice. While it is difficult to gauge their frequency in real-world applications, the literature abounds with situations in which NeSy predictors, trained in standard conditions, fall prey to RSs (Manhaeve et al., 2018; Wang et al., 2019; Chang et al., 2020; Manhaeve et al., 2021a; Marconato et al., 2023b, 2024; Bortolotti et al., 2024; Yang et al., 2024; DeLong et al., 2024), suggesting that RSs occur naturally in NeSy benchmarks. Do RSs make NeSy predictors unusable? The short answer is no. More precisely, in applications where the only goal is to obtain accurate (B1) and valid (B2) predictions, RSs are not an issue. In fact, Definition 3.1 makes it clear that RSs are a symbol grounding issue that leaves label accuracy unchanged. Hence, the label predictions of affected and unaffected NeSy can be just as accurate. We also remark that RSs have no impact on validity (B2). As mentioned in Section 2.3, predictors like PNSPs, ABL, and LTN ensure their output is always consistent with the prior knowledge by explicitly searching for a label ${\bm{y}}$ that, paired with the predicted concepts vectors $\hat{\bm{c}}$ (or their predictive distribution $p({\bm{C}}\mid{\bm{x}})$ ), satisfies the knowledge $K$ . Validity is built-in, i.e., it does not depend on the concepts predicted by the model. A tricky aspect of RSs is that we cannot detect them by monitoring label predictions alone. We will discuss more reliable diagnostic techniques in Section 5.2 and strategies for rendering predictors aware of their own RSs in Section 5.4. Furthermore, RS mitigation strategies can greatly reduce the presence of RSs or even entirely remove them to ensure NeSy predictors are also reusable (B3) and interpretable (B4). We overview these strategies in Section 5.3. What consequences do RSs have? RSs can seriously compromise reusability (B3) and interpretability (B4). Let us begin from reusability. RSs may exploit concept groundings that do not work in other application settings, meaning that affected concepts suffer from poor generalization beyond the specific data and prior knowledge used during training (Marconato et al., 2023b; Li et al., 2023; Bortolotti et al., 2024). To see this, consider the following example, taken from (Marconato et al., 2023b): **Example 3.4** *Consider the NeSy predictor confusing pedestrians with red lights in Example 3.3. Imagine pairing it with prior knowledge intended for an autonomous ambulance use-case, encoding that, in case of an emergency, it is allowed to cross red lights. When emergencies arise, the resulting NeSy predictor would decide to $y={\tt go}$ when there are pedestrians on the road.* RSs also affect concept reuse in downstream applications where the semantics of the concepts matters. For instance, in continual learning one is concerned with learning models that generalize across a sequence of learning tasks. NeSy predictors were shown to struggle with this precisely because of RSs (Marconato et al., 2023a): **Example 3.5** *Consider the shortcut predictor from Example 3.3. Imagine fine-tuning it on a new NeSy task that also includes the concept of “stop sign” and a new rule based on it, and specifically that whenever a stop sign is detected, the vehicle must stop. When fine-tuned on this new task, the concept extractor may mistakenly learn to equate stop signs and red lights, whose semantics is corrupted.* Another affected application is neuro-symbolic verification (Xie et al., 2022; Zaid et al., 2023; Morettin et al., 2024; Manginas et al., 2025). In model verification, the goal is to formally verify whether a neural network satisfies some property of interest, such as robustness to adversarial attacks or fairness. This involves writing a formal specification of the property, translating the model into a logical formula (or a similar representation), and then using tools from automated reasoning to check whether the model’s formula is compatible with the specification. NeSy verification is similar, except that the property of interest is specified in terms of high-level concepts. Following the neuro-symbolic setup, their definitions are provided by a trained concept extractor – provided by a third party – that also gets translated into a logical formula for the purpose of verification. Clearly, if the concepts are incorrect, this will affect the meaning of the property to be verified. Finally, interpretability (B4) also relies on the concepts being grounded appropriately (Marconato et al., 2023b; Koh et al., 2020; Poeta et al., 2023). Unless these possess human-aligned semantics, their meaning might be opaque to stakeholders, impairing understanding: **Example 3.6** *In Example 3.3, imagine the input scene portrays a pedestrian. An affected NeSy predictor would (correctly) predict stop, but its explanation would indicate that the decision depends on the presence of a red light. This explanation is misleading, and in fact it does not reflect the input at all.* ## 4 Theory of Reasoning Shortcuts In this section, we outline the two mainstream formalizations of RSs. Section 4.2 discusses RSs from an identifiability perspective, investigating under what conditions models achieving high likelihood can in fact learn correct concepts in the infinite sample case. Section 4.3 instead is concerned with guarantees – in terms of empirical risk minimization – for learning correct concepts from finite data. The so-far vague notion of “correct” concept will be clarified in the following sections. Additional notation Going forward, we will often treat distributions over a finite set of values as vectors in a simplex. This simplex has one dimension for each value in the set. We will write $Δ_C$ to indicate the simplex defined by the values contained in $C$ : $$ \textstyleΔ_C=\{{\bm{p}}∈[0,1]^|C|\mid∑_i=1^|C|p_i=1\} $$ Furthermore, we will write $Vert(Δ_C)=\{{\bm{p}}∈\{0,1\}^|C|\mid∑_i=1^|C|p_i=1\}$ to indicate the vertices of the simplex $Δ_C$ . This is simply a collection of one-hot vectors, one for each value in $C$ . ### 4.1 Setting and Assumptions In order to formally describe RSs, we have to clarify the link between the observed data, the concepts it implicitly encodes, and the concepts learned by the predictor. This link is provided by the data generating process, described next, which forms a solid basis for both theoretical approaches. $C_1$ $⋯$ $C_k$ ${\bm{X}}$ $G_k$ $⋯$ $G_1$ ${\bm{X}}$ ${\bm{Y}}$ ${\bm{Y}}$ $α_f$ $f$ $f^*$ $β$ $β^*$ Figure 4: Left. The data generation process: $f^*$ maps inputs ${\bm{X}}$ onto (distributions over) ground-truth concepts ${\bm{G}}$ , and $β^*$ maps these to (distributions over) labels ${\bm{Y}}$ . In a standard learning setting, only ${\bm{X}}$ and ${\bm{Y}}$ are observed (in light gray), whereas ${\bm{G}}$ are latent. Right. A NeSy predictor maps inputs ${\bm{X}}$ to concepts ${\bm{C}}$ via a learned $f$ and infers labels ${\bm{Y}}$ via an inference layer $β$ . By Eq. 20, doing so entails a map $α_f$ (in orange) from ground-truth concepts ${\bm{G}}$ to learned concepts ${\bm{C}}$ . The data generating process First we distinguish between ground-truth concepts ${\bm{g}}=(g_1,…,g_k)∈C$ underlying the data and learned concepts ${\bm{c}}=(c_1,…,c_k)∈C$ . For instance, in Example 3.3 there are two binary concepts encoding the presence of pedestrians and red traffic lights: $g_\tt ped$ and $g_\tt red$ are the true values, while $c_\tt ped$ and $c_\tt red$ are the predicted values. The training examples $({\bm{x}},{\bm{y}})$ are sampled from a ground-truth joint distribution $p^*({\bm{X}},{\bm{Y}})=p^*({\bm{Y}}\mid{\bm{X}})p^*({\bm{X}})$ , which is determined by the ground-truth concepts ${\bm{g}}∈C$ associated with the input ${\bm{x}}$ . Specifically, the values of the ground-truth concepts are sampled from a ground-truth conditional distribution $p^*({\bm{G}}\mid{\bm{X}})$ , while the ground-truth labels ${\bm{y}}$ are sampled from $p^*({\bm{Y}}\mid{\bm{G}};K)$ . It is assumed that this distribution implements the background knowledge $K$ , that is, it assigns zero or low probability to labels incompatible with the constraints. For technical reasons, it is assumed it implements the same reasoning process as the target NeSy predictor. It is useful to think of these two distributions as functions: the former as a function $f^*:X→Δ_C$ mapping each input to a distribution over ground-truth concepts, and the latter as a function $β^*:Δ_C→Δ_Y$ mapping each distribution over concepts to a distribution over labels. Elements $f({\bm{x}})∈Δ_C$ are probability distributions $p({\bm{C}}\mid{\bm{x}})$ over the possible concept vectors ${\bm{c}}∈C$ . Whenever $f({\bm{x}})$ allocates all probability mass to one specific ${\bm{c}}∈C$ , i.e., it is one-hot, it is a vertex of the simplex and as such it belongs to $Vert(Δ_C)⊂Δ_C$ . Overall, $p^*({\bm{Y}}\mid{\bm{X}},{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}K})$ results from composing $f^*$ and $β^*$ . A visualization of the data generation process is reported in Fig. 4 (Left). With this in mind, we present the first assumption capturing how data are generated, which will be used throughout the theoretical material. Assumption 4.1 (Completeness). Let $f^*:X→Δ_C$ be the ground-truth concept extractor, and $β^*:Δ_C→Δ_Y$ the inference function induced by the prior knowledge $K$ . Let $p^*({\bm{X}})$ be the marginal distribution over input variables. Then, we have: $p^*({\bm{X}},{\bm{G}},{\bm{Y}})=p^*({\bm{Y}}\mid{\bm{X}};K)p^*({\bm{G}}\mid{\bm{X}})p^*({\bm{X}})$ (10) where $p^*({\bm{G}}\mid{\bm{X}}):=f^*({\bm{X}})$ and $p^*({\bm{Y}}\mid{\bm{X}};K):=(β^*∘ f^*)({\bm{X}})$ . Under Assumption 4.1, the ground-truth concept extractor $f^*$ becomes a sufficient statistic of the input ${\bm{X}}$ for inferring the labels ${\bm{Y}}$ through its composition with $β^*$ . In other words, mapping inputs to the concept distribution with $f^*$ retains all information necessary to infer the labels. For concreteness, this assumption fails when the concept space is “too restrictive”. For example, in MNIST-Add with two digits, sums above $18$ cannot be generated. The above assumption excludes such cases from the training data. Simplifying assumptions on the data generation process While Assumption 4.1 underlies all theoretical works on RSs (Yang et al., 2024; Marconato et al., 2023b), we introduce two additional assumptions that make the theoretical analysis more manageable. The first assumption restricts the ground-truth concept extractor $f^*$ : It should assign all probability mass to a single vector of concepts ${\bm{g}}$ for each input ${\bm{x}}$ . Assumption 4.2 (Extrapolability). The ground-truth concept extractor is a function $f^*:X→Vert(Δ_C)$ , i.e., for all inputs ${\bm{x}}∈X$ , the ground-truth concept distribution $f^*({\bm{x}})$ is a one-hot (deterministic) distribution. Assumption 4.2 allows us to retrieve ground-truth concepts ${\bm{G}}$ from inputs ${\bm{X}}$ via $f^*$ . The second assumption ensures that, with the prior knowledge $K$ , each vector of concepts ${\bm{g}}$ is associated with a single label ${\bm{y}}$ . To formally introduce this assumption, we define, overloading notation, $$ β^*({\bm{c}}):=β^*(1≤ft\{{\bm{C}}={\bm{c}}\right\}), ∀{\bm{c}}∈C. \tag{11} $$ $β^*:C→Δ_Y$ is a map from the set of concepts to distribution over labels. It describes how deterministic distributions are mapped to distributions over labels by the knowledge $K$ . Assumption 4.3 (Determinism). The ground-truth inference layer defines a function $β^*:C→Vert(Δ_Y)$ that maps each concept vector ${\bm{c}}∈C$ to a one-hot (deterministic) distribution over labels $β^*({\bm{c}})$ . In words, Assumption 4.3 ensures that $β^*$ assigns a single label ${\bm{y}}$ to each concept vector ${\bm{c}}$ . Assumptions 4.2 and 4.3 often apply in experimental regimes. For MNIST-Add and BDD-OIA, the concepts in the input image can be unambiguously determined (Assumption 4.2 is satisfied), and all concept vectors ${\bm{c}}$ determine a single output label (Assumption 4.3 is satisfied), e.g., in MNIST-Add we use $Y=C_1+C_2$ . Together, Assumptions 4.2 and 4.3 ensure that each input ${\bm{x}}∈X$ is mapped to a single label ${\bm{y}}∈Y$ . The same holds for many common NeSy datasets (Bortolotti et al., 2024). Before proceeding, we have to define the support of the input and ground-truth concept distributions. We consider the marginal distributions $p^*({\bm{X}})$ and $p^*({\bm{G}}):=E_{\bm{x}∼ p^*({\bm{X}})}[f^*({\bm{x}})]$ . We indicate the support of input data with $supp({\bm{X}})⊆X$ , which contains the inputs with non-zero probability according to $p^*({\bm{X}})$ . Similarly, we denote the support of ground-truth concepts by $supp({\bm{G}})⊆C$ . ### 4.2 Perspective: Identification In this section, we ask whether a NeSy predictor trained in a standard supervised fashion is guaranteed to learn the ground-truth concept extractor. Specifically, we want to determine whether—depending on the choice of knowledge, training data, and architectural bias—the ground-truth concept extractor is the unique maximum of the likelihood of the training data. If so, then any NeSy predictor that attains maximum likelihood will necessarily learn the ground-truth concept extractor, and thus ground all concepts correctly. This question puts us firmly in the context of identifiability theory; for an expanded presentation, please see Section 7. Below we provide a revised perspective on the identifiability of RSs (Marconato et al., 2023a, b; Bortolotti et al., 2025) by clarifying the setting, presenting a first non-identifiability result from (Marconato et al., 2023a) in Theorem 4.4, and reviewing the characterization of RSs from (Marconato et al., 2023b; Bortolotti et al., 2025) in Theorems 4.10 and 4.12. All results assume the data follows the generating process described in Fig. 4 and that we have access to possibly infinite amounts of data. #### 4.2.1 Model class of NeSy predictors Like in the data generation process, NeSy predictors can be understood as a pair of functions: a concept extractor $f:X→Δ_C$ and an inference layer $β:Δ_C→Δ_Y$ . We indicate with $F$ the space of learnable concept extractors $f$ . Most theoretical accounts on RSs work in a non-parametric setting, whereby $F$ contains all possible functions from $X$ to $Δ_C$ . In words, they assume the neural network implementing the concept extractor can express any function $f:X→Δ_C$ (Marconato et al., 2023b; Yang et al., 2024; Bortolotti et al., 2025). We do the same, but we will relax this assumption when discussing mitigation and awareness strategies in Section 5.3 and Section 5.4. Furthermore, we also assume that, when provided with the prior knowledge $K$ , the inference layer $β$ of a NeSy predictor architecture behaves exactly like the ground-truth inference layer $β^*$ , i.e., $$ β({\bm{c}})=β^*({\bm{c}}), ∀{\bm{c}}∈C. \tag{12} $$ We denote with $B$ the space of inference layers that satisfy Eq. 12. The choice of the NeSy predictor architecture (e.g., PNSPs or LTN) determines how the inference layer $β$ behaves, as discussed in Section 2.2. A specific NeSy predictor thus amounts to a pair $(f,β)∈F×B$ , Unless mentioned otherwise, we present the results in the non-parametric setting where $F$ contains all possible concept extractors that map inputs to concept conditional probabilities (Bortolotti et al., 2025). ; its label distribution is given by: $$ p_f({\bm{Y}}\mid{\bm{X}};K):=(β∘ f)({\bm{X}}) , \tag{13} $$ For this class of models, we consider the maximum log-likelihood objective over infinite data: $$ {\arg\max}_f∈F E_({\bm{x},{\bm{y}})∼ p^*({\bm{X}},{\bm{Y}})}[\log p_f({\bm{y}}\mid{\bm{x}};K)] . \tag{14} $$ It turns out that this problem may not admit a unique solution, as explained next. #### 4.2.2 Non-identifiability of the concept extractor We start by presenting the necessary and sufficient conditions for a NeSy predictor to achieve maximum likelihood on data and then show that this can lead to non-identifiability. Before proceeding, for a probability measure $p^*({\bm{X}})$ and any two measurable functions $r({\bm{x}})$ and $s({\bm{x}})$ with respect to $p^*$ , we will use the shorthand $$ r({\bm{X}})=s({\bm{X}}) \tag{15} $$ to mean that $r$ takes values equal to $s$ for $p^*$ -almost every ${\bm{x}}∈supp({\bm{X}})$ . Theorem 4.4 (Revisited from (Marconato et al., 2023a)). Under Assumption 4.1, a NeSy predictor $(f,β)∈F×B$ attains maximum likelihood with respect to the distribution $p^*({\bm{X}},{\bm{Y}})$ if and only if $(β∘ f)({\bm{X}})=(β^*∘ f^*)({\bm{X}}) .$ (16) This immediately yields the following important corollary: when the same label can be predicted using different concept vectors, we cannot identify the concept extractor, i.e., $f^*∈F$ is not the only choice that leads to obtaining maximum likelihood for the ground-truth inference layer $β^*$ , but there are alternatives $f≠ f^*$ . Corollary 4.5 (Non-identifiability). Let $Δ_Y^*⊆Δ_Y$ be the subspace obtained from mapping inputs to labels, which is given by $Δ^*_Y:=(β^*∘ f^*)(supp({\bm{X}}))$ , and let $Δ^*_C⊆Δ_C$ be the preimage of $Δ^*_Y$ over the ground-truth inference layer $β^*$ , defined as $Δ^*_C:=\{{\bm{p}}∈Δ_C\mid β^*({\bm{p}})∈Δ_Y^*\}$ . Under Assumption 4.1, if $β^*$ is not injective from $Δ_C^*$ to $Δ_Y^*$ , then $(β^*∘ f)({\bm{X}})=(β^*∘ f^*)({\bm{X}})\mathchoice{\mathrel{\hbox to0.0pt{\kern 3.75pt\kern-5.30861pt$\displaystyle\not$\hss}{\implies}}}{\mathrel{\hbox to0.0pt{\kern 3.75pt\kern-5.30861pt$\textstyle\not$\hss}{\implies}}}{\mathrel{\hbox to0.0pt{\kern 2.625pt\kern-4.48917pt$\scriptstyle\not$\hss}{\implies}}}{\mathrel{\hbox to0.0pt{\kern 1.875pt\kern-3.9892pt$\scriptscriptstyle\not$\hss}{\implies}}}f({\bm{X}})=f^*({\bm{X}}) .$ (17) * Proof* By construction. If $β^*:Δ^*_C→Δ^*_Y$ is not injective, we can find ${\bm{p}},{\bm{p}}^\prime∈Δ_C^*$ such that ${\bm{p}}≠{\bm{p}}^\prime$ and $$ β^*({\bm{p}})=β^*({\bm{p}}^\prime) . \tag{18} $$ Hence, for all ${\bm{x}}∈supp({\bm{X}})$ such that $f^*({\bm{x}})={\bm{p}}$ we can construct another function $f∈F$ such that $f({\bm{x}})={\bm{p}}^\prime$ . By Eq. 18, the NeSy predictor $(f,β^*)$ will obtain maximum likelihood (Theorem 4.4), but $f({\bm{X}})≠ f^*({\bm{X}})$ . ∎ Corollary 4.5 formalizes the intuition in (Marconato et al., 2023a) that, when ground-truth inference layer $β^*$ maps multiple concept vectors to the same label, an optimal NeSy predictor can learn an RS. The important consequence is that the concept extractor $f$ attaining maximum likelihood is not unique, even when using the ground-truth inference layer $β^*$ and an infinite amount of data. In other words, we do not always correctly ground concepts. Motivated by these failures modes, we provide a formal definition of faulty NeSy predictors: This was the original definition of RSs in (Marconato et al., 2023b). Here, we change the original name to specialize it to NeSy predictors and distinguish it from the abstract notion of RSs, as explained in Definition 4.7. Definition 4.6 (Faulty NeSy predictor (Marconato et al., 2023b)). Consider a data generation process that satisfies Assumption 4.1 and let $(f^*,β^*)∈F×B$ be the ground-truth concept extractor and inference layer. We say that $(f,β)∈F×B$ is a faulty NeSy predictor if $(f,β)$ attains maximum likelihood on data (Theorem 4.4) and the disagreement set $D(f,f^*):=\{{\bm{x}}∈supp({\bm{X}}):f({\bm{x}})≠ f^*({\bm{x}})\}$ (19) has strictly positive $p^*$ -measure, i.e., $p^*(D(f,f^*))>0$ . From the definition, the set of faulty NeSy predictors both depends on the choice of admitted concept extractors $F$ and the possible inference layers $B$ . We will show in Section 5.3 how, by constraining the space $F$ through mitigation strategies, certain faulty NeSy predictors can vanish. From Section 2.2, each NeSy architecture specifies a particular $β∈B$ , which limits faulty NeSy predictors to concept extractors $f∈F$ for which $(f,β)$ is faulty. This also implies that some concept extractors $f≠ f^*$ may constitute a faulty NeSy predictor when paired with the inference layer of one model class, e.g., PNSPs, but not of another, e.g., LTN. While Corollary 4.5 establishes non-identifiability in general, it does not provide insight on what characterizes faulty NeSy predictors. Without this characterization, it is unclear when the ground-truth concept extractor can be identified and RSs consistently avoided. We next describe results that characterize such concept extractors $f∈F$ . #### 4.2.3 Concept remapping distributions. Following (Marconato et al., 2023b; Bortolotti et al., 2025), we shift the perspective from describing which concept extractors $f∈F$ are valid optima of the likelihood, to studying how ground-truth concept vectors can be remapped by NeSy models while retaining optimal likelihood. Formally, under Assumption 4.1, we let $p^*({\bm{X}}\mid{\bm{G}})$ be the posterior distribution induced by the ground-truth concept extractor $f^*$ . Then, to describe how concepts learned by a NeSy predictor $(f,β)∈F×B$ relate to ground-truth concepts, we define $$ \textstyleα_f({\bm{g}}):=E_{\bm{x}∼ p^*({\bm{X}}\mid{\bm{g}})}≤ft[f({\bm{x}})\right] , \tag{20} $$ the concept remapping distribution $α_f:C→Δ_C$ induced by $f$ (van Krieken et al., 2025a) . These distributions $α_f$ ’s describe how ground-truth concept vectors are mapped to learned concept vectors by $f$ . Hereafter, we use $A$ to denote the space of these $α_f$ ’s, which is a simplex (Bortolotti et al., 2025). Furthermore, we call $Vert(A)$ the set of (deterministic) concept remappings: for each ground-truth vector ${\bm{g}}∈C$ , the output of a concept remapping $α_f({\bm{g}})$ is one-hot. That is, for such (deterministic) concept remappings $α_f$ , there is a single concept vector ${\bm{c}}∈C$ with probability one: $$ \max_{\bm{c}∈C}α_f({\bm{g}})_\bm{c}=1 . \tag{21} $$ Any element $α_f∈A$ can be expressed using convex combinations of concept remappings, ${\bm{a}}∈Vert(A)$ , that is: $$ \textstyleα_f({\bm{g}})=∑_{\bm{a}∈Vert(A)}λ^f_\bm{a} {\bm{a}}({\bm{g}}), \tag{22} $$ where we have $λ^f_\bm{a}≥ 0$ for all ${\bm{a}}∈Vert(A)$ and $∑_{\bm{a}∈Vert(A)}λ^f_\bm{a}=1$ . Hence, a NeSy predictor can be seen both as a pair $(f,β)∈F×B$ and as a pair $(α_f,β)∈A×B$ . Notice that, because the learnable $α$ ’s depend on the space of learnable concept extractors $F$ , constraining $F$ also reduces the size of $A$ . This fact is also explicit in some mitigation strategies (see Section 5.3.8). Therefore, the intended $α_f^*$ implies that the learned concepts match the ground-truth concepts. #### 4.2.4 Reasoning shortcuts as unintended, optimal concept remapping distributions Based on this construction, we can give a formal definition of what a reasoning shortcut is and illustrate it with an example: Definition 4.7 (Reasoning Shortcut for $β$ (Formal)). Let $β∈B$ be the inference layer of a NeSy predictor (e.g., PNSPs or LTN) and $F$ the space of learnable concept extractors. We say that a concept remapping distribution $α∈A$ is a reasoning shortcut for the inference layer $β$ if there exists a concept extractor $f∈F$ , such that $(f,β)$ is a faulty NeSy predictor (Definition 4.6) and $E_{\bm{x}∼ p^*({\bm{X}}\mid{\bm{G}})}[f({\bm{x}})]=α({\bm{G}}).$ (23) <details> <summary>x3.png Details</summary>  ### Visual Description ## Diagram: Mathematical Mapping Between Sets ℱ and 𝒜 ### Overview The image displays a theoretical or mathematical diagram illustrating mappings between two abstract sets. The left set, labeled with a script "ℱ" (likely representing a function space or feature space), contains several points and an unlabeled region. The right set, labeled with a script "𝒜" (likely representing an action space, parameter space, or algebra), is depicted as a triangle. Directed arrows (some solid, some dashed) indicate mappings or functions from elements in ℱ to elements in 𝒜. A mathematical expression is positioned above the diagram. ### Components/Axes * **Left Set (ℱ):** An oval shape labeled with the script letter **ℱ** in the top-left corner. * **Right Set (𝒜):** A triangle shape labeled with the script letter **𝒜** at the top vertex. * **Mathematical Expression:** Centered above the diagram is the text: **E_{x∼p(x)}[·]**. This denotes an expectation over a variable `x` drawn from a distribution `p(x)`. * **Points in ℱ:** * **f\***: A point in the upper-left region of the oval. * **f₁**: A point in the middle-right region of the oval. * **f₂**: A point in the lower-left region of the oval. * **Unlabeled Pink Region:** An irregular, solid pink shape located near point `f₂` in the lower-left quadrant of the oval. It is not associated with a text label. * **Points in 𝒜:** * **id**: A point at the top vertex of the triangle. * **a₁**: A point at the bottom-left vertex of the triangle. * **a₂**: A point at the bottom-right vertex of the triangle. * **α**: A point inside the triangle, closer to the bottom edge and slightly right of center. * **Mappings (Arrows):** * A **dashed arrow** from **f\*** in ℱ to **id** in 𝒜. * A **solid arrow** from **f₁** in ℱ to **a₁** in 𝒜. * A **solid arrow** from **f₂** in ℱ to **a₂** in 𝒜. * A **solid arrow** from **α** in 𝒜 to **a₂** in 𝒜 (an internal mapping within 𝒜). * A **dashed arrow** from the **pink region** in ℱ to **a₁** in 𝒜. ### Detailed Analysis The diagram defines specific correspondences: 1. The element `f*` maps to `id` (likely the identity element). 2. The element `f₁` maps to `a₁`. 3. The element `f₂` maps to `a₂`. 4. The unlabeled pink region within ℱ maps to `a₁`. 5. There is an internal mapping within 𝒜 from `α` to `a₂`. The expression **E_{x∼p(x)}[·]** suggests the entire diagram or the mappings shown may be related to an expectation or average taken over a data distribution `p(x)`. ### Key Observations * **Dual Representation:** The diagram contrasts a potentially continuous or complex space (ℱ, depicted as an oval) with a structured, discrete, or geometric space (𝒜, depicted as a triangle). * **Mapping Types:** The use of both solid and dashed arrows may differentiate between types of mappings (e.g., deterministic vs. stochastic, primary vs. secondary, or defined vs. induced). * **Unlabeled Element:** The pink region is a significant visual component but lacks a textual label, implying it represents a subset, a neighborhood, or a region of interest rather than a single point. * **Internal Structure of 𝒜:** The triangle is not just a set of points; it has internal structure, as shown by the arrow from `α` to `a₂`. The vertices (`id`, `a₁`, `a₂`) may form a basis or a simplex. ### Interpretation This diagram likely illustrates a concept from machine learning, optimization, or functional analysis. It visually represents a **mapping from a function/feature space (ℱ) to an action/parameter space (𝒜)**. * **What it suggests:** The mapping could represent how different functions or models (`f*`, `f₁`, `f₂`) correspond to specific actions, parameters, or outcomes (`id`, `a₁`, `a₂`). The pink region mapping to `a₁` suggests that a whole family or neighborhood of functions in ℱ leads to the same outcome `a₁`. * **Relationships:** The diagram establishes a structured relationship between the two spaces. The point `f*` mapping to `id` might represent an optimal or reference function. The internal arrow from `α` to `a₂` indicates that the space 𝒜 has its own dynamics or relationships independent of ℱ. * **Role of the Expectation:** The overarching **E_{x∼p(x)}[·]** implies that these mappings or the properties being illustrated are considered in an average sense over a data distribution, which is a common theme in statistical learning theory and risk minimization. * **Anomaly/Notable Feature:** The most notable feature is the **unlabeled pink region**. Its presence and specific mapping to `a₁` (shared with `f₁`) highlight that the relationship between the spaces is not necessarily one-to-one and that regions in ℱ can have significant meaning. This could represent a concept like a "basin of attraction" or a "region of equivalence" in the function space. In essence, the diagram is a conceptual tool for visualizing how elements or regions in one mathematical space relate to elements in another, under the umbrella of an expected value calculation. </details> Figure 5: We connect concept extractors $f∈F$ and maps $α∈A$ through Eq. 20. Here, we represent the space of learnable concept extractors $F$ and the space $A$ as a two-dimensional simplex. The three vertices correspond to the concept remappings $id,{\bm{a}}_1,{\bm{a}}_2∈Vert(A)$ . $id$ is the intended solution, and the remaining mappings are deterministic RSs. From (Marconato et al., 2023b, Lemma 1), any concept remapping ${\bm{a}}∈Vert(A)$ is in one-to-one correspondence with a concept extractor $f∈F$ with domain restricted to $supp({\bm{X}})$ and, under Assumption 4.2, $f^*$ is in one-to-one correspondence with the identity function. However, non-deterministic concept remapping distributions $α∈A∖Vert(A)$ can arise from several concept extractors in $F$ , here represented by the magenta colored subset mapping to one point in the simplex $A$ . With this definition, we formalize reasoning shortcuts as concept remapping distributions that originate from faulty NeSy predictors. In contrast to (Marconato et al., 2023b, Definition 1), we refer to RSs as the maps $α∈A$ for a given $β∈B$ , not as the concept extractors $f∈F$ that induce faulty NeSy predictors. That is, a NeSy predictor $(f,β)∈F×B$ subsumes an RS $α_f$ when its concept extractor $f$ is optimal but does not match the ground-truth one, $f^*$ . The definition of RSs allows us to focus on concept remapping distributions $α∈A$ instead of concept extractors, that is, on the symbolic level rather than the sub-symbolic level. We depict the precise relation between the learnable concept extractors $F$ and concept remapping distributions $A$ in Fig. 5, where a one-to-one correspondence exists only between concept remappings $α∈Vert(A)$ and functions $f∈F$ restricted to the support of ${\bm{X}}$ . For the precise relation, refer to (Marconato et al., 2023b, Lemma 1). Studying concept remapping distributions is especially helpful under Assumption 4.2. Then, the concept remapping for the ground-truth concept extractor $f^*$ is the identity function $id(·)$ (Bortolotti et al., 2024), i.e., $$ α_f^*({\bm{G}})=id({\bm{G}}). \tag{24} $$ Notice that Definition 4.7 specializes RSs to the specific inference layer $β$ at hand. This, in turn, implies that some concept remapping distributions $α$ ’s can be an RS for one NeSy model class, say PNSPs, but not for another, say LTN. This may happen when two inference layers $β,β^\prime∈B$ differ in behavior when mapping non-deterministic distributions $p({\bm{C}}\mid{\bm{x}})∈Δ_C∖Vert(Δ_C)$ . This distinction is particularly evident between probabilistic logic approaches (which essentially implement integration through marginalization, like PNSPs and SL) and fuzzy logic methods (where the inference layer depends on the fuzzy logic chosen). However, all deterministic concept remappings $α∈Vert(A)$ that are RSs, are RSs for all inference layers. This is because, by Eq. 12, they output the same values on one-hot concept distributions $Vert(Δ_C)$ (see also (Marconato et al., 2023b, Appendix A)). We will refer to these RSs as deterministic RSs. Definition 4.8 (Deterministic Reasoning Shortcut). We say that $α∈Vert(A)$ is a deterministic reasoning shortcut if $α$ is a reasoning shortcut for $β^*∈B$ (Definition 4.7). **Example 4.9** *Consider a simplified version of BDD-OIA (Xu et al., 2020) with label $y∈\{\tt go,stop\}=Y$ , three binary concepts $(C_\tt green,C_\tt red,C_\tt ped)$ with domain $C$ , and prior knowledge $K=(C_\tt ped∨ C_\tt red⇔(Y=\tt stop))∧(C_\tt green∧\lnot(Y={\tt stop})⇔(Y=\tt go))$ . In Fig. 6, we visualize the four concept remapping distributions appearing in Fig. 5. The concept remapping distributions $α∈A$ determine how ground-truth concepts are mapped to conditional probability distributions on model concepts. First, we can find an RS ( ${\bm{a}}_1$ in Fig. 6 left-center) that predicts $C_\tt ped=1$ with probability one whenever $G_\tt red=1$ or $G_\tt ped=1$ . Similarly, we have an RS ( ${\bm{a}}_2$ in Fig. 6 right-center) that predicts $C_\tt red=1$ with probability one whenever $G_\tt red=1$ or $G_\tt ped=1$ . A nondeterministic RS ( $α$ in the rightmost part of Fig. 6) assigns non-zero probability to both $C_\tt red=1$ and $C_\tt ped=1$ .* <details> <summary>x4.png Details</summary>  ### Visual Description ## Diagram: Four Interaction Configurations Between Goal, Reward, and Policy ### Overview The image displays four schematic diagrams arranged horizontally, each illustrating a different configuration of relationships between three core components. The diagrams are labeled from left to right as: **id**, **a₁**, **a₂**, and **α**. Each diagram uses a consistent set of symbols: three vertical arrows labeled `g_l`, `r_l`, and `pe`, originating from colored circles at the bottom. The primary variation between diagrams is the presence and direction of curved connecting lines between these components, indicating different interaction or dependency patterns. ### Components/Axes * **Titles (Top of each box):** `id`, `a₁`, `a₂`, `α` * **Component Labels (Top of arrows):** `g_l`, `r_l`, `pe` * **Base Symbols (Bottom of arrows):** * A **green circle** at the base of the `g_l` arrow. * A **red circle** at the base of the `r_l` arrow. * A **smiley face icon** at the base of the `pe` arrow. * **Spatial Layout:** Each configuration is contained within a rectangular box. The three arrows are evenly spaced within each box. The connecting lines are drawn as smooth curves between the arrow shafts or bases. ### Detailed Analysis **Diagram 1: `id` (Leftmost)** * **Configuration:** All three components are independent. * **Flow:** Three straight, parallel vertical arrows (`g_l`, `r_l`, `pe`) point upward from their respective base symbols. There are no connecting lines between them. **Diagram 2: `a₁` (Second from left)** * **Configuration:** The `pe` component influences or connects to the `r_l` component. * **Flow:** The `g_l` arrow remains straight and independent. A curved line originates from the `pe` arrow's shaft and connects downward to the **red circle** at the base of the `r_l` arrow. The `r_l` arrow itself remains straight. **Diagram 3: `a₂` (Third from left)** * **Configuration:** The `r_l` component influences or connects to the `pe` component. * **Flow:** The `g_l` arrow remains straight and independent. A curved line originates from the `r_l` arrow's shaft and connects downward to the **smiley face icon** at the base of the `pe` arrow. The `pe` arrow itself remains straight. **Diagram 4: `α` (Rightmost)** * **Configuration:** A bidirectional or crossed interaction between `r_l` and `pe`. * **Flow:** The `g_l` arrow remains straight and independent. Two curved lines cross each other: 1. One line originates from the `r_l` arrow's shaft and connects to the **smiley face icon** at the base of the `pe` arrow. 2. Another line originates from the `pe` arrow's shaft and connects to the **red circle** at the base of the `r_l` arrow. ### Key Observations 1. **Consistent Element:** The `g_l` (green circle) component is always depicted as an independent, straight vertical arrow in all four configurations. 2. **Variable Interaction:** The core variation is in the relationship between `r_l` (red circle) and `pe` (smiley face). The diagrams progress from no interaction (`id`), to one-way influence from `pe` to `r_l` (`a₁`), to one-way influence from `r_l` to `pe` (`a₂`), and finally to a mutual or crossed interaction (`α`). 3. **Visual Encoding:** The use of color (green, red) and an icon (smiley face) provides immediate visual distinction between the three component types. The curved lines explicitly denote a directional relationship or flow of influence. ### Interpretation This diagram set likely illustrates different models or architectures for how a learning or decision-making system processes information. The labels are suggestive: * `g_l` could stand for "goal" or "guidance signal." * `r_l` could stand for "reward" or "reinforcement signal." * `pe` could stand for "policy" or "prediction error." The configurations demonstrate a progression of complexity in how the reward signal (`r_l`) and the policy/prediction (`pe`) interact: * **`id` (Independent):** The reward and policy are processed separately, with no direct feedback loop. This might represent a baseline or non-interactive system. * **`a₁` (Policy affects Reward):** The policy's output directly influences the reward signal. This could model a system where the agent's actions shape its own reward environment. * **`a₂` (Reward affects Policy):** The reward signal directly updates the policy. This is a classic reinforcement learning setup where rewards guide policy improvement. * **`α` (Mutual Interaction):** The reward and policy are fully coupled, each influencing the other. This represents the most complex, integrated system, potentially modeling advanced adaptive or co-evolutionary learning processes where the agent and its evaluation criteria dynamically shape each other. The consistent independence of `g_l` suggests it is an external, fixed objective or target that the interacting `r_l`-`pe` system operates upon. The diagrams effectively visualize the structural differences between these four conceptual models. </details> Figure 6: The four concept remapping distributions appearing in Fig. 5 according to Example 4.9. To simplify visualization, we display the concept remapping distributions $α$ as directed graphs where arrows denote how ground-truth concepts are mapped to the predicted concepts. On the left, $id$ corresponds to the ground-truth map $α_f^*$ from the data generation process. At the center, ${\bm{a}}_1,{\bm{a}}_2∈Vert(A)$ represent two deterministic RSs. On the right, $α∈A∖Vert(A)$ is a non-deterministic RS. Among these RSs, $C_\tt green$ is always predicted correctly, whereas $C_\tt red$ and $C_\tt ped$ are incorrectly predicted. #### 4.2.5 Characterization of deterministic reasoning shortcuts. The description using maps $α∈A$ is useful because we can count the number of deterministic RSs explicitly. As noted above, under Assumption 4.2, the ground-truth concept remapping $α_f^*$ is the identity $id(·)$ . Therefore, any $α∈Vert(A)$ that differs from the identity function is a deterministic RS. The following result makes the number of deterministic RSs explicit: Theorem 4.10 (Number of deterministic reasoning shortcuts (Marconato et al., 2023b)). Let $F$ be a space of concept extractors. Under Assumptions 4.1, 4.2 and 4.3, the number of deterministic RSs (Definition 4.7) is $\textstyle∑_α∈Vert(A_\mathcal{F)}1≤ft\{\bigwedge_{\bm{g}∈supp({\bm{G}})}(β^*∘α)({\bm{g}})=β^*({\bm{g}})\right\}-1 ,$ (25) where $A_F:=\{α∈A:∃ f∈F s.t. α=α_f\}$ . When this count is greater than $1$ , the NeSy task admits (deterministic) RSs. A NeSy predictor can attain high label accuracy by learning any one of them or any (non-deterministic) mixture thereof. Because the count does not depend on the particular choice of $β∈B$ , all NeSy architectures have the same deterministic reasoning shortcuts (Marconato et al., 2023b). This happens because all NeSy inference layers $β∈B$ integrating the prior knowledge $K$ satisfy Eq. 12. When $Vert(A_F)$ contains all maps, i.e., $Vert(A_F)=Vert(A)$ , and $supp({\bm{G}})$ is complete, i.e., $supp({\bm{G}})=C$ , it is possible to explicitly count the number of deterministic reasoning shortcuts in the task (Marconato et al., 2023b). Refer to (Marconato et al., 2023b, Appendix C) for the analytical expression of the count. The number of deterministic RSs rapidly explodes based on how many concept vectors correspond to a single label through $β$ . For example, if we can predict a label ${\bm{y}}_0$ from $M$ different concept vectors, the number multiplies by a factor of $M^M$ . Accounting for practical restrictions in the set $Vert(A_F)⊆Vert(A)$ due to, e.g., the choice of specific neural backbones, complicates the explicit counting of RSs, but this can be done using model counting, see Section 5.2.1. Theorem 4.10 only counts deterministic RSs. If the count is reduced to zero, then NeSy predictors may still suffer from non-deterministic RSs (those $α$ ’s in $A∖Vert(A_F)$ ). To ensure that we can deal with all RSs by avoiding deterministic RSs, we need to introduce an additional assumption on the inference layer $β$ : Assumption 4.11 (Extremality (Bortolotti et al., 2025)). The inference layer $β∈B$ satisfies extremality if, for all $λ∈(0,1)$ and for all ${\bm{c}}≠{\bm{c}}^\prime∈C$ such that $\operatorname*{argmax}_{\bm{y}∈Y}β({\bm{c}})_\bm{y}≠\operatorname*{argmax}_{\bm{y}∈Y}β({\bm{c}}^\prime)_\bm{y}$ , where $β(·)_\bm{y}$ is the ${\bm{y}}$ -th component of $β$ , we have $\max_{\bm{y}∈Y}β(λ1≤ft\{{\bm{C}}={\bm{c}}\right\}+(1-λ)1≤ft\{{\bm{C}}={\bm{c}}^\prime\right\})_\bm{y}<\max≤ft(\max_{\bm{y}∈Y}β({\bm{c}})_\bm{y},\max_{\bm{y}∈Y}β({\bm{c}}^\prime)_\bm{y}\right) .$ <details> <summary>x5.png Details</summary>  ### Visual Description ## 2D Line Plot: Convex Combination Function Analysis ### Overview The image displays a 2D line plot showing a V-shaped function. The plot illustrates how a specific mathematical expression, involving a parameter λ and a function β, varies as λ changes from 0 to 1. The function reaches its minimum value at the midpoint (λ=0.5) and its maximum values at the endpoints (λ=0 and λ=1). ### Components/Axes * **X-Axis:** * **Label:** `λ` (Greek letter lambda). * **Scale:** Linear scale from 0 to 1. * **Tick Marks:** Major ticks at 0, 0.2, 0.4, 0.6, 0.8, and 1. * **Y-Axis:** * **Label:** `max_{y∈{1,2}} β(λc + (1-λ)c')_y`. This is a complex mathematical expression indicating the maximum value over y (where y is either 1 or 2) of the function β applied to a convex combination of vectors `c` and `c'`. * **Scale:** Linear scale from 0.5 to 1. * **Tick Marks:** Major ticks at 0.5, 0.6, 0.7, 0.8, 0.9, and 1. * **Data Series:** * A single blue line connects three key points. * **Data Points:** Two prominent red circular markers are placed at the endpoints of the line. * **Annotations:** * **Top-Left Corner:** Text annotation `β(c)_1 = 1`. This is positioned directly above the red data point at (λ=0, y=1). * **Top-Right Corner:** Text annotation `β(c')_2 = 1`. This is positioned directly above the red data point at (λ=1, y=1). ### Detailed Analysis * **Trend Verification:** The blue line exhibits a symmetric V-shape. It slopes downward linearly from the left endpoint to the center, then slopes upward linearly from the center to the right endpoint. * **Data Points & Values:** 1. **Left Endpoint (λ=0):** The red marker and annotation confirm the value is exactly `y = 1`. The annotation `β(c)_1 = 1` suggests that when λ=0, the convex combination simplifies to vector `c`, and the first component (y=1) of β(c) equals 1. 2. **Minimum Point (λ=0.5):** The blue line reaches its lowest point. The y-value is visually exactly `0.5`. No annotation is present, but the line's vertex aligns perfectly with the 0.5 tick mark. 3. **Right Endpoint (λ=1):** The red marker and annotation confirm the value is exactly `y = 1`. The annotation `β(c')_2 = 1` suggests that when λ=1, the convex combination simplifies to vector `c'`, and the second component (y=2) of β(c') equals 1. * **Line Segments:** * From λ=0 to λ=0.5: The line is straight, decreasing from y=1 to y=0.5. * From λ=0.5 to λ=1: The line is straight, increasing from y=0.5 to y=1. ### Key Observations 1. **Perfect Symmetry:** The plot is perfectly symmetric about the vertical line λ=0.5. 2. **Linear Segments:** The function is piecewise linear, composed of two straight line segments. 3. **Endpoint Maxima:** The maximum value of the function (y=1) occurs at the boundaries of the domain (λ=0 and λ=1). 4. **Midpoint Minimum:** The minimum value of the function (y=0.5) occurs exactly at the midpoint of the domain (λ=0.5). 5. **Annotation Specificity:** The annotations at the endpoints are specific to different components (y=1 for c, y=2 for c'), indicating that the `max` operation selects the larger value between the two components at each λ. ### Interpretation This plot visualizes a fundamental property of the function `max_{y∈{1,2}} β(λc + (1-λ)c')_y` under the given conditions (`β(c)_1 = 1` and `β(c')_2 = 1`). * **What the data suggests:** The V-shape demonstrates that the maximum of the two components of β is highest when the input is purely one of the original vectors (`c` or `c'`). As the input becomes a mixture of the two vectors (as λ moves from 0 to 1), the maximum value decreases, reaching its lowest point when the mixture is an equal blend (λ=0.5). This is characteristic of a convex function. * **Relationship between elements:** The parameter λ controls the interpolation between vectors `c` and `c'`. The y-axis value tracks the "worst-case" or "dominant" output of the function β across its two components for each interpolated vector. The annotations define the boundary conditions that create the symmetric V-shape. * **Notable pattern:** The linearity of the segments suggests that the function β, or at least its maximum over the two components, behaves linearly with respect to the convex combination parameter λ in this specific example. The exact minimum value of 0.5 at λ=0.5 implies a specific relationship between the components of β(c) and β(c') that leads to this balanced outcome. This could represent a scenario of perfect trade-off or balance between two competing objectives or properties represented by the two components. </details> Figure 7: The extremality condition holds for PNSPs and SL. A NeSy predictor satisfies Assumption 4.11 if its inference layer $β$ is “peaked”: for any two concept vectors ${\bm{c}}$ and ${\bm{c}}^\prime$ with different predictions, the label probability given by $β$ for a mixture of these predictions is no larger than the label probability it associates to ${\bm{c}}$ and ${\bm{c}}^\prime$ . This happens in PNSPs on MNIST-Add, where the knowledge specifies that $β:({\bm{C}}=(0,1)^⊤)↦(Y=1)$ and $β:({\bm{C}}=(0,2)^⊤)↦(Y=2)$ , both with probability one. Any convex combination crossing ${\bm{c}}=(0,1)^⊤$ to ${\bm{c}}^\prime=(0,2)^⊤$ would not result in any label with probability one, see Fig. 7. E.g., PNSPs will assign a higher probability to the sum $Y=1$ when $λ<0.5$ . In essence, distributing probability mass across different sets of concepts does not increase label likelihood. Extremality holds for PNSPs and SL, but for LTN it depends on the fuzzy logic chosen (van Krieken et al., 2022). Under Assumption 4.11, the next result holds: Theorem 4.12 (Identifiability (Bortolotti et al., 2025)). Under Assumptions 4.1, 4.2 and 4.3 and for all $β∈B$ satisfying Assumption 4.11, any NeSy predictor $(f,β)∈F×B$ attaining maximum likelihood (Theorem 4.4) finds the ground-truth concept extractor, i.e., $f({\bm{X}})=f^*({\bm{X}})$ , if and only if the count of deterministic RSs is zero (Eq. 25). This theorem indicates a viable way to mitigate RSs for all NeSy predictors that obey Assumption 4.11. In fact, zeroing the count on deterministic RSs (Theorem 4.10) implies avoiding all RSs in the learning task. For NeSy predictors model classes obeying Assumption 4.11, the identifiability result gives that, in the absence of deterministic RSs, maximizing the likelihood on labels permits the correct grounding of the concepts. This, in turn, presupposes finding additional mitigations that, paired with maximum-likelihood Eq. 2, guarantee the updated count of deterministic reasoning shortcuts equals zero. We will discuss next (Section 5.3) how different mitigations act on the count reduction. **Remark 4.13** *As a corollary of Theorem 4.12, for probabilistic logic methods (including PNSPs like DPL (Manhaeve et al., 2018), SL (Xu et al., 2018), and SPL (Ahmed et al., 2022)), all non-deterministic RSs can be constructed from convex combinations of deterministic RSs (Marconato et al., 2023b, Proposition 3). This guarantees that if there are no deterministic RSs, then there are also no other RSs. Moreover, as will be discussed in Section 5.4, it is possible to leverage this geometrical aspect to construct non-deterministic RSs from convex combinations of deterministic ones, whose uncertainty on (some of) the learned concepts reveals which of them are affected by RSs.* ### 4.3 Perspective: Statistical Learning Learning a NeSy model amounts to learning (the parameters of) a concept extractor that—once combined with the inference layer—attains high label likelihood. From a statistical learning perspective, this is equivalent to finding a model with minimal empirical label risk. Statistical learning theory allows one to bound the true risk of a model—that is, to assess how well it generalizes—based on its observed empirical risk and other factors, such as training set size and model complexity (Vapnik, 2013; Shalev-Shwartz and Ben-David, 2014). Since the inference layer of a NeSy model is fixed and supplied upfront via the background knowledge $K$ it is natural to ask when and how minimizing the label risk causes a reduction of the concept risk. Or similarly, when does low risk on label predictions bound the mismatch between the predicted and the ground-truth concepts? We expect that if RSs are present, such a bound could be vacuous for some concepts. In fact, from the results in the previous section, we know that the concepts predicted by a faulty NeSy predictor can drastically differ from ground-truth ones, even when the model is trained with an infinite amount of data.To make this explicit, we follow the analysis of (Yang et al., 2024), which indicates from a statistical learning perspective the relation between label and concept risks. #### 4.3.1 Knowledge complexity The statistical learning perspective on RSs follows steps similar to identifiability analysis (Section 4.2). First, data are assumed to be generated as per Assumption 4.1, whereby the ground-truth concepts and labels are sampled from the ground-truth concept extractor $f^*:X→Δ_C$ and inference function $β^*:Δ_C→Δ_Y$ . From the joint distribution $p^*({\bm{X}},{\bm{G}},{\bm{Y}})$ of Eq. 10, it is possible to determine the marginal distribution over labels $p^*({\bm{Y}})$ . This marginal distribution is used together with the knowledge $K$ to determine the complexity of the knowledge: Definition 4.14 (Knowledge complexity (Yang et al., 2024)). The knowledge complexity of $K$ for labels distributed according to the label distribution $p^*({\bm{Y}})$ (Assumption 4.1) is defined as $KC(K;p^*):=E_{\bm{y}∼ p^*({\bm{Y}})}[∑_{\bm{c}∈C}1≤ft\{({\bm{c}},{\bm{y}})\not\modelsK\right\}] .$ (26) This quantity gives a global description of how many concepts are not compatible with the knowledge, based on the labels ${\bm{y}}∈supp({\bm{Y}})$ observed during training. Intuitively, when a few concepts are associated with a single label vector, the knowledge complexity increases, indicating that the knowledge is more explicit about what concept vectors lead to that label vector. On the other hand, a low knowledge complexity indicates that many concept vectors are mapped to same label vector. For example, in the go/stop example of BDD-OIA (Example 2.1) with three binary concepts ( $|C|=8$ ), the go label is predicted only when $C_green=1$ , $C_ped=0$ , and $C_red=0$ , giving $7$ out of $8$ concept vectors ${\bm{c}}$ such that $({\bm{c}},\texttt{go})\not\modelsK$ . The label stop, on the contrary, gives only $1$ out of $8$ concept vectors ${\bm{c}}$ such that $({\bm{c}},\texttt{stop})\not\modelsK$ . Hence, whenever both go and stop instances are observed, we get $KC(K;p^*)<|C|-1=7$ . Notice that, by construction, the complexity of any knowledge $K$ on any distribution $p^*({\bm{Y}})$ over labels is upper-bounded: $$ KC(K;p^*)≤|C|-1 . \tag{27} $$ Next, we show how the complexity of the knowledge is intimately connected to the emergence of RSs. #### 4.3.2 Impossibility result in bounding the reasoning shortcut risk We start by recalling the relevant notation introduced in Section 4.2. For a NeSy predictor $(f,β)∈F×B$ , the corresponding concept and label distributions are given by $$ p_f({\bm{C}}\mid{\bm{X}}):=f({\bm{X}}), p_f({\bm{Y}}\mid{\bm{X}};K):=(β∘ f)({\bm{X}}) . \tag{28} $$ We assume, as before, that the space of learnable concept extractors $F$ is unrestricted and that the inference layer $β$ matches the ground-truth inference layer at the vertices (Eq. 12). The expected label risk and expected concept risk are given by $$ R_\bm{Y}(f):=-E_({\bm{x},{\bm{y}})∼ p^*({\bm{X}},{\bm{Y}})}[\log p_f({\bm{y}}\mid{\bm{x}};K)], R_\bm{C}(f):=-E_({\bm{x},{\bm{g}})∼ p^*({\bm{X}},{\bm{G}})}[\log p_f({\bm{C}}={\bm{g}}\mid{\bm{x}})] , \tag{29} $$ respectively. Their difference gives the expected reasoning shortcut risk (Yang et al., 2024): $$ R_RS(f):=R_\bm{C}(f)-R_\bm{Y}(f) . \tag{30} $$ Notice that the RS risk, when it is positive, indicates that concept risk exceeds label risk; hence, despite having more accurate label predictions, concepts are less accurately predicted. Vice versa, a concept risk less than or equal to zero indicates that concept predictions are at least as accurate as label predictions. The next result connects RSs to the knowledge complexity showing that, when it is not high enough, an analogous result to Corollary 4.5 holds: Theorem 4.15 (Unbounded RSs risk (Yang et al., 2024)). Under Assumption 4.1, if the knowledge complexity is bounded by the size of the concept space as $KC(K;p^*)<|C|-1$ , then there is a concept extractor $f$ such that $R_\bm{Y}=0$ and $R_\bm{C}→∞$ , so that the RS risk approaches infinity, i.e., $R_RS→∞ .$ (31) Notice that the main causes for RSs are that: (1) the knowledge complexity does not reach the maximum, and therefore each label ${\bm{y}}∈supp({\bm{Y}})$ is associated to multiple concept vectors ${\bm{c}}∈C$ , (2) the fact that the space of learnable concept extractors is unrestricted and can express all possible conditional concept distributions. Both causes are targets to design mitigation strategies: (1) making knowledge more explicit by increasing the knowledge of the model, and (2) reducing the space of learnable concept extractors $F$ by constraining the family of concept extractor, either through architectural biases, or through concept supervision. This perspective on RSs can naturally be used to estimate bounds on the reasoning shortcut risk, which was done for the case of concept smoothing and pretraining of the concept extractor. We refer the reader to (Yang et al., 2024) for more details on these bounds. At the same time, we will provide in Section 5.3 the main intuitions behind how mitigation strategies that can reduce the RS risk. PAC learnability with factorized concept extractors Theorem 4.15 shows that for an unconstrained space $F$ of concept extractors and a not-enough-complex knowledge $K$ , the NeSy task is subject to RSs. As mentioned above, restricting the space of learnable concept extractors is one primary route to investigate whether RSs can be mitigated. The work by Wang et al. (2023) investigate one special case which is often available in NeSy tasks: when the input can be naturally decomposed in a tuple of independent units, e.g., the digits in MNIST-Add, and the same concept extractor can be employed to predict the concepts of each input instance, e.g., the value of each digit is obtained by mapping the input alone, what conditions are required to avoid RSs and guarantee correct learning of the concept extractor? This setting, which builds on factorized concept extractors of the form $f=\bar{f}_1×⋯×\bar{f}_k$ , has been studied via the probably approximate correct (PAC) framework (Shalev-Shwartz and Ben-David, 2014), and connects the learnability of NeSy predictors to that of weakly-supervised models (Steinhardt and Liang, 2015; Raghunathan et al., 2016). #### 4.3.3 Setting Following (Wang et al., 2023), we consider training samples described as pairs $({\bm{x}},y)∈X×Y$ , Following (Wang et al., 2023), we consider one categorical label associated to each input $\bar{\bm{x}}_i$ and we explicitly use a scalar label $y$ instead of the vector-valued ${\bm{y}}$ , as in previous sections. where inputs ${\bm{x}}$ represent a sequence of high-dimensional vectors ${{\bm{x}}=(\bar{\bm{x}}_1,…,\bar{\bm{x}}_k)}∈X=\bar{}X^k$ and $\bar{}X⊆ℝ^n$ . For example, this is the case when sequence elements $\bar{\bm{x}}_i$ are MNIST digits and the final label corresponds to their sum (Example 3.2). In this setting, we can write the overall concept vector ${\bm{c}}=(c_1,…,c_k)∈C$ , where the concept space repeats a fixed set of symbols $\bar{}C$ $k$ times, i.e., $C=\bar{}C^k$ . We also consider a restricted space of learnable concept extractors $f∈F$ constructed as $$ f({\bm{x}})=(\bar{f}(\bar{\bm{x}}_1),…,\bar{f}(\bar{\bm{x}}_k)), \tag{32} $$ where a shared classifier $\bar{f}:\bar{}X→Δ_\bar{C}$ maps sequence elements $\bar{\bm{x}}_i$ to a probability distribution over the possible values of the concept. Hence, each MNIST digit is processed independently from the others and mapped by the same classifier $\bar{f}∈\bar{}F$ , where $\bar{}F$ is a hypothesis class of classifiers. This construction restricts the space of learnable concept extractors $F$ to be factorized as $F=\bar{}F^k$ . This is equivalent to imposing that the concept extractor is architecturally disentangled, as we will discuss in Section 5.3.8. In contrast, both the results of Corollaries 4.5 and 4.15 assume a non-factorized concept extractor space and violate this construction. In this section, we also use $[\bar{f}](\bar{\bm{x}}):=\operatorname*{argmax}\bar{f}(\bar{\bm{x}})$ to denote the concept-wise prediction of $\bar{f}$ . By construction, we have that $[\bar{f}]:\bar{}X→\bar{}C$ , and we denote with $[\bar{}F]$ the space of these functions. Training data $({\bm{x}},y)∈X×Y$ are distributed according to a joint $p^*({\bm{X}},Y)$ and we refer to $p^*({\bm{X}},{\bm{G}})$ for the joint distribution of inputs and ground-truth concept vectors. Similarly to Section 4.3, we consider the concept risk associated to the concept extractor, but for the zero-one loss, defined as $\ell^01_\bm{C}(c,c^\prime):=1≤ft\{c≠ c^\prime\right\}$ for two values $c,c^\prime∈\bar{}C$ . In this case, we have the expected concept risk $$ \textstyleR^01_\bm{C}(f):=E_({\bm{x},{\bm{g}})∼ p^*({\bm{X}},{\bm{G}})}≤ft[\frac{1}{k}∑_j=1^k\ell^01_\bm{C}([\bar{f}](\bar{\bm{x}}_j),g_j)\right]. \tag{33} $$ Also, we consider the zero-one loss of the label for a given $β^*∈B$ , also referred to as the zero-one partial loss in (Wang et al., 2023), given by $\ell^01_β^*({\bm{c}},y):=1≤ft\{β^*({\bm{c}})≠ y\right\}$ . The expected label risk is given by $$ R^01_\bm{Y}(f,β^*):=E_({\bm{x},y)∼ p^*({\bm{X}},Y)}≤ft[\ell_β^*^01\big(([\bar{f}](\bar{\bm{x}}_1),…,[\bar{f}](\bar{\bm{x}}_k)),y\big)\right]. \tag{34} $$ When there is a concept extractor ${f^*∈F}$ such that ${R}^01_\bm{Y}(f^*;β^*)=0$ , we say that the space $F$ is realizable under $\ell^01_β^*$ . As customary in empirical risk minimization, we consider learning the concept extractor through a finite number of samples $({\bm{x}},y)$ , which we denote by $m_p^*$ . The corresponding dataset of these pairs is denoted as $D_p^*$ . Here, the empirical risk associated to the dataset is defined as $$ \hat{R}^01_\bm{Y}(f,β^*,D_p^*):=\frac{1}{m_p^*}∑_({\bm{x},y)∈D_p^*}\ell_β^*^01(([\bar{f}](\bar{x}_1),\dots,[\bar{f}](\bar{x}_k)),y). \tag{35} $$ For PAC learnability of this NeSy task (Shalev-Shwartz and Ben-David, 2014; Wang et al., 2023), we ask if for any given distribution $p^*({\bm{X}},Y)$ of input-label pairs and any $ε,δ∈(0,1)$ , the zero-one concept risk of $f$ is less or equal to $ε$ , i.e., $R_\bm{C}^01(f)≤ε$ , with probability at least $1-δ$ when the number of training samples $m_p^*$ is greater or equal than an integer $m_ε,δ$ . Next, we present how Wang et al. (2023) derive such a $m_p^*$ for this learning setting. #### 4.3.4 PAC learnability with $k$ -unambiguous inference layers To prove the PAC learnability of the NeSy task, we have to find the conditions that $β^*$ should satisfy so that the concept risk ${R_\bm{C}^01(f)}$ is bounded by the label risk ${R^01_\bm{Y}(f;β^*)}$ for any distribution of the training data. The next condition is central to find such a guarantee: Definition 4.16 ( $k$ -unambiguity (Wang et al., 2023)). An inference layer $β^*$ is $k$ -unambiguous if for any two concept vectors ${\bm{c}}=(c,\dots,c)$ and ${\bm{c}}^\prime=(c^\prime,\dots,c^\prime)$ $∈C$ , such that $c≠ c^\prime$ , we have that ${β^*({\bm{c}}^\prime)≠β^*({\bm{c}})}$ . Otherwise, the inference layer $β^*$ is $k$ -ambiguous. This definition essentially guarantees that passing concept vectors of the form ${\bm{c}}=c(1,…,1)^⊤$ and ${\bm{c}}^\prime=c^\prime(1,…,1)^⊤$ to $β^*$ never returns the same label predictions for $c≠ c^\prime$ , guaranteeing the injectivity of $β^*$ for these pairs. Although Definition 4.16 might seem strict, NeSy tasks like MNIST-Add satisfy this condition, see Example 3.2. Moreover, it is possible to extend $k$ -unambiguity to the case where $β^*$ is non-deterministic and to establish the relationship with small ambiguity degree from partial label learning (Liu and Dietterich, 2014). With this condition, the following result holds: Lemma 4.17 (Risk bounds under $k$ -unambiguity (Wang et al., 2023)). If the space of learnable concept extractors $F$ is constructed like in Eq. 33 and $β^*$ is $k$ -unambigous, then we have: $R^01_\bm{C}(f)≤O(R^01_\bm{Y}(f;β^*)^1/k) as R^01_\bm{Y}(f;β^*)→ 0$ (36) Moreover, if $β^*$ is $k$ -ambiguous, then a concept extractor $f$ attaining label risk ${R^01_\bm{Y}(f;β^*)=0}$ can have a concept risk of ${R^01_\bm{C}(f)=1}$ . This result highlights that $k$ -unambiguity is central to avoiding RSs for the specific concept extractors modeled in $F$ . In fact, whenever $β^*$ is $k$ -ambiguous, concept risk can be non-zero, resulting in RSs. This happens naturally for the XOR of two bits, but not for the sum of the two, as detailed next. **Example 4.18** *Consider a setting where the input ${\bm{x}}=(\bar{\bm{x}}_1,\bar{\bm{x}}_2)$ comprises two bits, e.g., ${\bm{x}}=(\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-0.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}})$ . Consider the knowledge for the XOR between the two, i.e., $K=(C_1⊕ C_2⇔ Y)$ . The function $β^*$ constructed from $K$ is $k$ -ambiguous ( $k=2$ ), as there are two vectors with only the same concept value that return the label. That is, $β^*((0,0)^⊤)=β^*((1,1)^⊤)$ , since $0⊕ 0=1⊕ 1=0$ . We can see that, in this setting, there is a deterministic RS that confuses the zero for one and vice versa, that is, $\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-0.png}}↦ 1$ and $\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}}↦ 0$ . Consider now, instead, the sum of the two digits, with the knowledge given by $K=(Y=C_1+C_2)$ . Here, the inference layer $β^*$ is $k$ -unambiguous ( $k=2$ ), and there are no concept remapping distributions other than the identity that give the correct label prediction. $k$ -unambiguity is also satisfied for the complete MNIST-Add, which is a NeSy task that does not have any RS for the disentangled concept extractors in $F$ .* Based on this result, Wang et al. (2023) show that it is possible to guarantee PAC learnability of the NeSy task. The result incorporates the Natarajan dimension of the space of classifiers $[\bar{f}]$ , denoted as $d_[\bar{F]}$ . We refer readers to (Shalev-Shwartz and Ben-David, 2014) for the explicit definition of the Natarajan dimension. Theorem 4.19 (ERM learnability under $k$ -unambiguity (Wang et al., 2023)). Let $F$ be the space of learnable concept extractors $F$ constructed like in Eq. 33. Suppose that $F$ is realizable under $\ell^01_\bm{Y}$ and $[\bar{}F]$ has a finite Natarajan dimension $d_[\bar{F]}$ . Then, for any $ε,δ∈(0,1)$ , there exists a universal constant $C_0>0$ , such that with probability at least $1-δ$ , the empirical label risk minimizer $f$ on the dataset $T_p^*$ , such that $\hat{R}^01_\bm{Y}(f;β^*;T_p^*)=0$ , has a concept risk $R^01_\bm{C}(f)<ε$ , if $\displaystyle m_p^*≥ C_0\frac{η_1}{ε^k}≤ft(η_2\log\frac{1}{ε^k}+\log\frac{1}{δ}+η_3\right),$ (37) where $η_1,η_2,η_3∈ℝ^+$ are constants that depend on $k$ , the $d_[\bar{F]}$ , and the number of labels $|Y|$ . We refer the reader to (Wang et al., 2023) for the explicit values of the constants. Furthermore, the work presents other results regarding the PAC learnability of the NeSy task under unknown $β^*$ ’s and error bounds when considering the semantic loss (Xu et al., 2018). ### 4.4 Relationship between Theories A common trait between the two perspectives can be found in Corollary 4.5 and Theorem 4.15: both results indicate that, even in the limit of infinite data, when conditions on the knowledge are not met (either because of non-injectivity or low knowledge complexity), RSs can be found during training. Both results highlight a general problem in NeSy predictors: when we have a large hypothesis space of learnable concept extractors $F$ and if the knowledge admits multiple solutions for label predictions, there is no unique and correct grounding of the concepts. Vice versa, if a one-to-one mapping between concept and labels is implemented by the knowledge (both because the ground-truth inference layer $β^*$ is injective in Corollary 4.5 and $KC(K,p^*)=|C|-1$ in Theorem 4.15), achieving optimal likelihood on data implies learning well-grounded concepts. Achieving this condition, however, may be difficult in practice whenever the number of possible concept vectors exceeds that of labels, see e.g. BDD-OIA in Example 4.9. Constrained hypothesis spaces for the concept extractor can likelier meet the conditions for avoiding RSs with a less restricted $K$ , as shown in Lemma 4.17, and lead to statistical learning bounds for recovering ground-truth concepts with finite-sized estimates (Theorem 4.19). The two perspectives differ in how they specialize in treating mitigations. The identifiability perspective (Marconato et al., 2023b, 2024) has investigated so far the impact of strategies on count reduction for deterministic $α$ ’s, studying how different mitigations change the quantities in the count (Eq. 25). This also includes evaluating how the count reduces when different strategies are used together, but it is limited to study optima of the learning problem. The statistical learning perspective (Wang et al., 2023; Yang et al., 2024) has mainly explored how different mitigations that constrain the space of learnable concept extractors $F$ can be used to bound the empirical RSs risk. These results do not explicitly require studying the optima of the learning problem, in the sense that, constraints on $F$ can be directly translated to bounds on the RS risk (Eq. 30). In fact, strategies like concept smoothing change the optima of the learning problem, and may not allow reaching maximum likelihood. Despite the different treatments, we advance that the effect of some mitigation strategies can be described from both perspectives: a mitigation strategy can both reduce the count in Eq. 25 and reduce the risk Eq. 30. In this sense, we view both perspectives as complementary: through identifiability theory, one can analyze how mitigations act to reduce deterministic RSs at optimal likelihood, whereas through statistical learning theory, bounds on the empirical concept risk can be found even when likelihood is not optimal. Formulating a theory that takes both perspectives into account remains open. ## 5 Handling Reasoning Shortcuts We proceed by detailing the root causes of RSs (in Section 5.1), and outlining existing diagnostic tools (Section 5.2), mitigation strategies (Section 5.3), and awareness strategies (Section 5.4). ### 5.1 Root Causes As anticipated in Section 3.1, RSs arise whenever the NeSy learning process does not penalize models that learn incorrect concepts, i.e., concepts that do not match the ground-truth ones. Moreover, the theory in Section 4 provides additional details. Specifically, Theorem 4.15 suggests that RSs can occur when the knowledge complexity $KC$ (Definition 4.14) is less than $|C|-1$ and the space of learnable concept extractors $F$ is broad enough. Corollary 4.5 supports the same conclusion. Following (Marconato et al., 2023b), additional insights can be gleaned by studying the structure of Eq. 25, reported below for reference, which counts the number of deterministic RSs (Definition 4.8) affecting a NeSy task. Since this is useful for understanding the mitigation strategies in Section 5.3, we analyze it here: learnable maps $α$ support prior knowledge $K$ optimality condition $L$ The count is controlled by the four colored elements, which we discuss below. #### 5.1.1 The knowledge The prior knowledge $K$ determines the inference layer $β^*$ . To understand its role, | 0 | 0 | 0 | 0 | | --- | --- | --- | --- | | 0 | 1 | 1 | 1 | | 1 | 0 | 2 | 1 | | 1 | 1 | 3 | 1 | Figure 8: $K_1$ is immune to RSs, $K_2$ is not. imagine two NeSy tasks with two binary concepts that are identical except for the prior knowledge $K$ : one uses $K_1=(Y_1=2C_1+C_2)$ and the other $K_2=(Y_2=C_1∨ C_2)$ , see Fig. 8. Assuming the training set covers all classes (see Section 5.1.2), the task using $K_1$ does not admit any RSs: each label $y$ can only be inferred by a unique choice of ${\bm{c}}$ , so high label accuracy entails high concept accuracy. The task using $K_2$ , however, is affected by RSs: the positive label can be inferred from ${\bm{c}}=(0,1)$ , ${\bm{c}}=(1,0)$ and ${\bm{c}}=(1,1)$ , hence NeSy predictors can confuse them with no impact on accuracy. From the perspective of Eq. 38, this happens because, all else being equal, swapping $K_1$ with $K_2$ softens the equality condition, increasing the count. #### 5.1.2 The training distribution Another important factor is the set of configurations of ground-truth concepts ${\bm{g}}$ that underly the training data, that is, the support $supp({\bm{G}})$ of the training distribution. To see this, consider the MNIST-Add example in Example 3.2. Suppose we add two new examples, $(\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}},10)$ and $(\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}},8)$ , to the training set. Intuitively, the concept extractor has to map <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> to the digit $5$ , otherwise it would mispredict the $(\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}},10)$ example; likewise, now that <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> has only one correct grounding, it has to map <details> <summary>figures/mnist-4.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> to $4$ otherwise it would mispredict the $(\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}},9)$ example. The same reasoning applies for the remaining examples, namely ( <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> <details> <summary>figures/mnist-3.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> , 8) and ( <details> <summary>figures/mnist-3.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> <details> <summary>figures/mnist-2.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> , 5). In summary, this small change completely removes all RSs, as the only way to match all training examples is to assign each MNIST digit to its correct numeric value. Intuitively, the larger the support – i.e., the more combinations of ground-truth concepts ${\bm{g}}$ the training set represents – the more restrictive the conjunction in Eq. 38, lowering the count. #### 5.1.3 The optimality condition The optimality condition $L$ is responsible for filtering out those maps $α$ that do not fit the ground-truth labels. Consider Example 3.2 again. One possible RS is $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 4,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 1,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 5,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 4\}$ , which collapses <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> and <details> <summary>figures/mnist-2.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> into the same concept value, $4$ . Now, imagine augmenting the training objective, by default the cross-entropy, with a reconstruction loss. This mapping no longer achieves low loss, since the value $4$ decodes to both <details> <summary>figures/mnist-2.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> and <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> , which clearly hinders reconstruction. In terms of Eq. 38, modifying the training objective changes the optimality condition, potentially decreasing the number of RSs. #### 5.1.4 The family of learnable maps Finally, the sum in Eq. 25 runs over (the vertices of) the space of learnable concept remappings $α$ , denoted by $Vert(A_F)$ . This, in turn, depends on the space of learnable concept extractors $F$ (Theorem 4.10). By introducing architectural bias in the architecture of the concept extractor (e.g., smoothing concept extractors by tuning a temperature parameter), we can constrain the space of learnable concept remappings $α$ ’s and therefore decrease the number of learnable RSs $Vert(A_F)$ . We will see in Section 5.3 how to effectively control this aspect of the learning problem. ### 5.2 How to Diagnose Reasoning Shortcuts As mentioned in Section 3, in-distribution label accuracy is not affected by RSs and therefore insufficient for diagnosing them. In this section, we briefly outline more effective diagnostic techniques. #### 5.2.1 Task-level diagnosis Even before we commit to an architecture and train the model, we can quantify how many deterministic RSs can affect a given NeSy task. We accomplish this by evaluating Eq. 25, which is defined on known properties, such as the number of concepts, their cardinality and the given annotations. However, the count is very hard to compute manually, even for relatively small problems, as the number of maps $α$ increases doubly exponentially with the number of concepts $k$ . The countrss tool (Bortolotti et al., 2024) works around this by building on the following observation: all deterministic maps $α$ in $Vert(A)$ can be viewed as binary matrices that indicate which predicted concepts $C_j$ correspond to which ground-truth concepts $G_i$ . The assumptions that constrain the number of mappings can be encoded into a propositional logical formula, whose solutions (that is, admissible binary matrices) are in one-to-one correspondence to the deterministic RSs $α$ . Then, the count in Eq. 25 can be reduced to model counting (#SAT) (Gomes et al., 2021), i.e., the task of counting the solutions of a logical formula. | $G_\tt red$ $¬ G_\tt red$ $G_\tt ped$ | 0 0 1 | 0 0 0 | 1 0 0 | 0 1 0 | | --- | --- | --- | --- | --- | | $¬ G_\tt ped$ | 0 | 1 | 0 | 0 | (a) | $G_\tt red$ $¬ G_\tt red$ $G_\tt ped$ | 1 0 0 | 0 1 0 | 0 0 0 | 0 0 1 | | --- | --- | --- | --- | --- | | $¬ G_\tt ped$ | 0 | 0 | 0 | 1 | (b) Figure 9: Binary matrices encoding two possible mappings $α$ from Example 2.1. (a) is a solution, resulting in a mapping that mistakes red lights with pedestrians and vice versa but is otherwise consistent with $K$ for any combination of ground truth concepts. (b) is not a solution, since the resulting mapping makes the predictor unable to detect pedestrians and give correct predictions on some, but not all, combinations of ground truth concepts. Although one could use any #SAT procedure, the complexity of the encoding for any task of practical interest warrants the use of approximate #SAT solvers (Chakraborty et al., 2021). Under the hood, countrss employs the state-of-the-art hashing-based counter ApproxMC https://github.com/meelgroup/approxmc, which outputs an $ε$ approximation of the exact count with probability $δ$ . countrss inherits these statistical (PAC-style) guarantees, providing close approximations to Eq. 25 with high probability Regardless of the solver used, the reduction assumes disentanglement. Otherwise, the resulting number of logical variables in the encoding would be orders of magnitude above the capabilities of current solvers.. The estimates returned by countrss can provide an initial indication of the hardness of the learning problem and help making informed architectural choices according to the task at hand. countrss is included in the rsbench library (Bortolotti et al., 2024). #### 5.2.2 Model metrics Whenever concept annotations are available, the most effective way to evaluate the quality of the learned concepts is to use standard metrics such as accuracy, $F_1$ score, and concept-level confusion matrices. Another useful metric is concept collapse (Bortolotti et al., 2024). Roughly speaking, it quantifies to what extent the learned concept extractor compresses distinct ground-truth concepts into the same learned concept. More formally, let $C∈[0,1]^m× m$ be a concept confusion matrix, where $m$ denotes its size (e.g.,, $m=2^k$ when all ground-truth concepts are observed). We define $Cls=1-\tfrac{p}{m}$ , where $p=∑_j=1^m1≤ft\{∃ i : C_ij>0\right\}$ . We remark that, however, collapse cannot detect all RSs. For instance, imagine a NeSy predictor trained on some variant of MNIST-Add affected by an RS mapping <details> <summary>figures/mnist-0.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> to $1$ and <details> <summary>figures/mnist-1.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> to $0$ . This RS is completely invisible to the collapse metric, as the model does not collapse ground-truth concepts together, cf. Section 5.3.6. Still, it can be useful to gauge the presence of RSs that do. Finally, a major downside of these metrics is that they all require concept-level annotations, which are not always available. With insufficient or missing concept annotations, another approach is to train multiple NeSy predictors to high label accuracy, and check their concept extractors align on concept predictions. When (deterministic) RSs are many in a NeSy task, it is likely that each predictor finds a different RS. Furthermore, if there are no RSs, concept extractors giving optimal likelihood must predict the same concepts, although the reverse is not necessarily true. The alignment between different concept extractors of the ensemble can be indirectly measured via the overall conditional entropy of the averaged concept predictions of the ensemble. Precisely, for $r∈ℕ$ members of the ensemble, we can account the mean conditional Shannon entropy $H(\frac{1}{r}∑_jp_j({\bm{C}}\mid{\bm{x}}))$ . This quantity is central for unsupervised RS-aware methods such as bears (Marconato et al., 2024) and NeSyDM (van Krieken et al., 2025b), see Section 5.4. ### 5.3 A Tour of Mitigation Strategies A wealth of strategies for mitigating RSs have been proposed. We list them in Table 2 and discuss them next. #### 5.3.1 Concept supervision The most direct approach for encouraging correct grounding is to exploit concept annotations during training, thus altering the optimality condition. This requires introducing an additional loss term penalizing mispredicted concepts. Doing so effectively constraints the concept extractor $f$ to recover the annotations, dramatically reducing the number of learnable RSs, and potentially avoiding them altogether. This solution is however neither cheap nor perfect. It can be costly because concept-level annotations are frequently unavailable in the wild, and collecting them involves running potentially expensive annotation campaigns. Following recent trends (Oikarinen et al., 2022; Yang et al., 2023b; Rao et al., 2024; Srivastava et al., 2024; Yuksekgonul et al., 2023), one could obtain these with foundation models. This solution, however, risks injecting substantial amounts of noise into the learning loop (Debole et al., 2025): despite being trained on vast amounts of data, foundation models can mispredict even basic concepts (Wüst et al., 2025). It is also imperfect because, in the worst case, ruling out all RSs requires exhaustively annotating all possible combinations of concepts. Unless this is the case, a sufficiently expressive concept extractor might fit all annotations on the supervised combinations and fluke the remaining ones. Naturally, annotating an exponential (in the number of concepts) number of combinations can be infeasible. A possible work-around is to restrict the expressivity of the concept extractor – e.g., by processing inputs separately, as typically done in MNIST-Add – so as to effectively transfer concept annotations across concept combinations. Another option is to smartly select instances or concepts to be annotated, see Section 5.5. Table 2: Overview of mitigation strategies. The columns indicate, for each strategy, what component(s) of Eq. 38 it targets (Target), whether it requires extra annotations (Annot) and guarantees reducing the count in Eq. 25 (Count), and how well it worked on different NeSy Tasks. Specifically, “ ✓ ” means that it prevents all RSs, “ ✗ ” that it fails to do so, and “? ” that it has not been evaluated. Note: With MNIST-Add ∗ we refer to the family of datasets based on the MNIST-Add benchmark (Manhaeve et al., 2018). Strategy Properties Arithmetic Logic High-Stakes Target Annot Count MNIST-Add ∗ Kandinsky Clevr BDD-OIA Concept Supervision (Koh et al., 2020) ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}L}$ ✓ ✓ ✓ ✓ ✓ ✗ Multi-task Learning (Caruana, 1997) ${\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}K}$ ✓ ✓ ✓ ? ? ? Weak Supervision (Yang et al., 2024) ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}L}$ ✗ ✗ ✓ ? ? ✗ Entropy Maximization (Manhaeve et al., 2021a) ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}L}$ ✗ ✗ ? ? ? ✗ Smoothing (Szegedy et al., 2016) ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}L}$ ✗ ✗ ? ? ? ? Reconstruction (Kingma, 2013) ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}L}$ ✗ ✓ ✓ ? ? ? Contrastive Learning (Chen et al., 2020) ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}L}$ ✗ ✗ ? ? ? ? Disentanglement (Suter et al., 2019) $$ ✗ ✓ ? ✗ ? ? #### 5.3.2 Multi-task learning Another way to improve the prior knowledge $K$ is to train a NeSy predictor to solve multiple NeSy tasks (sharing some of the same concepts) jointly (Caruana, 1997). Doing so equates to training on a single NeSy task whose prior knowledge is the conjunction of the prior knowledge of the different tasks (Marconato et al., 2023b). As per Eq. 38, doing so reduces the number of potential RSs. However, not all combinations of NeSy tasks are equally effective. To see this, imagine training a NeSy predictor to output both the sum of two MNIST digits and whether the first one is greater than the second one. Observations take the form $((\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}),(9,0))$ and $((\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}),(5,1))$ , where the second label indicates whether the inequality holds. The RS in Example 3.2, i.e., $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 4, \raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 1, \raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 3, \raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 6\}$ , is no longer valid, as it incorrectly predicts $(5,0)$ for the second example. However, an alternative shortcut exists, namely $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 1, \raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 4, \raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 3, \raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 6\}$ . A better alternative is to pair MNIST addition and multiplication. This is identical to Example 3.2, except now there are two labels, e.g., $((\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}),(9,20))$ and $((\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}),(5,6))$ . It can be verified that the only concept mapping that correctly predicts both the sum and the product is the intended one, i.e., the identity, ruling out all RSs. Naturally, multi-task learning requires gathering label annotations for all involved tasks, which can be non-trivial. Another downside is that it may not be obvious how to construct a “natural” NeSy task that guarantees the removal of all RSs. We will discuss possible solutions in Section 6.6. #### 5.3.3 Abductive weak supervision This strategy requires training the concept extractor using procedurally generated pseudo-labels obtained with logical abduction (Yang et al., 2024). ABL embodies this approach: in ABL, the model uses logical abduction to infer the most plausible concept vector ${\bm{c}}$ that logically explains the ground-truth label ${\bm{y}}$ according to the prior knowledge. In practice, ABL uses a pre-defined distance function $d({\bm{c}},{\bm{c}}^\prime)$ to select this concept vector from alternatives. Then, ABL uses this new concept vector as the learning target for the loss function. Choosing an appropriate distance function can provide ABL with concept risk reduction; specifically, when the distance metric is always zero, Yang et al. (2024) showed that the risk of learning a shortcut solution reduces compared to other approaches. The downside is that there is no guarantee that the abduction process will recover the ground-truth concepts. #### 5.3.4 Entropy maximization Perhaps the simplest unsupervised mitigation strategy is entropy maximization (Manhaeve et al., 2021a). A simple idea here is to introduce an entropy-based loss term encouraging the model to evenly distribute probability mass across all concept combinations. This can be made practical by averaging over all predictions in a batch, and maximizing the Shannon entropy $H(p({\bm{C}}))$ (Manhaeve et al., 2021a). Entropy maximization is easy to implement and computationally lightweight. However, it only encourages the predictor to spread concept predictions, but does not guarantee that the learned concepts will match the ground-truth ones. Furthermore, there are no formal results that show this method reduces the number of RSs. #### 5.3.5 Smoothing The idea behind concept smoothing is to encourage the probabilities of the most and least probable concept vectors ${\bm{c}}$ to be “close”. The main idea is to never allow NeSy predictors to learn ”peaked” concept distributions, i.e., $p({\bm{C}}\mid{\bm{X}})$ of the model is never one-hot. Smoothing can be enforced either architecturally, e.g., by introducing a temperature parameter that reduces the magnitude of the concept activations, or during training, by penalizing high activations on input samples. This strategy permits reducing the RSs risk (Eq. 30), the full derivation can be found in (Yang et al., 2024). This strategy, however, only circumvents some RSs: while smoothing prevents deterministic RSs, it does not affect non-deterministic ones. E.g., imagine that $K$ allows inferring the same label vector ${\bm{y}}$ from both ${\bm{c}}^\prime$ and ${\bm{c}}^\prime\prime$ (that is, $β^*({\bm{c}}^\prime)=β^*({\bm{c}}^\prime\prime)={\bm{y}}$ , and that a NeSy predictor predicts a mixture of the two, e.g., because it has learned the map $α({\bm{g}})=0.51≤ft\{{\bm{C}}={\bm{c}}^\prime\right\}+0.51≤ft\{{\bm{C}}={\bm{c}}^\prime\prime\right\}$ , where ${\bm{g}}∈\{{\bm{c}}^\prime,{\bm{c}}^\prime\prime\}$ . The mapping $α$ is smooth – the highest probability it can output is $0.5$ – but it is also a (non-deterministic) reasoning shortcut for PNSPs and SL, as it produces a correct prediction: $0.5β^*({\bm{c}}^\prime)+0.5β^*({\bm{c}}^\prime\prime)=1≤ft\{{\bm{Y}}={\bm{y}}\right\}$ . #### 5.3.6 Reconstruction Another option is to introduce a reconstruction penalty into the learning process (Cottrell et al., 1987). Doing so encourages learning concepts that allow reconstructing the input ${\bm{x}}$ , or part of it. This rules out RSs that conflate unrelated concepts together and thus prevent reconstruction. To see how it works, consider MNIST-Add (Example 3.2) again: NeSy predictors achieving low reconstruction loss cannot map different MNIST digits to the same concept, as doing so prevents them from faithfully reconstructing the original inputs (see Section 5.1.3). For instance, the mapping ${\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 4,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 1,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 5,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 4}$ implies that a single concept, $2$ , has to decode into two different digits, <details> <summary>figures/mnist-2.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> and <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> , leading to subpar reconstruction loss. While reconstruction can effectively rule out such RSs, it cannot prevent all of them. In fact, a mapping like ${\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}}↦ 4,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}}↦ 1,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}}↦ 3,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}}↦ 6}$ satisfies the knowledge and does not hinder reconstruction. It is easy to see that this mitigation targets the optimality condition in Eq. 38, and that as such it can decrease the number of RSs. Moreover, it comes at a cost: reconstruction requires adding a decoder module, increasing model size, computational overhead, and training complexity in practice. Crucially, for reconstruction to be effective, the learned concepts must correspond to distinct and separable visual features, as in MNIST-Add. This is not always the case. In real-world settings, e.g., BDD-OIA (Xu et al., 2020), reconstructing the input image can be very difficult, requiring additional information such as object bounding boxes and segmentation masks. E.g., reconstructing a pedestrian requires isolating it from the background, which is not possible from raw pixel input alone. This makes reconstruction impractical in NeSy tasks involving complex, real-world inputs. #### 5.3.7 Contrastive learning Another effective unsupervised strategy targeting the optimality condition is contrastive learning (Chen et al., 2020). The core idea is to encourage similar inputs to produce the same concepts. This is typically achieved by forcing augmented views of the same input (positive pairs) to predict the same concepts, while ensuring that different inputs (negative pairs) yield distinct ones. For instance, given a MNIST digit, slight rotations or shifts can be applied to create positive pairs, while the remaining images in the batch serve as negatives. This encourages the model to map together different visual appearances of the same concept (e.g., <details> <summary>figures/mnist-3.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> ) while keeping the others separate (e.g., <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> ). Although contrastive learning provides no formal guarantees for reducing RSs, in practice contrastive learning has a similar effect to reconstruction, in that it addresses those RSs that collapse semantically distinct inputs into the same concept. For instance, in Example 3.2 both strategies can prevent the model from mapping visually distinct digits such as <details> <summary>figures/mnist-3.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> and <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> to the same concept, e.g., $4$ . Contrastive learning is, however, easier to implement (e.g., using stock implementations of the infoNCE loss (Oord et al., 2018)) and significantly easier to optimize than reconstruction-based approaches. Nevertheless, we must take care when designing augmentations to ensure they preserve the semantic meaning of the underlying concepts. For example, rotating a digit such as <details> <summary>figures/mnist-6.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> in MNIST can transform it into a <details> <summary>figures/mnist-9.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> , which completely changes the semantics of the image, introducing noise (Chang et al., 2020). #### 5.3.8 Architectural disentanglement The last unsupervised strategy we consider is architectural disentanglement. While the notion of disentanglement has different meanings in the literature (Higgins et al., 2018; Suter et al., 2019), here it simply means that the concept extractor processes independent objects in the input separately. For instance, in visual recognition tasks in which multiple objects appear in the scene and each object has is associated with a separate set of concepts (e.g., shape, size and color), architectural disentanglement amounts to processing each object separately. To see why this makes sense, consider MNIST-Add again. Here, the input comprises two digits. When using a single concept extractor that processes both digits at once, there is a chance that it will swap the order of the two digits, as doing so leaves their sum unchanged. Separate processing resolves this ambiguity. More generally, this strategy prevents the model from confusing the concepts of one object with those of a different object, thus reducing the number of RSs. A nice benefit of this architectural bias is that it dramatically reduces the space of learnable concept extractors $F$ , which shrinks the space of learnable models (aka $Vert(A_F)$ in Eq. 38) and therefore the number of learnable RSs. Moreover, it enables parameter sharing across objects, reducing sample complexity. It is however not always applicable or easy to implement. On the one hand, it only makes sense for NeSy tasks in which the inputs contain multiple independent objects, such as MNIST-Add. On the other hand, it requires being able to isolate what part of the input encodes what object, which can be non-trivial depending on the complexity of the input. For instance, applying disentanglement to tasks like BDD-OIA requires using either ground-truth bounding boxes, which are seldom available, or predicted segmentations from tools such as Yolo (Redmon et al., 2016; Shindo et al., 2023) and FastRCNN (Girshick, 2015; Marconato et al., 2023b). These, however, can introduce noise in the learning process. #### 5.3.9 Which mitigation strategy is best? As shown in Table 2, different strategies offer different efficacy-cost trade-offs. The main take-away is twofold: on the one hand, supervised strategies like concept supervision Section 5.3.1 and multi-task learning Section 5.3.2 can be very effective but also expensive; on the other, unsupervised strategies are cheaper to implement but also less reliable. Specifically, while some of them (like reconstruction Section 5.3.6, abduction-based weak supervision Section 5.3.3 and smoothing Section 5.3.5) provably avoid certain kinds of RSs, they are ineffective for others. There is no one-size-fits-all strategy and the best option is application specific, in that it depends on factors like the cost of obtaining annotations and on the kind of RSs that one wishes to get rid of. One option is to mix multiple strategies, yet not all combinations are useful. For instance, high smoothness is at odds with both concept supervision and reconstruction: the former aims to spread probability mass across concepts, increasing their uncertainty and un-informativeness, the latter attempt to fit the annotations with high certainty and to inject as much information as possible into the concepts, respectively. No model can fully satisfy both objectives. Finally, combining multiple strategies can yield overly complex loss functions that are challenging to optimize in practice, e.g., requiring careful hyperparameter tuning. #### 5.3.10 Can one simply recover learned concepts? Concepts learned without supervision risk encoding incorrect, misaligned semantics. A natural workaround, that has been sometimes used in the literature (Daniele et al., 2023; Tang and Ellis, 2023), is to figure out what concepts the model has learned in a post-hoc fashion, by matching concept predictions with ground-truth concept annotations. However, this strategy may not work: it only works insofar as the concept extractor identifies (in a technical sense, see (Bortolotti et al., 2025)) the ground-truth concepts modulo a permutation of their indices and element-wise invertible transformations (Bortolotti et al., 2025). Given how difficult it is to ensure latent variables can be identified (Hyvärinen et al., 2024), this is unlikely to happen by chance. For instance, in MNIST-Add, this strategy only works if the concept extractory has learned to predict the ground-truth digits, excepts shuffled – e.g., it predicts $4$ with $8$ , but predicts them perfectly when reordered. On the flip side, this strategy cannot work whenever the concept extractor collapses together multiple ground-truth concepts, e.g., it predicts $6$ whenever it sees a <details> <summary>figures/mnist-5.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> and an <details> <summary>figures/mnist-6.png Details</summary>  ### Visual Description Icon/Small Image (48x48) </details> . ### 5.4 Awareness Strategies <details> <summary>x6.png Details</summary>  ### Visual Description ## Diagram: Comparison of Neural-Symbolic (NeSy) Approaches for Autonomous Driving Logic ### Overview The image is a technical diagram comparing two approaches to implementing a logical rule for autonomous driving. The left panel shows a real-world driving scene with annotated objects. The middle and right panels contrast a "State-of-the-Art" (SotA) Neural-Symbolic (NeSy) method with a proposed "RS-aware" (Rule-Semantics-aware) method, using entropy charts and connection diagrams to illustrate their performance and internal reasoning. ### Components/Axes The image is divided into three vertical panels separated by dashed lines. **1. Left Panel (Problem Definition):** * **Header Text:** `K = (pedestrian ∨ red ⇒ stop)` * This is a logical rule: "If there is a pedestrian OR a red light, then stop." * **Image Content:** A street scene from a vehicle's perspective. * **Annotations:** * A cyan bounding box around a traffic light on the left, labeled `green` in cyan text. * A cyan bounding box around a traffic light further ahead, labeled `red` in cyan text. * A magenta bounding box around two pedestrians crossing the street, labeled `red` in magenta text. This label likely indicates the logical condition "red" (as in red light) is being triggered by the presence of pedestrians, or it's a mislabel highlighting the conflict. **2. Middle Panel (NeSy SotA):** * **Header Text:** `NeSy SotA` * **Top Chart - "Entropy":** * A bar chart with three bars. * **Bar 1 (Left):** Short blue bar with a green checkmark (✓) above it. * **Bar 2 (Middle):** Taller blue bar with a red cross (✗) above it. * **Bar 3 (Right):** Taller blue bar (similar height to Bar 2) with a red cross (✗) above it. * **Bottom Diagram:** * Three nodes at the top labeled: `g_l` (green light), `r_l` (red light), `pe` (pedestrian). * Three colored circles at the bottom: a green circle, a red circle, and a neutral face (😐) circle. * **Connections (Arrows):** * `g_l` → green circle. * `r_l` → red circle. * `pe` → red circle. * `pe` → neutral face circle. **3. Right Panel (RS-aware):** * **Header Text:** `RS-aware` * **Top Chart - "Entropy":** * A bar chart with three bars. * **All Three Bars:** Are of equal, moderate height (shorter than the tall bars in the NeSy SotA chart). Each has a green checkmark (✓) above it. * **Bottom Diagram:** * Same three nodes at the top: `g_l`, `r_l`, `pe`. * Same three colored circles at the bottom: green, red, neutral face. * **Connections (Arrows):** * `g_l` → green circle. * `r_l` → red circle. * `r_l` → neutral face circle. * `pe` → red circle. * `pe` → neutral face circle. ### Detailed Analysis * **Logical Rule Application:** The left panel establishes the ground truth rule `K`. The scene contains a `green` light, a `red` light, and `pedestrians`. According to the rule, the presence of pedestrians (`pe`) should trigger the `stop` condition. * **NeSy SotA Performance (Middle Panel):** * **Entropy Chart:** Shows low entropy (certainty) for the first condition (likely `g_l`), but high entropy (uncertainty) for the second and third conditions (likely `r_l` and `pe`). The red crosses indicate failure or high uncertainty in correctly classifying or reasoning about the red light and pedestrian. * **Connection Diagram:** Shows a direct, one-to-one mapping for `g_l` and `r_l`. The `pe` node connects *only* to the `red` circle and the `neutral` circle. This suggests the model is uncertain whether the pedestrian should map to the "red" (stop) condition or a neutral state, failing to confidently link it to the required logical outcome. * **RS-aware Performance (Right Panel):** * **Entropy Chart:** Shows moderate, equal entropy across all three conditions, with green checkmarks indicating successful handling or correct classification. * **Connection Diagram:** Shows a more complex, cross-connected mapping. Both `r_l` and `pe` connect to *both* the `red` circle and the `neutral` circle. This indicates the model acknowledges the ambiguity or shared semantic role between a red light and a pedestrian in triggering the stop rule, distributing its reasoning across both relevant outputs. ### Key Observations 1. **Spatial Grounding:** The legend (Entropy chart symbols: ✓/✗) is placed directly above its corresponding bar. The connection diagrams are positioned directly below their respective entropy charts. 2. **Trend Verification:** The NeSy SotA entropy trend is "low, high, high". The RS-aware entropy trend is "moderate, moderate, moderate". The connection complexity increases from NeSy SotA (mostly separate lines) to RS-aware (crossed lines). 3. **Component Isolation:** The left panel defines the problem. The middle and right panels are direct comparisons of solution methods, each with a performance metric (entropy) and a model of internal reasoning (connection diagram). 4. **Anomaly/Highlight:** The magenta label `red` on the pedestrians in the left image is a critical annotation. It visually forces the connection between the object "pedestrian" and the logical condition "red" (stop), which is the core challenge the diagrams explore. ### Interpretation This diagram argues that a standard Neural-Symbolic (NeSy) approach struggles with the semantic ambiguity inherent in real-world rules. The rule `pedestrian ∨ red ⇒ stop` requires the system to understand that two different physical objects (a traffic light showing red, and a pedestrian) map to the same logical condition (`stop`). The NeSy SotA model shows high uncertainty (high entropy) when processing these ambiguous cases (`r_l` and `pe`), as its internal connections are too rigid. The proposed "RS-aware" model is designed to handle this rule semantics explicitly. Its lower, uniform entropy indicates greater confidence. The cross-connected diagram reveals its strategy: it doesn't force a one-to-one mapping. Instead, it allows both `r_l` and `pe` to influence both the `red` (stop) and `neutral` outputs, better modeling the "OR" logic where either input can trigger the same outcome. The diagram suggests that explicitly modeling the semantics of logical rules (how inputs relate to outputs) improves the robustness and certainty of AI systems in safety-critical tasks like autonomous driving. </details> Figure 10: NeSy predictors are not RS-aware (Marconato et al., 2024). Left: in the BDD-OIA autonomous driving task (Xu et al., 2020), NeSy predictors can achieve high accuracy and satisfy the prior knowledge even when they confuse pedestrians (ped) and red lights (red) (Marconato et al., 2023b). Right: An RS-aware predictor assigns high confidence to shortcut concepts, while giving low confidence to concepts not affected by RS. Another strategy for dealing with RSs is to make NeSy predictors aware of the RSs they are affected by. We say that a NeSy predictor is RS-aware if it picks a non-deterministic RS that exhibits high confidence for those predicted concepts that are not affected by RSs and low confidence for the others (Marconato et al., 2024). As customary, confidence can be assessed using the entropy of the predictive concept distribution $p({\bm{C}}\mid{\bm{x}})$ . While awareness does not reduce RSs, RS-aware models supply stakeholders with valuable insights into the quality of the learned concepts, helping them to identify and distrust predictions obtained with poorly ground concepts. To see this, consider the following example adapted from (Marconato et al., 2024): **Example 5.1** *The left NeSy predictor in Fig. 10 is not RS-aware: despite confusing pedestrians with red lights, it is highly certain about the concepts it predicts. The one on the right is equally confused, but it is also RS-aware, in that it is more uncertain about low-quality concepts (pedestrian and red lights) than about high-quality ones (green lights). This enables users to distinguish between these cases.* Moreover, meaningful uncertainty estimates are also a powerful tool for steering interactive acquisition of annotations (Section 5.5), which can substantially reduce the cost of supervised mitigation (see Section 5.3.1). RS-awareness is complementary to mitigation strategies: when all realistic mitigation strategies are exhausted, some RSs may still be present. Then RS-aware models will improve uncertainty quantification. Unfortunately, in practice, NeSy predictors are not RS-aware: they tend to be very confident even about poorly ground concepts (Marconato et al., 2024; van Krieken et al., 2025b). What sets awareness apart from sheer mitigation is that it is possible to make NeSy predictors RS-aware in a principled manner, but without concept supervision, as we will see. As mentioned in Section 5.3.4, a simple heuristic is to maximize the entropy of the predictive concept distribution. Is this always the best we can do? To answer this question, we will need to define RS-awareness. #### 5.4.1 RS-awareness as mixtures of deterministic RSs RS awareness can be understood as mixing over the set of deterministic RSs that the NeSy predictor is affected by (van Krieken et al., 2025a). We denote this set as $Vert(A)_RS⊆Vert(A)$ , and it can be seen as the set of “irreducible” deterministic RSs that are still present after exhausting the feasible mitigation strategies. Formally, we consider RS-aware NeSy predictors as distributions that “mix” over the concept remapping distributions (Eq. 20) induced by each irreducible deterministic RS ${\bm{a}}∈Vert(A)_RS$ . We achieve this using the convex combination of deterministic RSs from Eq. 22, where each weight $λ_{\bm{a}}>0$ . The convex combination ensures each RS contributes to the final concept remapping distribution. The resulting distribution is itself a (nondeterministic) RS (Marconato et al., 2024, Lemma 1), and hence has perfect label accuracy. #### 5.4.2 Building RS-aware NeSy predictors How do we achieve such nondeterministic RSs in NeSy predictors? This usually requires that the space of learnable concept extractors $F$ (Section 4.2.1) is powerful enough to model dependencies between different concepts (van Krieken et al., 2025a). However, many implementations of NeSy predictors use concept extractors that assume (when conditioned on the input ${\bm{x}}$ ) the different concepts $C_1,…,C_k$ are independent, i.e., $p(C_1,…,C_k\mid{\bm{x}})=∏_i=1^kp(C_i\mid{\bm{x}})$ (van Krieken et al., 2024a). NeSy predictors exploit the independence assumption for efficient inference (Darwiche and Marquis, 2002; van Krieken et al., 2023; Choi et al., 2025; De Smet et al., 2023a). However, it also greatly restricts the space of learnable concept extractors $F$ . **Example 5.2 (Example in(van Krieken et al.,2025a))** *We consider the MNIST-XOR task where the NeSy predictor should predict the output of the XOR function applied to two MNIST digit. The XOR function returns 1 if the two input digits are different, and 0 otherwise. E.g., ${\bm{x}}=\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-0.png}}$ gives $y=1$ , while ${\bm{x}}=\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}}$ gives $y=0$ . This problem contains the shortcut $\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}}↦ 0,\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-0.png}}↦ 1$ (under disentanglement, Section 5.3.8). Given input ${\bm{x}}=\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}}\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-0.png}}$ , an RS-aware NeSy predictor should assign 0.5 probability to both the ground-truth concept vector $(1,0)$ and the RS concept vector $(0,1)$ . If we would try to model this distribution with the independence assumption, we find that $p(C_1=1\mid{\bm{x}})=p(C_2=1\mid{\bm{x}})=0.5$ . But then $p(C_1=1,C_2=1\mid{\bm{x}})=p(C_1=1\mid{\bm{x}})· p(C_2=1\mid{\bm{x}})=0.25$ , i.e., it assigns probability to a concept vector that does not give the correct output label. Hence, the resulting distribution is not an RS, and might make incorrect label predictions.* In general, van Krieken et al. (2025a) proves that the independence assumption prevents models from being aware of RSs for most inference layers. Motivated by these results, we discuss two recent methods that go beyond the independence assumption to build RS-aware NeSy predictors. <details> <summary>x7.png Details</summary>  ### Visual Description ## Diagram: Weighted Process Combination with Entropy States ### Overview The image is a technical diagram illustrating a mathematical or logical combination of two processes, each weighted at 0.5, resulting in a final combined state. The diagram uses symbolic representations for states, pathways, and an "Entropy" metric to show a transformation from problematic initial states to a resolved final state. ### Components/Axes The diagram is structured as an equation: `[Process A] + [Process B] = [Combined Result]`. **1. Left Input (Process A):** * **Top Label:** "Entropy" * **Entropy State:** A green checkmark (✓) above the "green" column, and a red cross (✗) above both the "red" and "ped" columns. * **Main Box:** Contains three columns labeled "green", "red", and "ped". * **green:** A straight upward arrow (↑) originating from a solid green circle (●). * **red:** A straight upward arrow (↑) originating from a solid red circle (●). * **ped:** A curved arrow that starts straight up from a smiley face (☺) and then bends to the left, pointing towards the "red" column. * **Weight:** The number "0.5" is positioned to the left of this entire block. **2. Right Input (Process B):** * **Top Label:** "Entropy" * **Entropy State:** Identical to Process A: a green checkmark (✓) above "green", and red crosses (✗) above "red" and "ped". * **Main Box:** Contains three columns labeled "green", "red", and "ped". * **green:** A straight upward arrow (↑) originating from a solid green circle (●). * **red:** A curved arrow that starts straight up from a solid red circle (●) and then bends to the right, pointing towards the "ped" column. * **ped:** A straight upward arrow (↑) originating from a smiley face (☺). * **Weight:** The number "0.5" is positioned to the left of this block, preceded by a plus sign (+). **3. Combined Result:** * **Top Label:** "Entropy" * **Entropy State:** A green checkmark (✓) above the "green" column, and blue checkmarks (✓) above both the "red" and "ped" columns. * **Main Box:** Contains three columns labeled "green", "red", and "ped". * **green:** A straight upward arrow (↑) originating from a solid green circle (●). * **red:** An upward arrow (↑) originating from a solid red circle (●) that crosses over the arrow from "ped". * **ped:** An upward arrow (↑) originating from a smiley face (☺) that crosses under the arrow from "red". * **Equation:** An equals sign (=) precedes this block. ### Detailed Analysis The diagram depicts a weighted summation. Two processes, each contributing a weight of 0.5, are added together. * **Process A (0.5):** Characterized by a stable "green" state, an unstable "red" state, and a "ped" state whose pathway is influenced by or diverts to the "red" state. Its entropy profile shows one success (green) and two failures (red, ped). * **Process B (0.5):** Also has a stable "green" state. Its "red" state is unstable and influences the "ped" state, while the "ped" state itself is stable. Its entropy profile is identical to Process A's. * **Combined Result (1.0):** The summation produces a new state. The "green" pathway remains stable and unchanged. The key transformation occurs in the "red" and "ped" pathways: their arrows now cross, indicating an interaction, exchange, or integration that was not present in the individual processes. Correspondingly, the entropy state for "red" and "ped" changes from red crosses (failure/instability) to blue checkmarks (success/stability). ### Key Observations 1. **Symmetry and Asymmetry:** The two input processes are not identical but are mirror-like in their instability patterns (A's "ped" influences "red"; B's "red" influences "ped"). 2. **Entropy Transformation:** The most significant change is in the "Entropy" labels. The combination of two processes, each with two failing components, results in a process where all components are marked as successful (green or blue checks). 3. **Pathway Integration:** The crossing arrows in the result are the visual mechanism explaining the entropy improvement. The previously separate, unstable pathways for "red" and "ped" become intertwined, suggesting that their interaction resolves their individual instabilities. 4. **Color Coding:** Colors are used consistently: green for stable/success, red for unstable/failure, blue for a new state of success in the combination, and black for pathways and labels. ### Interpretation This diagram likely models a concept from systems theory, control theory, or machine learning (e.g., ensemble methods, sensor fusion, or decision integration). It demonstrates the principle that **combining two imperfect or partially failing processes can yield a superior, fully stable system**. The "Entropy" label suggests a measure of disorder, error, or uncertainty. The individual processes have high entropy (two red crosses each). The combination, through the interaction of their previously problematic components (the crossing of "red" and "ped" pathways), achieves a state of low entropy (all checks). This is a visual metaphor for synergy, where the whole becomes greater than the sum of its parts. The 0.5 weights imply an equal contribution from each sub-process to the final outcome. The diagram argues that the value of integration isn't just additive; it's transformative, creating new, stable interactions that didn't exist in the components alone. </details> Figure 11: Intuition behind bears. bears (Marconato et al., 2024) constructs ensembles of concept predictors that employ different sets of concepts to solve the task. Concepts that need to be learned correctly have to match across models, while those that cannot will have different groundings. As a result, averaging their predictions increases uncertainty only where disagreement exists. #### 5.4.3 RS-Awareness via Ensembles bears (Marconato et al., 2024) is a model-agnostic technique for making NeSy predictors RS-aware. bears replaces the concept extractor with an ensemble of concept extractors, each using the independence assumption. bears constructs this ensemble specifically such that each member captures a different deterministic RS in $Vert(A)_RS$ . This construction has ties with Bayesian deep learning techniques (Wang and Yeung, 2020; Daxberger et al., 2021; Osawa et al., 2019; Lakshminarayanan et al., 2017; Gal and Ghahramani, 2016) but it is also backed by a separate theoretical setup (Marconato et al., 2024). For instance, in BDD-OIA (Example 3.3) one extractor might conflate pedestrians with red lights, while another might do the opposite, as in Fig. 11 (left). bears computes the predictive concept distribution by averaging over all extractors in the ensemble. Intuitively, this works for two reasons. First, if all extractors have high label accuracy, their average will as well (Eq. 22). Second, the concept entropy of the ensemble hinges on the semantics associated by the different extractors to the concepts themselves. Then, the ensemble has high entropy for concepts are affected by RSs – like pedestrian and red light – as different extractors each learn different semantics. For unaffected concepts – like green light – the extractors agree on their semantics and the ensemble has low entropy. This intuition is illustrated in Fig. 11 (right). In practice, bears extends deep ensembles (Lakshminarayanan et al., 2017) with a loss term that encourages ensemble members to model different RSs, and an entropy term (reminiscent of Section 5.3.4) that spreads concept predictions. Compared to other Bayesian deep learning methods, such as MC dropout (Gal and Ghahramani, 2016) and the Laplace approximation (Daxberger et al., 2021), deep ensembles allow learning highly multi-modal distributions, making them an excellent choice for learning multiple, diverse RSs. This is supported by experiments, which show that bears significantly improves RS-awareness of NeSy architectures on different tasks and that it does so better than other Bayesian alternatives (Marconato et al., 2024). One downside of bears is that it requires learning an ensemble of concept extractors. Fortunately, small ensembles of $5$ to $10$ members were shown to be sufficient in practice. #### 5.4.4 RS-Awareness via Diffusion Motivated by going beyond the independence assumption, NeSyDM (van Krieken et al., 2025b) uses expressive discrete diffusion models as concept extractors. Unlike bears, it trains a single concept extractor $p({\bm{C}}\mid{\bm{x}})$ via masked diffusion models (MDMs) (Austin et al., 2021; Sahoo et al., 2024). MDMs start with a concept vector that consists entirely of masks. Masks essentially indicate we do not yet know what the value of the concept should be. Then, MDMs are tasked with iteratively unmasking the various concepts. The main motivation for using MDMs to go beyond independence is scalability: each unmasking step can be seen as a concept extractor with the independence assumption. Hence, NeSyDM can directly reuse existing inference methods that exploit the independence assumption. NeSyDM consists of three loss components. The first loss aims to unmask partially masked concept, the second unmasks partially masked labels, and the final loss encourages high entropy among the concepts. One benefit compared to bears is that NeSyDM ’s only train a single model, preventing the need to choose the ensemble size. Experimentally, NeSyDM also matches or improves on calibration and out-of-distribution concept accuracy compared to bears. ### 5.5 Awareness Helps Mitigation To support mitigation strategies, specifically concept supervision (discussed in Section 5.3.1), one can leverage human input to request targeted annotations. This idea is well established in the field of Interactive Machine Learning (Ware et al., 2001; Fails and Olsen, 2003; Amershi et al., 2014; Teso and Kersting, 2019). In the context of RSs, the most relevant method explored so far is Active Learning (Settles, 2012), which we briefly describe below. Active Learning Rather than relying on costly full supervision, Active Learning (Settles, 2012) allows a model to interactively query a human (or oracle) for labels on the most informative or uncertain examples. The core idea is to focus annotation efforts where they are most impactful, thereby improving model performance while minimizing human effort. As demonstrated in (Marconato et al., 2024), uncertainty estimates over concepts – provided by bears (see Section 5.4) or NeSyDM (van Krieken et al., 2025b) (see Section 5.4.4) – can be used to identify and query the most uncertain concepts, effectively addressing RSs with fewer annotations on concepts. This targeted strategy outperforms traditional active learning methods, such as querying concept uncertainty in PNSPs, where the uncertainty signal is often uninformative (Marconato et al., 2024). ## 6 Extensions and Open Problems In this section, we discuss RSs in contexts beyond those studied in this paper, and outline potential extensions and open research problems. ### 6.1 Reasoning Shortcuts in NeSy AI Beyond Predictors RSs stem from the intrinsic ambiguity of logic inference when the prior knowledge $K$ admits inferring the ground-truth label from unintended concepts. This suggests that RSs arise regardless of how the reasoning step is implemented, that is, also for NeSy architectures beyond the four predictors we considered here. Models like DPL and LTN are full-fledged first-order programming languages, i.e., they can handle first-order logic specifications and work with a variable number of logic entities and concepts. Given that RSs affect them in the propositional case and that FOL is a strict generalization of propositional logic, we can safely assume that RSs exist also in the FOL case. E.g., RSs plausibly affect extensions of these models, such as (Huang et al., 2021a; Winters et al., 2022; De Smet et al., 2023b; Badreddine et al., 2023), as well as predictors that combine probabilistic and fuzzy semantics, like NeuPSL (Pryor et al., 2022), models that embed symbols into continuous vector spaces, such as neural theorem provers with fuzzy semantics (Rocktäschel and Riedel, 2016) and probabilistic semantics (Maene and Raedt, 2023), models that implement hard constraints with fuzzy logic, such as (Giunchiglia and Lukasiewicz, 2020; Giunchiglia et al., 2024), and diffusion-based models (van Krieken et al., 2025b). Moreover, RSs have also been identified in SatNet (Wang et al., 2019; Simard et al., 1991), an architecture that performs perception and reasoning entirely in embedding space. This leads us to postulate that—barring implicit mitigation due to the use of reconstruction penalties, see Section 5.3.6 —RSs might also affect generative NeSy models, such as (Misino et al., 2022; Ferber et al., 2024). While it is likely that RSs affect a broad class of NeSy architectures, more work is needed to extend the existing theoretical frameworks and mitigation strategies to these models. ### 6.2 Reasoning Shortcuts in Concept-based Models So far, we only considered NeSy predictors where the prior knowledge $K$ , and therefore the inference layer, is supplied externally and fixed. Recently, RSs have also been studied in two related families of models whose inference layer is learned (Bortolotti et al., 2025): i) NeSy modular predictors that acquire the knowledge from data, and ii) Concept-based Models (CBMs), which pair a concept extractor with an interpretable (typically linear) inference layer (Koh et al., 2020; Espinosa Zarlenga et al., 2022; Marconato et al., 2022; Schwalbe, 2022; Poeta et al., 2023). It was shown that, without the aid of concept supervision, these models suffer from joint reasoning shortcuts (JRSs): failure modes involving extracting wrong concepts and/or wrong logical assignments that lead to correct label predictions (see Example 6.1 for an example). **Example 6.1 (MNIST-SumParity)** *To provide an example, imagine a variant of MNIST-Add where the goal is to predict whether the sum of two digits is odd or even, that is, $Y=(C_1+C_2)\mod 2$ . Suppose that the training set includes only examples for which $C_2$ is odd. A possible JRS amounts to mapping all even digits to $0$ , i.e., $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-0.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-2.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-4.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-6.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-8.png}}\}↦ 0$ , and all odd digits to $1$ , i.e., $\{\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-1.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-3.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-5.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-7.png}},\raisebox{-1.0pt}{∈cludegraphics[width=7.96527pt]{figures/mnist-9.png}}\}↦ 1$ . Consistently, the learned knowledge is the difference among the predicted digits $\hat{Y}=\hat{C}_1-\hat{C}_2$ , as it would return the very same labels as using $Y=(C_1+C_2)\mod 2$ . In contrast to regular RSs, here both the concepts and the learned knowledge are “wrong”, with the result that in the OOD setting where $C_2$ includes even digits, the learned model can predict $\hat{Y}=-1$ (see Fig. 12 for a visual representation).* All RSs can also be viewed as JRSs, but the opposite is not true: the set of learnable JRSs can vastly outnumber that of regular RSs (Bortolotti et al., 2025). For example, permuting concept values does not alter the concept semantics and the learned rules, leading to equivalent solutions upon renaming the extracted concepts. Specifically, since concepts are anonymous during training, they can be easily aligned to ground-truth concepts at test time by leveraging annotations. This, however, is not possible if a joint-RS is learned by the model (Bortolotti et al., 2025). Counting RSs can be adapted to joint-RSs, and an equivalent result to Theorem 4.12 also holds when $K$ is learned, showing that dealing with deterministic joint-RSs is, in many cases, sufficient to rule out all joint-RSs. However, the benefits of many mitigation strategies for RSs discussed in Section 5.3 do not transfer to joint-RSs. Supervised strategies, like concept supervision and multi-tasking learning, offer some improvement but are insufficient to address all joint-RSs. In particular, concept supervision can lead to high concept accuracy, yet it does not necessarily return the correct knowledge. It is unknown whether more powerful strategies for combating JRSs exist, and a theoretical characterization of JRSs from a statistical learning perspective is currently missing. In the context of RSs, investigating the scenario where learned concept vectors ( ${\bm{c}}∈C$ ) and ground-truth concept vectors ( ${\bm{g}}∈G$ ) belong to different spaces, i.e., $C≠G$ , is open and so far not explored. Considering two different spaces $C$ and $G$ results in having the NeSy predictor inference layer $β$ structurally different from $β^*$ , because of the different mapping domain. This setting naturally draws a connection between RSs and JRSs, where in the latter a concept-based model may learn to use fewer concepts in the bottleneck and fine-tune a $β≠β^*$ that still matches label predictions. In this sense, because NeSy predictors $(f,β)$ have $β≠β^*$ , one has to check if learned concepts and the inference layer possess (a variation of) the intended semantics (Bortolotti et al., 2025, Definition 3.3) We expect that, in this scenario, RSs can likely increase due to mismatch between the NeSy predictor’s and the ground-truth knowledge. <details> <summary>x8.png Details</summary>  ### Visual Description \n ## Technical Diagram: Neural Network Parity Operations ### Overview The image displays a technical diagram consisting of three horizontally arranged, similar computational blocks. Each block illustrates a different configuration for computing an output value `y` from inputs `c₁` and `c₂`, involving a neural network component (NN*) and a parameter (β or β*). The diagram appears to compare different model architectures or parameter settings for a parity-related task. ### Components/Axes The diagram is segmented into three distinct, side-by-side components. Each component contains the following elements in sequence from left to right: 1. **Input Box**: A black square containing two white numbers stacked vertically. 2. **Neural Network Block**: A blue trapezoid labeled "NN*". 3. **Parameter Block**: A colored trapezoid (red or purple) labeled "β*" or "β". 4. **Condition Label**: The word "odd" placed to the right of the parameter block. 5. **Output Equation**: A mathematical equation defining `y` placed below the blocks. ### Detailed Analysis **Component 1 (Left):** * **Input Box**: Contains the numbers `7` (top) and `2` (bottom). * **Neural Network Block**: Blue trapezoid labeled `NN*`. Adjacent to it is a bracket containing the vector `[7, 2]`. * **Parameter Block**: Red trapezoid labeled `β*`. * **Condition Label**: `odd`. * **Output Equation**: `y = (c₁ + c₂) mod 2`. **Component 2 (Center):** * **Input Box**: Contains the numbers `7` (top) and `2` (bottom). * **Neural Network Block**: Blue trapezoid labeled `NN*`. Adjacent to it is a bracket containing the vector `[1, 0]`. * **Parameter Block**: Red trapezoid labeled `β*`. * **Condition Label**: `odd`. * **Output Equation**: `y = (c₁ + c₂) mod 2`. **Component 3 (Right):** * **Input Box**: Contains the numbers `7` (top) and `2` (bottom). * **Neural Network Block**: Blue trapezoid labeled `NN*`. Adjacent to it is a bracket containing the vector `[1, 0]`. * **Parameter Block**: Purple trapezoid labeled `β` (note: no asterisk). * **Condition Label**: `odd`. * **Output Equation**: `y = (c₁ - c₂)`. ### Key Observations 1. **Constant Input**: All three components share the same input values (`7` and `2`) in the black box. 2. **Neural Network Consistency**: The `NN*` block is visually identical (blue trapezoid) across all components. 3. **Parameter Variation**: The parameter block changes in two ways: * **Label**: It is `β*` (with an asterisk) in the first two components and `β` (without an asterisk) in the third. * **Color**: It is red in the first two components and purple in the third. 4. **Vector Variation**: The vector adjacent to `NN*` changes from `[7, 2]` in the first component to `[1, 0]` in the second and third components. 5. **Equation Variation**: The output equation is `y = (c₁ + c₂) mod 2` for the first two components and `y = (c₁ - c₂)` for the third. 6. **Spatial Grounding**: The legend/label (`NN*`, `β*`, `β`, `odd`) is placed directly on or adjacent to its corresponding graphical element. The equation is consistently placed below its respective block. ### Interpretation This diagram likely illustrates a comparative analysis of different methods or models for solving a problem related to parity (odd/even) or binary classification, given the `mod 2` operation and the "odd" label. * **Components & Flow**: Each block represents a computational pipeline: Inputs (7,2) → Neural Network (NN*) → Parameter (β/β*) → Output (y). The "odd" label may indicate the target property or a condition the output must satisfy. * **Relationships & Anomalies**: * The first two components are identical except for the vector `[7, 2]` vs. `[1, 0]`. This suggests a comparison between using the raw input values versus a transformed or one-hot encoded representation (`[1, 0]`) within the same `NN*` and `β*` framework. * The third component introduces two changes: the parameter is now `β` (not `β*`) and the operation is subtraction instead of modular addition. The color change (red to purple) visually reinforces this as a distinct method. This could represent a baseline or alternative model without the asterisk-marked (possibly "trained" or "specialized") components. * **Underlying Meaning**: The diagram is investigating how different internal representations (the vector) and different final operations (addition mod 2 vs. subtraction) affect the computation of `y`, potentially in the context of ensuring the result is "odd". The consistent input (7,2) allows for a direct comparison of the outputs from these three different configurations. The use of `c₁` and `c₂` in the equations, rather than the explicit numbers 7 and 2, indicates these are general formulas where `c₁` and `c₂` are variables derived from the input. </details> Figure 12: Example of Joint Reasoning Shortcuts. The task is to predict whether the sum of two MNIST digits is odd, using training data that cover all (even, even), (odd, odd), and (odd, even) pairs. Red denotes fixed knowledge, purple denotes learned knowledge. In all the presented cases, the models achieve optimal performance. Left: ground-truth concepts with the correct inference layer. Middle: NeSy model with prior knowledge may still learn shortcuts by assigning unintended semantics to concepts. Right: CBMs, both concepts and the inference layer are misaligned, resulting in joint reasoning shortcuts. Note: In both of the previous cases, the OOD setting (even, odd) produces a consistent prediction, whereas the joint reasoning shortcut yields $-1$ as the predicted value. ### 6.3 Foundation and Large Language Models An interesting direction for future research is to investigate the connection between RSs and foundation models (Bommasani et al., 2021; Radford et al., 2021). Several recent approaches already leverage information elicited from foundation models to prime NeSy (Stammer et al., 2024) and concept-based architectures (Oikarinen et al., 2022; Yang et al., 2023b; Rao et al., 2024; Srivastava et al., 2024; Yuksekgonul et al., 2023), e.g., using pre-trained representations or weak concept annotations obtained by prompting visual-language models. Clearly, the benefit is that foundation models are trained on vast amounts of data, which allow covering a large support of (potentially new) inputs, and are multimodal — allowing to use concept textual description to estimate the concepts in the data. Moreover, we expect that pre-training implicitly implements some of the RS mitigation strategies we outlined: concept supervision (Section 5.3.1) and multi-task learning (Section 5.3.2). In fact, it is quite likely that some elementary concepts, like the color or shape of objects, need to be separated in model representations to allow them to match a huge amount of image-caption pairs (e.g., from the Web Image Text dataset (Srinivasan et al., 2021)). For these reasons, it is plausible that, depending on the domain, foundation models may produce useful, high-quality concepts. Empirical evidence, however, calls for caution. Recent findings (Debole et al., 2025; Wüst et al., 2025) in the context of VLM-augmented CBMs indicate that foundation models can fail to identify some basic visual concepts. In addition, they may fall short due to issues such as hallucinations (Huang et al., 2024), shortcuts (Yuan et al., 2024), and limited logical (Calanzone et al., 2025) and conceptual consistency (Sahu et al., 2022). In a direction closely related to RSs, Stein et al. (2025) have shown that foundation models are affected by symbol hallucinations, a phenomenon where models prompted to generate concepts and executable programs to solve a task end up hallucinating poorly ground symbols that still manage to achieve high accuracy. Intuitively, symbol hallucinations can be thought as the analog of RSs for prompting (although, they have not been characterized so far). An important direction is to prove whether foundation models themselves are affected by (joint) RSs proper, but we expect that the current theory need to be extended to also treat models that cannot be described as NeSy predictors. Attempts to provide identifiability guarantees for concept learning in model representations (among which those of foundation models) have recently surfaced (Zheng et al., 2025; Liu et al., 2025; Rajendran et al., 2024), but it remains to be understood how this relates to shortcuts when these guarantees are not given. Reasoning shortcuts have also been studied in language modelling, though with a different extent from the one of this paper. In question answering, multi-hop reasoning is the model’s ability to answer a question by combining information from multiple pieces of evidence rather than relying on a single fact (Yang et al., 2018). As an example of two-hop reasoning, we would expect that to answer the question “In which city of the Orange Free State was the author of The Lord of the Rings born?”, the model first has to identify J. R. R. Tolkien as the author of The Lord of the Rings, and second has to determine that Tolkien was born in Bloemfontein, which is in the Orange Free State. Jiang and Bansal (2019) found that when models are trained on certain multi-hop QA datasets, such as HotpotQA (Yang et al., 2018), they often exploit shortcuts that allow them to return correct answers by matching keywords or surface patterns in the question and context, without performing the intended multi-hop reasoning steps. In this context, the authors refer to it as a reasoning shortcut for the model. For instance, in the example above, an incorrect reasoning step consists of predicting Bloemfontein just because of the presence of ”the Orange Free State”. Although these unintended solutions differ from the reasoning shortcuts discussed in our manuscript, they can be related to joint reasoning shortcuts, whereby correct concept assignments are given but the learned knowledge fails to capture the ground-truth. Making this link concrete is open and is an important research direction. ### 6.4 Reasoning Shortcuts in Reinforcement Learning NeSy reinforcement learning (RL) (Acharya et al., 2024) is a new field in which logical knowledge is integrated into the interactive reinforcement learning workflow for various purposes, including guiding exploration (Anderson et al., 2020), ensuring safe trajectories (Yang et al., 2023c), improving explainability (Baugh et al., 2025; Deane and Ray, 2025), performance (Umili et al., 2024b; Mitchener et al., 2022), or generalization to new environments (Garnelo et al., 2016; Badreddine and Spranger, 2019). Most of the works consider environments with symbolic observations, where grounding symbols or concepts is not required. A smaller set, instead, tackles more realistic environments by discretizing or clustering the neural features extracted from high-dimensional inputs (Umili et al., 2021; Garnelo et al., 2016; Hafner et al., 2021). In such cases, the extracted concepts lack a predefined meaning, potentially limiting the human interpretability of the agent’s behavior. Only a few works have investigated embedding prior symbolic knowledge into RL policy learning (Badreddine and Spranger, 2019; Umili et al., 2024b; Amador and Gierasimczuk, 2025). While reasoning shortcuts are still relatively underexplored in NeSy RL, first studies in this direction have already proven valuable for: (i) certifying the quality of learned concepts and its reusability in single-task vs. multitask settings (Umili et al., 2024b), and (ii) improving the explainability of NeSy-RL agents when knowledge is learned rather than injected (Umili and Capobianco, 2025). Badreddine and Spranger (2019) and Amador and Gierasimczuk (2025) focus on solving the RL environment introduced in (Garnelo et al., 2016) and shown in Fig. 13 (Left). They both employ a NeSy shape recognizer exploiting knowledge encoded in LTN. In these works, RSs do not appear because they rely on mitigation strategies specifically tailored to that particular environment, such as the use of pretrained shape detectors in (Badreddine and Spranger, 2019), the additional knowledge of the environment model in (Amador and Gierasimczuk, 2025), and patch discretization in both works. | <details> <summary>figures/plus_env_2.png Details</summary>  ### Visual Description ## Diagram: Image Processing Pipeline ### Overview The image is a technical diagram illustrating a three-stage pipeline for processing visual information. It shows the transformation of a raw input image containing multiple geometric shapes into a set of discrete object maps, each representing a specific category of shape. The flow is linear, moving from left to right. ### Components/Axes The diagram is divided into three distinct sections, connected by right-pointing arrows indicating the process flow. 1. **Left Section: "raw image input"** * **Label:** "raw image input" (text positioned above the graphic). * **Graphic:** A square frame containing several black, pixelated shapes on a white background. * **Shapes & Positions:** * A plus sign (`+`) in the upper-center area. * A square (`□`) in the left-center area. * An 'X' (`×`) in the bottom-left corner. * A circle (`○`) in the bottom-center area. * Another 'X' (`×`) to the right of the circle, slightly overlapping it. 2. **Middle Section: "patch discretization"** * **Label:** "patch discretization" (text positioned above the graphic). * **Graphic:** The same set of shapes from the raw input, now overlaid with a 5x5 grid of thin gray lines. The shapes are aligned to the grid cells. * **Spatial Relationship:** An arrow points from the "raw image input" graphic to this section. 3. **Right Section: "object maps"** * **Label:** "object maps" (text positioned above the graphic). * **Graphic:** Four rectangular frames stacked with a slight offset, creating a layered, 3D effect. Each frame is labeled with a category name in its top-left corner. * **Labels & Order (from back to front):** * "square" (back-most layer) * "circle" * "cross" * "agent" (front-most layer) * **Spatial Relationship:** An arrow points from the "patch discretization" graphic to this section. ### Detailed Analysis The diagram depicts a specific computational process: 1. **Stage 1 - Raw Input:** The process begins with a continuous, pixel-based image containing multiple objects of interest (square, circle, crosses, plus sign). The objects are not separated. 2. **Stage 2 - Discretization:** The image is divided into a regular grid of patches (a 5x5 grid is shown). This step likely involves analyzing or representing the image content within each discrete patch rather than at the individual pixel level. 3. **Stage 3 - Object Mapping:** The final output is a set of separate "maps," one for each object category. The stacking implies these are parallel output channels. Notably, the "plus" sign from the input does not have a corresponding map in the output stack, while an "agent" map appears, which was not an explicit shape in the input. This suggests the "agent" may be a composite or inferred entity, or the diagram is illustrative and not exhaustive. ### Key Observations * **Process Flow:** The pipeline moves from unstructured pixel data to structured, categorical representations. * **Grid Alignment:** The "patch discretization" step imposes a rigid spatial structure on the previously continuous shapes. * **Category Abstraction:** The final "object maps" abstract away pixel data entirely, representing only the presence/location of object classes. * **Discrepancy:** The "plus" sign (`+`) present in the input stages does not have a corresponding labeled map in the output stack. Conversely, the "agent" map in the output has no direct, single-shape counterpart in the input. * **Visual Style:** All graphics are simple, black-and-white line drawings, emphasizing conceptual clarity over visual detail. ### Interpretation This diagram illustrates a fundamental concept in computer vision and robotic perception: the transformation of sensory input (a raw image) into a symbolic, object-centric representation suitable for reasoning and decision-making. * **What it demonstrates:** The pipeline shows how a system might move from "seeing" pixels to "understanding" scenes. The discretization step is a common technique to reduce computational complexity and create a structured feature space. The final object maps represent a semantic segmentation or object detection output, where the system has classified regions of the image into predefined categories. * **Relationships:** The arrows define a strict, sequential dependency. The quality and structure of the "object maps" are directly dependent on the "patch discretization" method, which in turn operates on the "raw image input." * **Notable Anomalies:** The absence of a "plus" map and the presence of an "agent" map are the most significant details. This implies one of two things: 1. The diagram is a simplified example, and the "plus" was omitted from the output for brevity, while "agent" might represent a higher-level concept (e.g., the entity controlling the shapes). 2. The system is designed to detect specific classes ("square," "circle," "cross," "agent") and ignore others ("plus"), or to infer composite entities ("agent") from the configuration of primitive shapes. * **Underlying Principle:** The core idea is **abstraction**. The system discards irrelevant pixel-level details (exact shape edges, noise) and retains only the information necessary for a task: what objects are present and where they are. This is a Peircean process of moving from a raw *icon* (the pixel image) to an *index* (the grid-aligned patches) to a *symbol* (the categorical object maps). </details> | | | <details> <summary>figures/reward_machine_trajectories_2.png Details</summary>  ### Visual Description ## State Transition Diagram: Game State Machine with Item-Based Transitions ### Overview The image displays a directed graph representing a finite state machine, likely modeling a game or puzzle scenario. The system consists of six states (nodes 0-5) connected by transitions (edges) labeled with item conditions or actions. Each state has an associated numerical value (in red), possibly representing a reward or cost. The diagram uses color-coding (blue, green, black) to highlight specific transition paths. ### Components/Axes **Nodes (States):** - **Node 0**: Oval, labeled "0", red value "-2". - **Node 1**: Oval, labeled "1", red value "-1". - **Node 2**: Oval, labeled "2", red value "-4". Has a self-loop labeled "True". - **Node 3**: Oval, labeled "3", red value "-2". - **Node 4**: Double-bordered oval, labeled "4", red value "0". Has a self-loop labeled "empty|door|pickaxe|gem|lava". - **Node 5**: Oval, labeled "5", red value "-3". Has an unlabeled incoming arrow from above. **Transitions (Edges):** Edges are directed arrows with text labels. Some are colored: - **Blue edges**: Form a path from Node 5 → Node 0 → Node 1 → Node 4. - **Green edges**: Form a path from Node 5 → Node 3 → Node 1 → Node 4. - **Black edges**: Represent alternative or default transitions. **Edge Labels (Transitions):** Labels use a pipe `|` notation, likely indicating possible items held or conditions met (e.g., "empty|door" means the agent has either nothing or a door). Common terms: `empty`, `door`, `pickaxe`, `gem`, `lava`. One label contains a typo: "picaxe" (likely intended as "pickaxe"). ### Detailed Analysis **State Values (Red Numbers):** - Node 5: -3 - Node 0: -2 - Node 3: -2 - Node 1: -1 - Node 4: 0 (highest value, likely goal state) - Node 2: -4 (lowest value, likely failure state) **Transition Paths & Labels:** 1. **From Node 5 (Start):** - Blue edge to Node 0: label "pickaxe" - Green edge to Node 3: label "gem" - Black edge to Node 3: label "empty|door|gem" - Black edge to Node 2: label "lava" 2. **From Node 0:** - Blue edge to Node 1: label "gem" - Black edge to Node 4: label "lava" - Black edge to Node 2: label "lava" 3. **From Node 3:** - Green edge to Node 1: label "picaxe" (typo for "pickaxe") - Black edge to Node 1: label "empty|pickaxe|gem" - Black edge to Node 2: label "lava" 4. **From Node 1:** - Blue edge to Node 4: label "door" - Green edge to Node 4: label "door" - Black edge to Node 2: label "lava" 5. **From Node 4 (Goal):** - Self-loop: label "empty|door|pickaxe|gem|lava" - Incoming black edge from Node 2: label "lava" 6. **From Node 2 (Failure):** - Self-loop: label "True" - Incoming black edges from Nodes 0, 1, 3, 5: all labeled "lava" **Spatial Grounding:** - The legend (color coding) is implicit in the edge colors. Blue and green paths are visually distinct. - Node 4 (goal) is positioned bottom-left. Node 2 (failure) is bottom-right. Node 5 (start) is top-center. - The "lava" transitions consistently lead to Node 2 from all other nodes except Node 4. ### Key Observations 1. **Two Primary Successful Paths:** The blue and green paths represent two distinct sequences of item acquisitions leading from the start (Node 5) to the goal (Node 4). - **Blue Path:** Requires `pickaxe` (at Node 5) then `gem` (at Node 0) then `door` (at Node 1). - **Green Path:** Requires `gem` (at Node 5) then `pickaxe` (at Node 3) then `door` (at Node 1). 2. **"Lava" is a Universal Failure Condition:** Any transition labeled "lava" from Nodes 0, 1, 3, or 5 leads directly to the lowest-value state (Node 2, value -4). 3. **Goal State is Stable:** Node 4 has a self-loop accepting all item combinations, indicating it is a terminal, absorbing state. 4. **Node 2 is a Trap State:** It has a self-loop labeled "True", meaning once entered, the system cannot leave. 5. **Item Dependency:** The labels suggest the agent must possess specific items (`pickaxe`, `gem`, `door`) to progress along the colored paths. The `empty` condition appears in alternative (black) transitions. ### Interpretation This diagram models a **resource-management puzzle or adventure game**. The agent starts at Node 5 with a penalty (-3) and must collect items in a specific order to reach the goal (Node 4, reward 0) while avoiding "lava" traps that lead to a severe penalty (Node 2, -4). - **The data suggests** that there are multiple valid solutions (blue vs. green paths), but both require the same three key items (`pickaxe`, `gem`, `door`) acquired in different sequences. This implies the puzzle has structured, non-linear solutions. - **The elements relate** through a state-space where transitions are gated by inventory. The colored paths highlight optimal strategies, while black edges represent suboptimal or dangerous choices. The numerical values likely represent a cumulative score or cost, with the goal being to maximize the final value (reach 0 instead of -4). - **Notable anomaly:** The typo "picaxe" on the green edge from Node 3 to Node 1. This is likely an error in the diagram's creation but does not obscure the intended meaning. - **Underlying logic:** The system enforces a "lock-and-key" design. The `door` item is the final key needed at Node 1 to access the goal. The `lava` transitions act as a penalty for attempting actions without the correct items or for making wrong choices. The structure encourages exploration and item collection while punishing recklessness. </details> | <details> <summary>figures/nesy_env_ltl.png Details</summary>  ### Visual Description ## Diagram: Grid-Based Pathfinding with Objects ### Overview The image displays a 4x4 grid containing various graphical objects and two colored paths (blue and green) connecting them. The diagram appears to illustrate possible routes or connections between a starting point (a robot) and an endpoint (a portal), passing through or around intermediate items (a pickaxe and a diamond) and avoiding obstacles (lava blocks). There are no textual labels, axis titles, or legends present in the image. ### Components/Axes * **Grid Structure:** A 4x4 grid of white squares with black borders. * **Objects (by grid position, using (row, column) from top-left):** * **(1,1):** A grey robot with red accents (headphones, chest light, feet). * **(1,4):** A block of orange/yellow pixelated lava. * **(2,3):** A pickaxe with a grey head and a yellow handle. * **(3,2):** A blue diamond. * **(4,1):** A second block of orange/yellow pixelated lava. * **(4,4):** A red nether-style portal frame. * **Paths:** * **Blue Path:** A continuous blue line. * **Green Path:** A continuous green line. ### Detailed Analysis **Path Tracing and Spatial Grounding:** 1. **Blue Path:** * **Start:** Originates from the robot at (1,1). * **Route:** Travels horizontally right across row 1 to column 3. Turns down, traveling vertically down column 3 to row 3. Turns left, traveling horizontally left across row 3 to column 2. Turns down, traveling vertically down column 2 to row 4. Turns right, traveling horizontally right across row 4 to the portal at (4,4). * **Trend Verification:** The path forms a rectangular detour. It moves right, then down, then left, then down, then right. It does not intersect the pickaxe or diamond directly but passes adjacent to them. * **Objects Touched:** Starts at robot, ends at portal. Passes through cells (1,2), (1,3), (2,3), (3,3), (3,2), (4,2), (4,3). 2. **Green Path:** * **Start:** Originates from the robot at (1,1). * **Route:** Travels vertically down column 1 to row 3. Turns right, traveling horizontally right across row 3 to column 2, intersecting the diamond. Turns up, traveling vertically up column 2 to row 2. Turns right, traveling horizontally right across row 2 to column 3, intersecting the pickaxe. Turns down, traveling vertically down column 3 to row 4. Turns right, traveling horizontally right across row 4 to the portal at (4,4). * **Trend Verification:** The path forms a more complex, zig-zagging route that actively visits the intermediate items. It moves down, right (to diamond), up, right (to pickaxe), down, then right. * **Objects Touched:** Starts at robot, intersects the diamond at (3,2), intersects the pickaxe at (2,3), ends at portal. Passes through cells (2,1), (3,1), (3,2), (2,2), (2,3), (3,3), (4,3). **Object Placement Summary:** * **Lava Blocks:** Positioned at opposite corners (top-right and bottom-left). Neither path intersects them. * **Intermediate Items:** The diamond (3,2) and pickaxe (2,3) are positioned such that the green path can connect them in sequence, while the blue path bypasses them. ### Key Observations 1. **Dual Pathways:** Two distinct, non-overlapping paths connect the same start (robot) and end (portal) points. 2. **Path Divergence:** The paths diverge immediately after the start. The blue path takes a wider, outer route, while the green path takes a more central, item-focused route. 3. **Item Interaction:** The green path is the only one that intersects the diamond and the pickaxe. The blue path avoids them. 4. **Obstacle Avoidance:** Both paths successfully navigate around the two lava blocks, which act as static obstacles. 5. **Convergence:** Both paths converge on the final segment, traveling along the bottom row (row 4) from column 3 to the portal at (4,4). ### Interpretation This diagram is likely a visual representation of a **pathfinding problem or solution**, possibly from a game, puzzle, or algorithmic demonstration. * **What it Suggests:** It contrasts two strategies for reaching a goal (the portal) from a starting point (the robot). * The **Blue Path** represents a **direct or efficient route** that minimizes interaction with intermediate objects, possibly prioritizing speed or safety by avoiding the items. * The **Green Path** represents a **collection or task-oriented route** that deliberately goes out of its way to acquire resources (the diamond) and tools (the pickaxe) before proceeding to the goal. This suggests a trade-off where time or path length is sacrificed to gain advantages. * **Relationships:** The objects are not random; they form a logical set for a mining/adventure theme (robot, pickaxe, diamond, portal, lava hazard). The paths define the narrative or functional relationship between them. * **Anomalies/Notable Points:** The precise, grid-aligned paths suggest this is a schematic or logical diagram rather than a free-form illustration. The lack of text implies the visual symbols are meant to be self-explanatory within a specific context (e.g., a game level design, a flowchart for a decision tree). The lava blocks serve as clear boundary constraints for the pathfinding logic. </details> | | --- | --- | --- | --- | --- | Figure 13: (Left) Environment introduced in (Garnelo et al., 2016) and mainly used in (Badreddine and Spranger, 2019; Amador and Gierasimczuk, 2025). The agent (+ sign) has to maximize the sum of rewards by collecting shapes, each shape gives a different immediate reward. In both works, they use a sort of soft knowledge injection that is biased for this specific environment: they divide the input image into 25 patches (one per grid cell), and the shape recognizer classifies each patch singularly. (Right) Environment introduced in (Umili et al., 2024b) for temporally extended visual tasks. The agent has to maximize the reward on the automaton. It changes automaton state by reaching specific items in the environment. On the left side, the automaton for the task ”reach the pickaxe and the gem before the door, while always avoiding the ” lava ”, is displayed.On the rightmost side, a visualization of the environment is portrayed. In green and blue two optimal trajectories causing the indistinguishability of symbols ” pickaxe ” and ” gem ” due to the structure of the task. A more recent work in NeSy-RL (Umili et al., 2024b) explicitly investigates reasoning shortcuts through neural reward machines (NRMs). There, the RL agent learns to solve temporally extended tasks expressed in linear temporal logic over finite traces (LTLf) (Giacomo and Vardi, 2013) formulas. Fig. 13 (Right) illustrates an example of such a task. NRMs encode temporal knowledge through a probabilistic relaxation of the deterministic finite automaton (DFA) equivalent to LTLf task. Here, the only bias is that rewards are shaped accordingly to the automaton: reward is maximum at the automaton’s accepting states (reachable only by satisfying trajectories) and reduced elsewhere. This is a minimal bias that allows applying the framework to diverse environments and tasks without any change in the framework. In the context of the example of Fig. 13, green and blue trajectories correspond to equivalent solutions for the agent, which may induce the agent to swap the concepts ‘‘gem’’ and ‘‘pickaxe’’, resulting in an RS. Moreover, in (Umili et al., 2024b) an efficient algorithm to identify all reasoning shortcuts that depend on the temporal logical specification is introduced, allowing to count RSs over three order of magnitude faster than a brute-force baseline. While the brute-force baseline constructs all possible $α$ ’s in $Vert(A)$ , the algorithm introduced in (Umili et al., 2024b) exploits common properties of RL tasks, such as terminal states and self-loop transitions in automata, to reduce the space of the search. Applying this algorithm to the LTLf formulas commonly used in non-Markovian RL showed an extremely high number of RSs. Interestingly, in single-task RL, these RSs do not harm learning performance, since they involve only concept renamings and do not induce any concept collapse. Thus, all task-relevant concepts remain distinguished, but they may be swapped. However, this swap can harm transferring the learned concepts to new multitask settings (Kuo et al., 2021). A follow-up work (Dewidar and Umili, 2025) extends NRMs to the cases where the task logical knowledge is not provided a priori: both the knowledge and the concepts are learned directly from exploration data (Umili and Capobianco, 2025). It investigates an algorithm to jointly minimize the automaton states and concepts learned via RSs detection, yielding a knowledge that is significantly more compact and explainable. Future research can likely explore RSs in multitask generalization. In this context, prior knowledge is increasingly used to automatically generate new environments and tasks (Team et al., 2021; Bauer et al., 2023), making it timely to exploit (parts of) this knowledge during training to boost the RL agent’s performance. Another important open question concerns the role of temporal dependencies in NeSy RL. Unlike static tasks (e.g., classification or regression), RL data consists of exploration traces, where each observation depends on the previous ones, raising the challenge of how best to exploit this interdependence for better RS detection (van Krieken et al., 2025a). ### 6.5 Imbalanced Learning Tsamoura et al. (2025) identified another phenomenon that is inherent to NeSy, that of learning imbalances, which are major differences in errors that occur when classifying instances of different classes (aka class-specific risks) and can lead to poor learning of concept extractor. Existing research on machine learning (Menon et al., 2021; Cao et al., 2019; Wang et al., 2022) has studied imbalances, but this was only under the prism of long-tailed (aka imbalanced) data: data in which instances of different classes occur with very different frequencies, (He and Garcia, 2009; Horn and Perona, 2017). However, these results cannot fully characterize learning imbalances in all NeSy tasks. This is due to the fact that the background knowledge may cause learning imbalances even when the data is uniformly distributed. We illustrate the above phenomenon by the following example: **Example 6.2 (Learning Imbalances inMNIST-Max(Tsamoura et al.,2025))** *Let us consider a variant of the classical MNIST-Add scenario, referred to as MNIST-Max, in which instead of predicting the sum, we predict the maximum of two MNIST digits. We will adopt the notation in Wang et al. (2023) and denote each training sample by $(x_1,x_2,y)$ , where $y$ corresponds to the target maximum. We distinguish the following two cases: 1. The marginal of $Y$ is uniform. To put it differently, the number of training samples where $y=0$ is equal to the number of training samples where $y$ is any of the remaining nine digits. In this case, we expect that learning the digit zero will be easier than learning the digit nine, i.e., the classifier will incur fewer errors in recognizing the zero digit than in any other digit. This is because training samples of the form $(x_1,x_2,0)$ reduce our NeSy task to learning under known concepts. 1. The marginal of the hidden concept $C$ , i.e., the hidden gold concept of each MNIST digit, is uniform. When creating $(x_1,x_2,y)$ training samples by sampling from the MNIST distribution in an i.i.d. way, then the number of training samples where $y=9$ is much larger than the number of training samples where $y=0$ . As a consequence, learning the digit nine will be easier than learning the digit zero, despite the fact that learning using samples of the form $(x_1,x_2,0)$ reduces our NeSy task to learning under known concepts.* Tsamoura et al. (2025) theoretically characterized this phenomenon in NeSy, extending previous results (Cour et al., 2011). The characterization involved solving a non-linear program whose solutions are upper bounds to class-specific risks. In addition, the authors proposed algorithms to mitigate imbalances during training and testing time. The former algorithm assigns pseudolabels to training data based on a novel formulation of NeSy as an integer linear program (Srikumar and Roth, 2023). The latter algorithm constrains the model’s predictions on test data using robust semi-constrained optimal transport (Le et al., 2021). Both mitigation algorithms rely on a common idea in long-tailed learning: enforcing the class priors to the predictions of a concept extractor. The intuition is that the concept extractor will tend to predict the labels that appear more often in the training data. Hence, enforcing the priors gives more importance to the minority concepts at training time and encourages the model to predict minority concepts at testing time. ### 6.6 Additional open problems Here, we list few more directions which were not treated in previous subsections: 1. Although several mitigation strategies have been proposed (some of them quite powerful, such as relying on concept supervision (Koh et al., 2020)), the most effective ones are costly in terms of human annotations. Related to this, multitask learning ( Section 5.3.2) can be employed to obtain such annotations by designing combinations of NeSy tasks that promote the removal of reasoning shortcuts. However, it remains unclear how to construct such tasks effectively. A naive approach might involve combining learned programs (Muggleton and De Raedt, 1994; De Raedt and Kersting, 2008; Wüst et al., 2024) with rs-count (Bortolotti et al., 2024) to evaluate how the newly generated programs affect the number of reasoning shortcuts. This, however, requires access to ground-truth concepts, which is typically infeasible in real-world applications, and can also be highly inefficient in terms of computational cost. Developing efficient methods for artificially constructing such NeSy tasks therefore remains an open research problem. On the other hand, inexpensive methods such as entropy regularization (Manhaeve et al., 2021a) do not offer guarantees regarding their effectiveness in reducing the reasoning shortcuts count. There is thus a need for mitigation strategies that are both theoretically grounded and cost-efficient, requiring minimal human annotation effort while still providing provable guarantees of effectiveness. 1. Another open direction concerns the development of stronger concept detection pipelines, which could improve the overall identification process (Locatello et al., 2020a). For example, in computer vision, RSs are closely related to visual grounding (Xiao et al., 2024), where much work has focused, for instance, on object detection; however, a concrete link between object detection and RSs has yet to be established. Moreover, incorporating temporal dependencies (e.g., in reinforcement learning or in textual data) may further improve early detection and mitigation of RSs (Lippe et al., 2022, 2023). 1. To better advance the study of RSs, it is also extremely useful to extend the available experimental resources. While rsbench (Bortolotti et al., 2024) represents the first benchmark suite for a standardized evaluation of the impact of RSs in NeSy predictors and the efficacy of mitigation strategies, the field lacks benchmarks to examine how RSs affect model performance in highly challenging settings. An unexplored direction is to evaluate RSs on large, real-world NeSy datasets such as ROAD-R (Giunchiglia et al., 2023). Furthermore, to facilitate deploying NeSy predictors, it is beneficial to develop modular implementations of NeSy models that are both easy to use and freely available. Recent initiatives, like ULLER (van Krieken et al., 2024b) and DeepLog (Derkinderen et al., 2025b), are promising steps towards these implementations. ## 7 Related Work Symbol Grounding. The symbol grounding problem concerns how symbols acquire meaning independently of human interpretation (Harnad, 1990). Reasoning shortcuts correspond to a faulty attribution of meaning to symbols (w.r.t. to their human-interpretable semantics), which can hinder interpretability and out-of-distribution generalization. Harnad (1990) pioneered that symbol grounding should chiefly depend on the identification of the symbols. He argues that discrimination is the ability of systems to assign inputs representing different concepts with different symbols (e.g., images of horses and cats are associated to different symbols), but that this is not sufficient: identification further requires that these symbols are exactly the concepts we have in mind for them (e.g., for an image of a horse, the symbol must be exactly the concept of horse). Identification is precisely what underlies the grounding of the symbols. Similarly, an RS (Section 4) is when we discriminate for the purposes of the task, but fail to completely identify concepts. The symbol grounding problem has inspired challenging datasets for visual reasoning like CLEVR (Johnson et al., 2017) and the visual abstractions benchmark (Hsu et al., 2025). The former requires scene understanding based on multiple 3D objects annotated with textual description, whereas the latter focuses on grounding highly varied instantiations of the same concept (e.g., mazes). They argue that schemas, a collection of connected lower-level concepts, together form the basis for grounding higher-level abstract concepts like a maze. Symbol grounding is also closely related to binding (Greff et al., 2020) in neural networks, that is how models derive a compositional understanding of the world in terms of symbolic entities, like objects or concepts. Recently, researchers asked to what degree foundation models require — or even have already achieved — symbol grounding (Hsu et al., 2023; Jiang et al., 2024). Some authors argue that the problem remains for models that are trained purely on language (Pavlick, 2023; Levine, 2025), which are symbolic in nature. Others argue that forming meaningful concepts requires embodied experiences (Barsalou, 2020, 1999), which current models fundamentally lack (Dove, 2024; Levine, 2025). Steels et al. (2008) argues the focus should be on AI systems that autonomously create and maintain the grounding of concepts, and that supervised machine learning is insufficient as the semantics still comes from humans. Part of the focus has shifted to what forms of grounding —if linguistic, multimodal, or embodied— are necessary for genuine conceptual understanding (Barsalou, 2020; Gubelmann, 2024) and to whether this should be more framed specifically for distributed representations (Mollo and Millière, 2023). While NeSy predictors go beyond supervised machine learning and reduce reliance on direct concept supervision, they still require human-provided knowledge to be effective and currently do not support autonomous concept acquisition. Inductive Logic Programming Inductive logic programming (ILP) (De Raedt and Kersting, 2008; Muggleton and De Raedt, 1994) is a branch of machine learning that induces logic programs from examples and background knowledge. Hypotheses are typically represented as sets of first-order Horn clauses, and the goal is to find a symbolic program that entails all positive examples while excluding negative ones. One important technique in ILP is predicate invention, where the system autonomously generates new predicates to capture latent relationships in the data (Stahl, 1993). By creating intermediate concepts, predicate invention allows ILP models to abstract complex relational patterns, simplify rule sets, and improve generalization. For example, in an animal classification task, an ILP system might invent a mammalian(X) predicate to encode the mammal concept. However, the grounding of invented predicates can misalign with the one intended by the human, which makes predicate invention an interesting research direction to study connections with reasoning shortcuts and human interpretability. A recurring challenge in ILP, both with and without predicate invention, is the presence of symmetries in the hypothesis space, which often leads to redundant search (Tarzariol et al., 2022). For instance, two clauses such as parent(X,Y) :- mother(X,Y). and parent(A,B) :- mother(A,B). are equivalent as they differ only in variable naming. Similarly, two clauses that differ only in the order of their literals are identical: for example, sibling(X,Y) :- parent(Z,X), parent(Z,Y). is equivalent to sibling(X,Y) :- parent(Z,Y), parent(Z,X). Exploring the hypothesis space in ILP without addressing symmetries can lead to the generation of many redundant, equivalent clauses. To mitigate this issue, various ILP systems employ symmetry-breaking techniques that constrain the search space and avoid unnecessary symmetries. For example, FOIL (Quinlan, 1990) addressed this by enforcing a consistent left-to-right ordering of literals, while Progol (Muggleton, 1995) and Aleph (Srinivasan, 2001) used mode declarations, which restrict the possible forms that clauses can take, and Metagol (Cropper and Muggleton, 2016), instead, provided metarules, which define general clause schemata that implicitly prevent the generation of equivalent variants. In ILP, symmetries arise from structural redundancies in the hypothesis space, such as clauses that are equivalent up to variable renaming, literal order, or clause permutation. These symmetries may not permit identification: there can be multiple programs that fit the data equally well, yet differ in how they capture the intended abstraction. In particular, recurring relational patterns in the optimal hypothesis space can create multiple equivalent representations, each providing the same explanatory power. An analogous problem also occurs in satisfiability (SAT), where variables or clauses that play equivalent roles introduce redundancy in the search space (Sakallah, 2021). Just as in ILP, SAT solvers employ symmetry-breaking techniques to eliminate redundant solutions (Anders et al., 2024; Ulyantsev et al., 2016; Bogaerts et al., 2022). Studying the connection between symmetries in the search space and reasoning shortcuts can be of interest, as it may offer insights into both improving learning efficiency and understanding how learned hypotheses relate to intended abstractions. Shortcuts in Machine Learning. Machine learning models often rely on shortcuts, spurious correlations between input features and target labels that are not causally related, leading to poor performance, particularly in OOD settings (Geirhos et al., 2020; Teso et al., 2023; Ye et al., 2024; Steinmann et al., 2024). Prior work related to concepts has examined how spurious correlations affect both CBMs and NeSy models (Margeloiu et al., 2021; Mahinpei et al., 2021; Havasi et al., 2022; Raman et al., 2023; Stammer et al., 2021). These studies primarily focus on understanding and correcting concept quality in the presence of spurious confounders in the data. To address this problem, strategies such as Explanatory Interactive Learning (Teso and Kersting, 2019; Teso et al., 2023; Bontempelli et al., 2021; Teso et al., 2021; Bontempelli et al., 2023) have been developed, aiming to identify such spurious correlation and allow humans to correct them during the model debugging process. However, this line of work does not consider the high-level abstraction required to connect symbols to input features, a challenge central to the symbol grounding problem. In contrast, this paper focuses on reasoning shortcuts (Li et al., 2023; Marconato et al., 2023b; Wang et al., 2023; Umili et al., 2024a; Marconato et al., 2024; Bortolotti et al., 2024; DeLong et al., 2024; Bortolotti et al., 2025), which arise not from low-level feature correlations but from the misuse of concepts during decision making, as their semantics differ from the intended ones. Identifiability Identifiability has been largely studied in representation learning, especially within independent component analysis (ICA) (Hyvärinen et al., 2009, 2024) and causal representation learning (CRL) (Schölkopf et al., 2021). In ICA, the goal is to determine the independent underlying components from observed data. There, the central question of identifiability is: under what conditions can the same independent components (up to tolerable ambiguities) be consistently recovered (Buchholz et al., 2022; Gresele, 2023)? If components are not identifiable, re-estimating them at different times may yield conflicting results, meaning there is no unique way to determine the same independent factors. The same lack of uniqueness also appears in reasoning shortcuts. Seminal works have explored conditions under which non-linear independent components can be uniquely recovered, including using auxiliary variables (Hyvarinen et al., 2019; Khemakhem et al., 2020), sparsity (Lachapelle et al., 2022), anchor points (Moran et al., 2021), weak supervision (Locatello et al., 2020b), multiple views (Gresele et al., 2020), and constraints to the model family (Gresele et al., 2021). These ideas have also been extended to CRL, where identifiability is necessary not only to discover the underlying components but also to reconstruct the causal relationships between them (Von Kügelgen et al., 2021; Lippe et al., 2022; Buchholz et al., 2023; Ahuja et al., 2023; Lippe et al., 2023; von Kügelgen et al., 2024; Fokkema et al., 2025). We foresee that many techniques and learning setups which give guarantees for identification can be of help also to mitigating RSs, but so far the connection to ICA and CRL has not been investigated. Identifiability is also studied in supervised classification (Reizinger et al., 2024), multi-task learning (Lachapelle et al., 2023; Fumero et al., 2023), contrastive learning (Von Kügelgen et al., 2021; Zimmermann et al., 2021), and language models (Roeder et al., 2021; Marconato et al., 2025). In these settings, models may achieve close predictive performance while learning dissimilar internal representations (Nielsen et al., 2024, 2025). This situation is analogous to reasoning shortcuts, where correct outputs are obtained through misaligned or unintended representations. Revealing the precise connection between these models and RSs may help us extend beyond the current paradigm of NeSy predictors. ## 8 Conclusion Symbol grounding aims to link humans’ and machines’ high-level abstractions of the world. Well-grounded symbols, or concepts, contribute to generalization and interpretability. NeSy AI holds the great promise of facilitating this alignment in hybrid systems that are human interpretable and understandable. Our work characterizes the past, present and future challenges of symbol grounding in NeSy AI systems through the lens of RSs. The presence of RSs prevents identification of the correct concepts, and thus possibly compromises the benefits of NeSy AI. Our theoretical analysis in Section 4 indicates that, in general, all NeSy predictors we consider – and likely most others – are naturally susceptible to RSs. That is, concepts are non-identifiable in general. However, the theory also characterizes the mechanisms behind the occurrence of RSs, and therefore paves the way for designing a new generation of NeSy AI systems that are more reliable and easier to align to humans’ expectations effective mitigation strategies. More work is however needed to articulate what conditions guarantee provable retrieval of the ground-truth concepts without requiring dense concept annotations. We expect that this can done in a similar spirit and with similar strategies as those use for guaranteeing of representations in independent component analysis and causal representation learning (see Section 7). ## Acknowledgements Funded by the European Union. The views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union, the European Health and Digital Executive Agency (HaDEA) or the European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Grant Agreement no. 101120763 - TANGO. Emile van Krieken is funded by the NWO AiNed project “Human-Centric AI Agents with Common Sense” under contract number NGF.1607.22.044. Paolo Morettin is supported by the MSCA project “Probabilistic Formal Verification for Provably Trustworthy AI - PFV-4-PTAI” under GA no. 101110960. AV is supported by the “UNREAL: Unified Reasoning Layer for Trustworthy ML” project (EP/Y023838/1) selected by the ERC and funded by UKRI EPSRC. We thank Samy Badreddine for useful input while writing this paper. ## References - Mandler [2004] Jean Matter Mandler. The foundations of mind: Origins of conceptual thought. Oxford University Press, 2004. - Spelke and Kinzler [2007] Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89–96, 2007. - Whitehead [1927] Alfred North Whitehead. Symbolism, its meaning and effect. Macmillan, 1927. - Johnson-Laird [1994] Philip N Johnson-Laird. Mental models and probabilistic thinking. Cognition, 50(1-3):189–209, 1994. - Smolensky [1987] Paul Smolensky. Analysis of distributed representation of constituent structure in connectionist systems. In Neural Information Processing Systems, 1987. - Sun [1992] Ron Sun. On variable binding in connectionist networks. Connection Science, 4(2):93–124, 1992. - Greff et al. [2020] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020. - Jo and Bengio [2017] Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities. arXiv preprint arXiv:1711.11561, 2017. - Lake and Baroni [2018] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pages 2873–2882. PMLR, 2018. - Park et al. [2024] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, pages 39643–39666. PMLR, 2024. - Harnad [1990] Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1–3):335–346, June 1990. ISSN 0167-2789. doi: 10.1016/0167-2789(90)90087-6. - McMillan et al. [1991] Clayton McMillan, Michael C Mozer, and Paul Smolensky. Rule induction through integrated symbolic and subsymbolic processing. Advances in neural information processing systems, 4, 1991. - De Raedt et al. [2021] Luc De Raedt, Sebastijan Dumančić, Robin Manhaeve, and Giuseppe Marra. From statistical relational to neural-symbolic artificial intelligence. In IJCAI, 2021. - Ellis et al. [2021] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation, pages 835–850, 2021. - Kambhampati et al. [2022] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12262–12267, 2022. - Marconato et al. [2023a] Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso. Neuro symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal. In ICML, 2023a. - Marconato et al. [2023b] Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini. Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. In NeurIPS, 2023b. - Yang et al. [2024] Xiao-Wen Yang, Wen-Da Wei, Jie-Jing Shao, Yu-Feng Li, and Zhi-Hua Zhou. Analysis for abductive learning and neural-symbolic reasoning shortcuts. In ICML, 2024. - Manhaeve et al. [2018] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS, 2018. - Chang et al. [2020] Oscar Chang, Lampros Flokas, Hod Lipson, and Michael Spranger. Assessing satnet’s ability to solve the symbol grounding problem. Advances in Neural Information Processing Systems, 33:1428–1439, 2020. - Topan et al. [2021] Sever Topan, David Rolnick, and Xujie Si. Techniques for symbol grounding with satnet. Advances in Neural Information Processing Systems, 34:20733–20744, 2021. - Umili et al. [2024a] Elena Umili, Francesco Argenziano, and Roberto Capobianco. Neural reward machines. In ECAI 2024, pages 3055–3062. Ios Press, 2024a. - DeLong et al. [2024] Lauren Nicole DeLong, Yojana Gadiya, Paola Galdi, Jacques D Fleuriot, and Daniel Domingo-Fernández. Mars: A neurosymbolic approach for interpretable drug discovery. arXiv preprint arXiv:2410.05289, 2024. - Wang et al. [2023] Kaifu Wang, Efthymia Tsamoura, and Dan Roth. On learning latent models with multi-instance weak supervision. In NeurIPS, 2023. - Bortolotti et al. [2025] Samuele Bortolotti, Emanuele Marconato, Paolo Morettin, Andrea Passerini, and Stefano Teso. Shortcuts and identifiability in concept-based models from a neuro-symbolic lens, 2025. URL https://arxiv.org/abs/2502.11245. - Garcez et al. [2022] Artur d’Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illuminoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342:1, 2022. - Feldstein et al. [2024] Jonathan Feldstein, Paulius Dilkas, Vaishak Belle, and Efthymia Tsamoura. Mapping the neuro-symbolic ai landscape by architectures: A handbook on augmenting deep learning through symbolic reasoning, 2024. URL https://arxiv.org/abs/2410.22077. - Hoernle et al. [2022] Nick Hoernle, Rafael Michael Karampatsis, Vaishak Belle, and Kobi Gal. Multiplexnet: Towards fully satisfied logical constraints in neural networks. In AAAI, 2022. - Ahmed et al. [2022] Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic probabilistic layers for neuro-symbolic learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 29944–29959. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/c182ec594f38926b7fcb827635b9a8f4-Paper-Conference.pdf. - Buffelli and Tsamoura [2023] Davide Buffelli and Efthymia Tsamoura. Scalable theory-driven regularization of scene graph generation models. In Thirty-Seventh AAAI Conference on Artificial Intelligence, Washington, DC, USA, February 7-14, 2023, pages 6850–6859, 2023. doi: 10.1609/AAAI.V37I6.25839. URL https://doi.org/10.1609/aaai.v37i6.25839. - Lippi and Frasconi [2009] Marco Lippi and Paolo Frasconi. Prediction of protein $β$ -residue contacts by markov logic networks with grounding-specific weights. Bioinformatics, 2009. - Diligenti et al. [2012] Michelangelo Diligenti, Marco Gori, Marco Maggini, and Leonardo Rigutini. Bridging logic and kernel machines. Machine learning, 86(1):57–88, 2012. - Zhou [2019] Zhi-Hua Zhou. Abductive learning: towards bridging machine learning and logical reasoning. Science China. Information Sciences, 62(7):76101, 2019. - Donadello et al. [2017] Ivan Donadello, Luciano Serafini, and Artur D’Avila Garcez. Logic tensor networks for semantic image interpretation. In IJCAI, 2017. - van Krieken et al. [2020] Emile van Krieken, Erman Acar, and Frank van Harmelen. Analyzing differentiable fuzzy implications. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, volume 17, pages 893–903, 2020. - Feldstein et al. [2023] Jonathan Feldstein, Modestas Jurcius, and Efthymia Tsamoura. Parallel neurosymbolic integration with concordia. In International Conference on Machine Learning (ICML), 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 9870–9885, 2023. - Rocktäschel and Riedel [2016] Tim Rocktäschel and Sebastian Riedel. Learning knowledge base inference with neural theorem provers. In Proceedings of the 5th workshop on automated knowledge base construction, pages 45–50, 2016. - Giunchiglia and Lukasiewicz [2020] Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. NeurIPS, 2020. - Badreddine et al. [2022] Samy Badreddine, Artur d’Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence, 303:103649, 2022. - Maene and Tsamoura [2025] Jaron Maene and Efthymia Tsamoura. Embeddings as probabilistic equivalence in logic programs. In Proceedings of the Thirty-Ninth Conference on Neural Information Processing Systems (NeurIPS), 2025. - Giunchiglia et al. [2022] Eleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. Deep learning with logical constraints. arXiv preprint arXiv:2205.00523, 2022. - Dash et al. [2022] Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports, 2022. - Xu et al. [2020] Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. - Manhaeve et al. [2021a] Robin Manhaeve, Sebastijan Dumančić, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Neural probabilistic logic programming in deepproblog. Artificial Intelligence, 298:103504, 2021a. - Yang et al. [2020] Zhun Yang, Adam Ishay, and Joohyung Lee. Neurasp: Embracing neural networks into answer set programming. In IJCAI 2020, 2020. - Huang et al. [2021a] Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. NeurIPS, 2021a. - De Raedt and Kimmig [2015] Luc De Raedt and Angelika Kimmig. Probabilistic (logic) programming concepts. Machine Learning, 100:5–47, 2015. - De Raedt et al. [2007] Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: A probabilistic prolog and its application in link discovery. In IJCAI, 2007. - Kimmig et al. [2011] Angelika Kimmig, Bart Demoen, Luc De Raedt, Vitor Santos Costa, and Ricardo Rocha. On the implementation of the probabilistic logic programming language problog. Theory and Practice of Logic Programming, 2011. - Koller and Friedman [2009] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009. - Darwiche and Marquis [2002] Adnan Darwiche and Pierre Marquis. A knowledge compilation map. Journal of Artificial Intelligence Research, 17:229–264, 2002. - Choi et al. [2020] Y Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. UCLA. URL: http://starai. cs. ucla. edu/papers/ProbCirc20. pdf, page 6, 2020. - Vergari et al. [2021] Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. Advances in Neural Information Processing Systems, 34:13189–13201, 2021. - Maene et al. [2025] Jaron Maene, Vincent Derkinderen, and Pedro Zuidberg Dos Martires. Klay: Accelerating arithmetic circuits for neurosymbolic ai. In ICLR, 2025. - Derkinderen et al. [2025a] Vincent Derkinderen, Robin Manhaeve, Rik Adriaensen, Lucas Van Praet, Lennert De Smet, Giuseppe Marra, and Luc De Raedt. The deeplog neurosymbolic machine, 2025a. URL https://arxiv.org/abs/2508.13697. - Manhaeve et al. [2021b] Robin Manhaeve, Giuseppe Marra, and Luc De Raedt. Approximate inference for neural probabilistic logic programming. In KR, 2021b. - Winters et al. [2022] Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. DeepStochLog: Neural Stochastic Logic Programming. In AAAI, 2022. - De Smet et al. [2023a] Lennert De Smet, Emanuele Sansone, and Pedro Zuidberg Dos Martires. Differentiable sampling of categorical distributions using the catlog-derivative trick. Advances in Neural Information Processing Systems, 36:30416–30428, 2023a. - van Krieken et al. [2023] Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomczak, Frank van Harmelen, and Annette ten Teije. A-nesi: A scalable approximate method for probabilistic neurosymbolic inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. - Choi et al. [2025] Seewon Choi, Alaia Solko-Breslin, Rajeev Alur, and Eric Wong. Ctsketch: Compositional tensor sketching for scalable neurosymbolic learning. arXiv preprint arXiv:2503.24123, 2025. - Chen et al. [2025] Weixin Chen, Simon Yu, Huajie Shao, Lui Sha, and Han Zhao. Neural probabilistic circuits: Enabling compositional and interpretable predictions through logical reasoning. arXiv preprint arXiv:2501.07021, 2025. - Xu et al. [2018] Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for deep learning with symbolic knowledge. In ICML, 2018. - Giannini et al. [2018] Francesco Giannini, Michelangelo Diligenti, Marco Gori, and Marco Maggini. On a convex logic fragment for learning and reasoning. IEEE Transactions on Fuzzy Systems, 27(7):1407–1416, 2018. - van Krieken et al. [2022] Emile van Krieken, Erman Acar, and Frank Van Harmelen. Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302:103602, 2022. - Di Liello et al. [2020] Luca Di Liello, Pierfrancesco Ardino, Jacopo Gobbi, Paolo Morettin, Stefano Teso, and Andrea Passerini. Efficient generation of structured objects with constrained adversarial networks. Advances in neural information processing systems, 33:14663–14674, 2020. - Giunchiglia et al. [2024] Eleonora Giunchiglia, Alex Tatomir, Mihaela Cătălina Stoian, and Thomas Lukasiewicz. Ccn+: A neuro-symbolic framework for deep learning with requirements. International Journal of Approximate Reasoning, page 109124, 2024. - Xie et al. [2022] Xuan Xie, Kristian Kersting, and Daniel Neider. Neuro-symbolic verification of deep neural networks. In IJCAI, 2022. - Zaid et al. [2023] Faried Abu Zaid, Dennis Diekmann, and Daniel Neider. Distribution-aware neuro-symbolic verification. AISoLA, pages 445–447, 2023. - Morettin et al. [2024] Paolo Morettin, Andrea Passerini, and Roberto Sebastiani. A unified framework for probabilistic verification of ai systems via weighted model integration. arXiv preprint arXiv:2402.04892, 2024. - Yang et al. [2023a] Wen-Chi Yang, Giuseppe Marra, Gavin Rens, and Luc De Raedt. Safe reinforcement learning via probabilistic logic shields. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 5739–5749, 2023a. - Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017. - Huang et al. [2021b] Xuanxiang Huang, Yacine Izza, Alexey Ignatiev, Martin C Cooper, Nicholas Asher, and Joao Marques-Silva. Efficient explanations for knowledge compilation languages. arXiv preprint arXiv:2107.01654, 2021b. - Teso et al. [2023] Stefano Teso, Öznur Alkan, Wolfang Stammer, and Elizabeth Daly. Leveraging explanations in interactive machine learning: An overview. Frontiers in Artificial Intelligence, 2023. - LeCun [1998] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998. - Wang et al. [2019] Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning, pages 6545–6554. PMLR, 2019. - Marconato et al. [2024] Emanuele Marconato, Samuele Bortolotti, Emile van Krieken, Antonio Vergari, Andrea Passerini, and Stefano Teso. BEARS Make Neuro-Symbolic Models Aware of their Reasoning Shortcuts. Uncertainty in AI, 2024. - Bortolotti et al. [2024] Samuele Bortolotti, Emanuele Marconato, Tommaso Carraro, Paolo Morettin, Emile van Krieken, Antonio Vergari, Stefano Teso, and Andrea Passerini. A neuro-symbolic benchmark suite for concept quality and reasoning shortcuts. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 115861–115905. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/d1d11bf8299334d354949ba8738e8301-Paper-Datasets_and_Benchmarks_Track.pdf. - Li et al. [2023] Zenan Li, Zehua Liu, Yuan Yao, Jingwei Xu, Taolue Chen, Xiaoxing Ma, L Jian, et al. Learning with logical constraints but without shortcut satisfaction. In ICLR, 2023. - Manginas et al. [2025] Vasileios Manginas, Nikolaos Manginas, Edward Stevinson, Sherwin Varghese, Nikos Katzouris, Georgios Paliouras, and Alessio Lomuscio. A scalable approach to probabilistic neuro-symbolic robustness verification. In 19th International Conference on Neurosymbolic Learning and Reasoning, 2025. URL https://openreview.net/forum?id=DAp8WCTGVj. - Koh et al. [2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338–5348. PMLR, 2020. - Poeta et al. [2023] Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept-based explainable artificial intelligence: A survey. arXiv preprint arXiv:2312.12936, 2023. - van Krieken et al. [2025a] Emile van Krieken, Pasquale Minervini, Edoardo Ponti, and Antonio Vergari. Neurosymbolic reasoning shortcuts under the independence assumption. In Proceedings of the 19th International Conference on Neurosymbolic Learning and Reasoning, volume 284 of Proceedings of Machine Learning Research. PMLR, 2025a. - Vapnik [2013] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013. - Shalev-Shwartz and Ben-David [2014] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, 2014. ISBN 1107057132. - Steinhardt and Liang [2015] Jacob Steinhardt and Percy S Liang. Learning with relaxed supervision. In NeurIPS, volume 28, 2015. - Raghunathan et al. [2016] Aditi Raghunathan, Roy Frostig, John Duchi, and Percy Liang. Estimation from indirect supervision with linear moments. In ICML, volume 48, pages 2568–2577, 2016. - Liu and Dietterich [2014] Li-Ping Liu and Thomas G. Dietterich. Learnability of the superset label learning problem. In ICML, pages 1629––1637, 2014. - Gomes et al. [2021] Carla P. Gomes, Ashish Sabharwal, and Bart Selman. Model counting. In Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh, editors, Handbook of Satisfiability - Second Edition, volume 336 of Frontiers in Artificial Intelligence and Applications, pages 993–1014. IOS Press, 2021. doi: 10.3233/FAIA201009. URL https://doi.org/10.3233/FAIA201009. - Chakraborty et al. [2021] Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. Approximate model counting. In Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh, editors, Handbook of Satisfiability - Second Edition, volume 336 of Frontiers in Artificial Intelligence and Applications, pages 1015–1045. IOS Press, 2021. doi: 10.3233/FAIA201010. URL https://doi.org/10.3233/FAIA201010. - van Krieken et al. [2025b] Emile van Krieken, Pasquale Minervini, Edoardo Ponti, and Antonio Vergari. Neurosymbolic diffusion models. arXiv preprint arXiv:2505.13138, 2025b. - Oikarinen et al. [2022] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In ICLR, 2022. - Yang et al. [2023b] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023b. - Rao et al. [2024] Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery. In European Conference on Computer Vision, 2024. - Srivastava et al. [2024] Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. Vlg-cbm: Training concept bottleneck models with vision-language guidance. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 79057–79094. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/90043ebd68500f9efe84fedf860a64f3-Paper-Conference.pdf. - Yuksekgonul et al. [2023] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=nA5AZ8CEyow. - Debole et al. [2025] Nicola Debole, Pietro Barbiero, Francesco Giannini, Andrea Passerini, Stefano Teso, and Emanuele Marconato. If concept bottlenecks are the question, are foundation models the answer?, 2025. URL https://arxiv.org/abs/2504.19774. - Wüst et al. [2025] Antonia Wüst, Tim Tobiasch, Lukas Helff, Inga Ibs, Wolfgang Stammer, Devendra S. Dhami, Constantin A. Rothkopf, and Kristian Kersting. Bongard in wonderland: Visual puzzles that still make ai go mad?, 2025. - Caruana [1997] Rich Caruana. Multitask learning. Machine learning, 1997. - Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016. - Kingma [2013] Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. - Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Everest Hinton. A simple framework for contrastive learning of visual representations. 2020. - Suter et al. [2019] Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In ICML, 2019. - Cottrell et al. [1987] Garrison W Cottrell, Paul Munro, and David Zipser. Learning internal representation from gray-scale images: An example of extensional programming. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 9, 1987. - Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. - Higgins et al. [2018] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018. - Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. - Shindo et al. [2023] Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting. (alpha)ilp: thinking visual scenes as differentiable logic programs. Machine Learning, 112(5):1465–1497, 2023. - Girshick [2015] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015. - Daniele et al. [2023] Alessandro Daniele, Tommaso Campari, Sagar Malhotra, and Luciano Serafini. Deep symbolic learning: discovering symbols and rules from perceptions. In IJCAI, 2023. - Tang and Ellis [2023] Hao Tang and Kevin Ellis. From perception to programs: regularize, overparameterize, and amortize. In ICML, 2023. - Hyvärinen et al. [2024] Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics, 76(1):1–33, 2024. - van Krieken et al. [2024a] Emile van Krieken, Pasquale Minervini, Edoardo Ponti, and Antonio Vergari. On the independence assumption in neurosymbolic learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 49078–49097. PMLR, 21–27 Jul 2024a. - Wang and Yeung [2020] Hao Wang and Dit-Yan Yeung. A survey on bayesian deep learning. ACM computing surveys (csur), 53(5):1–37, 2020. - Daxberger et al. [2021] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in neural information processing systems, 34:20089–20103, 2021. - Osawa et al. [2019] Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian principles. Advances in neural information processing systems, 32, 2019. - Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. - Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016. - Austin et al. [2021] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021. - Sahoo et al. [2024] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136–130184, 2024. - Ware et al. [2001] Malcom Ware, Eibe Frank, Geoffrey Holmes, Mark Hall, and Ian Witten. Interactive machine learning: Letting users build classifiers. International Journal of Human-Computer Studies, 55:281–292, 09 2001. doi: 10.1006/ijhc.2001.0499. - Fails and Olsen [2003] Jerry Alan Fails and Dan R. Olsen. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI ’03, page 39–45, New York, NY, USA, 2003. Association for Computing Machinery. ISBN 1581135866. doi: 10.1145/604045.604056. URL https://doi.org/10.1145/604045.604056. - Amershi et al. [2014] Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. Power to the people: The role of humans in interactive machine learning. AI Magazine, 35(4):105–120, Dec. 2014. doi: 10.1609/aimag.v35i4.2513. URL https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2513. - Teso and Kersting [2019] Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 239–245, 2019. - Settles [2012] Burr Settles. Active Learning, volume 6. 06 2012. doi: 10.2200/S00429ED1V01Y201207AIM018. - De Smet et al. [2023b] Lennert De Smet, Pedro Zuidberg Dos Martires, Robin Manhaeve, Giuseppe Marra, Angelika Kimmig, and Luc De Raedt. Neural probabilistic logic programming in discrete-continuous domains. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 529–538. PMLR, 31 Jul–04 Aug 2023b. URL https://proceedings.mlr.press/v216/de-smet23a.html. - Badreddine et al. [2023] Samy Badreddine, Luciano Serafini, and Michael Spranger. logltn: Differentiable fuzzy logic in the logarithm space, 2023. URL https://arxiv.org/abs/2306.14546. - Pryor et al. [2022] Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Wang, and Lise Getoor. Neupsl: Neural probabilistic soft logic. arXiv preprint arXiv:2205.14268, 2022. - Maene and Raedt [2023] Jaron Maene and Luc De Raedt. Soft-unification in deep probabilistic logic. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. - Simard et al. [1991] Patrice Simard, Bernard Victorri, Yann LeCun, and John S Denker. Tangent prop-a formalism for specifying selected invariances in an adaptive network. In NIPS, volume 91, pages 895–903, 1991. - Misino et al. [2022] Eleonora Misino, Giuseppe Marra, and Emanuele Sansone. VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming. NeurIPS, 2022. - Ferber et al. [2024] Aaron M Ferber, Arman Zharmagambetov, Taoan Huang, Bistra Dilkina, and Yuandong Tian. GenCO: Generating diverse designs with combinatorial constraints. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 13445–13459. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/ferber24a.html. - Espinosa Zarlenga et al. [2022] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, et al. Concept embedding models. In NeurIPS 2022-36th Conference on Neural Information Processing Systems, 2022. - Marconato et al. [2022] Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models. Advances in Neural Information Processing Systems, 35:21212–21227, 2022. - Schwalbe [2022] Gesina Schwalbe. Concept embedding analysis: A review. arXiv preprint arXiv:2203.13909, 2022. - Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, July 2021. - Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. - Stammer et al. [2024] Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting. Neural concept binder. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. - Srinivasan et al. [2021] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2443–2449, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380379. doi: 10.1145/3404835.3463257. URL https://doi.org/10.1145/3404835.3463257. - Huang et al. [2024] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2024. - Yuan et al. [2024] Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. Do llms overcome shortcut learning? an evaluation of shortcut challenges in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12188–12200, 2024. - Calanzone et al. [2025] Diego Calanzone, Stefano Teso, and Antonio Vergari. Logically consistent language models via neuro-symbolic integration. In The Thirteenth International Conference on Learning Representations, 2025. - Sahu et al. [2022] Pritish Sahu, Michael Cogswell, Yunye Gong, and Ajay Divakaran. Unpacking large language models with conceptual consistency. arXiv preprint arXiv:2209.15093, 2022. - Stein et al. [2025] Adam Stein, Aaditya Naik, Neelay Velingker, Mayur Naik, and Eric Wong. Neuro-symbolic programming in the age of foundation models: Pitfalls and opportunities. Proceedings of Machine Learning Research vol vvv, 1:18, 2025. - Zheng et al. [2025] Yujia Zheng, Shaoan Xie, and Kun Zhang. Nonparametric identification of latent concepts. In Forty-second International Conference on Machine Learning, 2025. - Liu et al. [2025] Yuhang Liu, Dong Gong, Yichao Cai, Erdun Gao, Zhen Zhang, Biwei Huang, Mingming Gong, Anton van den Hengel, and Javen Qinfeng Shi. I predict therefore i am: Is next token prediction enough to learn human-interpretable concepts from data? arXiv preprint arXiv:2503.08980, 2025. - Rajendran et al. [2024] Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Kumar Ravikumar. From causal to concept-based representation learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. - Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, 2018. - Jiang and Bansal [2019] Yichen Jiang and Mohit Bansal. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726–2736, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1262. URL https://aclanthology.org/P19-1262/. - Acharya et al. [2024] Kamal Acharya, Waleed Raza, Carlos M. J. M. Dourado Júnior, Alvaro Velasquez, and Houbing Herbert Song. Neurosymbolic reinforcement learning and planning: A survey. IEEE Trans. Artif. Intell., 5(5):1939–1953, 2024. doi: 10.1109/TAI.2023.3311428. URL https://doi.org/10.1109/TAI.2023.3311428. - Anderson et al. [2020] Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforcement learning with formally verified exploration. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/448d5eda79895153938a8431919f4c9f-Abstract.html. - Yang et al. [2023c] Wen-Chi Yang, Giuseppe Marra, Gavin Rens, and Luc De Raedt. Safe reinforcement learning via probabilistic logic shields. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 5739–5749, 2023c. doi: 10.24963/IJCAI.2023/637. URL https://doi.org/10.24963/ijcai.2023/637. - Baugh et al. [2025] Kexin Gu Baugh, Luke Dickens, and Alessandra Russo. Neural DNF-MT: A neuro-symbolic approach for learning interpretable and editable policies. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2025, Detroit, MI, USA, May 19-23, 2025, pages 252–260, 2025. doi: 10.5555/3709347.3743538. URL https://dl.acm.org/doi/10.5555/3709347.3743538. - Deane and Ray [2025] Oliver Deane and Oliver Ray. Neuro-symbolic inverse constrained reinforcement learning. In 19th International Conference on Neurosymbolic Learning and Reasoning, 2025. URL https://openreview.net/forum?id=oVb3sJAnfx. - Umili et al. [2024b] Elena Umili, Francesco Argenziano, and Roberto Capobianco. Neural reward machines. In ECAI 2024 - 27th European Conference on Artificial Intelligence, 19-24 October 2024, Santiago de Compostela, Spain - Including 13th Conference on Prestigious Applications of Intelligent Systems (PAIS 2024), pages 3055–3062, 2024b. doi: 10.3233/FAIA240847. URL https://doi.org/10.3233/FAIA240847. - Mitchener et al. [2022] Ludovico Mitchener, David Tuckey, Matthew Crosby, and Alessandra Russo. Detect, understand, act: A neuro-symbolic hierarchical reinforcement learning framework. Mach. Learn., 111(4):1523–1549, 2022. doi: 10.1007/S10994-022-06142-7. URL https://doi.org/10.1007/s10994-022-06142-7. - Garnelo et al. [2016] Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. CoRR, abs/1609.05518, 2016. URL http://arxiv.org/abs/1609.05518. - Badreddine and Spranger [2019] Samy Badreddine and Michael Spranger. Injecting prior knowledge for transfer learning into reinforcement learning algorithms using logic tensor networks. In Proceedings of the 2019 International Workshop on Neural-Symbolic Learning and Reasoning (NeSy 2019), Annual workshop of the Neural-Symbolic Learning and Reasoning Association, Macao, China, August 12, 2019, 2019. - Umili et al. [2021] Elena Umili, Emanuele Antonioni, Francesco Riccio, Roberto Capobianco, Daniele Nardi, and Giuseppe De Giacomo. Learning a symbolic planning domain through the interaction with continuous environments. In Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL), 2021. - Hafner et al. [2021] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0oabwyZbOu. - Amador and Gierasimczuk [2025] Ivo Amador and Nina Gierasimczuk. SymDQN: Symbolic knowledge and reasoning in neural network-based reinforcement learning. In 19th International Conference on Neurosymbolic Learning and Reasoning, 2025. URL https://openreview.net/forum?id=ncEGGRYska. - Umili and Capobianco [2025] Elena Umili and Roberto Capobianco. Learning minimal symbolic representations and temporal rules from visual sequences. 04 2025. doi: 10.13140/RG.2.2.27713.26729. URL http://dx.doi.org/10.13140/RG.2.2.27713.26729. - Giacomo and Vardi [2013] Giuseppe De Giacomo and Moshe Y. Vardi. Linear temporal logic and linear dynamic logic on finite traces. In IJCAI 2013, Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, China, August 3-9, 2013, pages 854–860, 2013. URL http://www.aaai.org/ocs/index.php/IJCAI/IJCAI13/paper/view/6997. - Kuo et al. [2021] Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Compositional RL agents that follow language commands in temporal logic. Frontiers Robotics AI, 8:689550, 2021. doi: 10.3389/FROBT.2021.689550. URL https://doi.org/10.3389/frobt.2021.689550. - Dewidar and Umili [2025] Hazem Dewidar and Elena Umili. Fully learnable reward machines. In 7th International Workshop on Artificial Intelligence and fOrmal VERification, Logic, Automata, and sYnthesis (OVERLAY 2025), 2025. URL https://arxiv.org/pdf/2509.19017. - Team et al. [2021] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michaël Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to generally capable agents. CoRR, abs/2107.12808, 2021. URL https://arxiv.org/abs/2107.12808. - Bauer et al. [2023] Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Satinder Singh, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M Zhang. Human-timescale adaptation in an open-ended task space. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1887–1935. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/bauer23a.html. - Tsamoura et al. [2025] Efthymia Tsamoura, Kaifu Wang, and Dan Roth. Imbalances in neurosymbolic learning: Characterization and mitigating strategies. In Proceedings of the Thirty-Ninth Conference on Neural Information Processing Systems (NeurIPS), 2025. - Menon et al. [2021] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021. - Cao et al. [2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In NeurIPS, pages 1567–1578, 2019. - Wang et al. [2022] Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. In NeurIPS, 2022. - He and Garcia [2009] Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284, 2009. - Horn and Perona [2017] Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. CoRR, abs/1709.01450, 2017. - Cour et al. [2011] Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12:1501–1536, 2011. ISSN 1532-4435. - Srikumar and Roth [2023] Vivek Srikumar and Dan Roth. The integer linear programming inference cookbook. ArXiv, abs/2307.00171, 2023. URL https://api.semanticscholar.org/CorpusID:259316294. - Le et al. [2021] Khang Le, Huy Nguyen, Quang M Nguyen, Tung Pham, Hung Bui, and Nhat Ho. On robust optimal transport: Computational complexity and barycenter computation. In Advances in Neural Information Processing Systems, pages 21947–21959, 2021. - Muggleton and De Raedt [1994] Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19:629–679, 1994. - De Raedt and Kersting [2008] Luc De Raedt and Kristian Kersting. Probabilistic inductive logic programming. In Probabilistic inductive logic programming: theory and applications, pages 1–27. Springer, 2008. - Wüst et al. [2024] Antonia Wüst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Pix2code: Learning to compose neural visual concepts as programs. arXiv preprint arXiv:2402.08280, 2024. - Locatello et al. [2020a] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525–11538, 2020a. - Xiao et al. [2024] Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey. arXiv preprint arXiv:2412.20206, 2024. - Lippe et al. [2022] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557–13603. PMLR, 2022. - Lippe et al. [2023] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence, pages 1263–1273. PMLR, 2023. - Giunchiglia et al. [2023] Eleonora Giunchiglia, Mihaela Cătălina Stoian, Salman Khan, Fabio Cuzzolin, and Thomas Lukasiewicz. Road-r: the autonomous driving dataset with logical requirements. Machine Learning, 112(9):3261–3291, 2023. - van Krieken et al. [2024b] Emile van Krieken, Samy Badreddine, Robin Manhaeve, and Eleonora Giunchiglia. Uller: A unified language for learning and reasoning, 2024b. URL https://arxiv.org/abs/2405.00532. - Derkinderen et al. [2025b] Vincent Derkinderen, Robin Manhaeve, Rik Adriaensen, Lucas Van Praet, Lennert De Smet, Giuseppe Marra, and Luc De Raedt. The deeplog neurosymbolic machine, 2025b. URL https://arxiv.org/abs/2508.13697. - Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. - Hsu et al. [2025] Joy Hsu, Jiayuan Mao, Joshua B Tenenbaum, Noah D Goodman, and Jiajun Wu. What makes a maze look like a maze? International Conference on Learning Representations (ICLR), 2025. - Hsu et al. [2023] Joy Hsu, Jiayuan Mao, Josh Tenenbaum, and Jiajun Wu. What’s left? concept grounding with logic-enhanced foundation models. Advances in Neural Information Processing Systems, 36:38798–38814, 2023. - Jiang et al. [2024] Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang, Weijie J Su, Camillo Jose Taylor, and Tanwi Mallick. Multi-modal and multi-agent systems meet rationality: A survey. In ICML 2024 Workshop on LLMs and Cognition, 2024. URL https://openreview.net/forum?id=9Rtm2gAVjo. - Pavlick [2023] Ellie Pavlick. Symbols and grounding in large language models. Philosophical Transactions of the Royal Society A, 381(2251):20220041, 2023. - Levine [2025] Sergey Levine. Language models in plato’s cave, 2025. URL https://sergeylevine.substack.com/p/language-models-in-platos-cave. Accessed: 2025-10-02. - Barsalou [2020] Lawrence W Barsalou. Challenges and opportunities for grounding cognition. Journal of Cognition, 3(1):31, 2020. - Barsalou [1999] Lawrence W. Barsalou. Perceptual symbol systems. Behavioral and Brain Sciences, 22(4):577–660, 1999. doi: 10.1017/S0140525X99002149. - Dove [2024] Guy Dove. Symbol ungrounding: what the successes (and failures) of large language models reveal about human cognition. Philosophical Transactions B, 379(1911):20230149, 2024. - Steels et al. [2008] Luc Steels et al. The symbol grounding problem has been solved. so what’s next. Symbols and embodiment: Debates on meaning and cognition, pages 223–244, 2008. - Gubelmann [2024] Reto Gubelmann. Pragmatic norms are all you need – why the symbol grounding problem does not apply to LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11663–11678, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.651. URL https://aclanthology.org/2024.emnlp-main.651/. - Mollo and Millière [2023] Dimitri Coelho Mollo and Raphaël Millière. The vector grounding problem. arXiv preprint arXiv:2304.01481, 2023. - Stahl [1993] Irene Stahl. Predicate invention in ilp—an overview. In European conference on machine learning, pages 311–322. Springer, 1993. - Tarzariol et al. [2022] Alice Tarzariol, Martin Gebser, and Konstantin Schekotihin. Lifting symmetry breaking constraints with inductive logic programming. Machine Learning, 111(4):1303–1326, 2022. - Quinlan [1990] J. Ross Quinlan. Learning logical definitions from relations. Machine learning, 5(3):239–266, 1990. - Muggleton [1995] Stephen Muggleton. Inverse entailment and progol. New generation computing, 13(3):245–286, 1995. - Srinivasan [2001] Ashwin Srinivasan. The aleph manual. 2001. - Cropper and Muggleton [2016] Andrew Cropper and Stephen H Muggleton. Metagol system, 2016. - Sakallah [2021] Karem A Sakallah. Symmetry and satisfiability. In Handbook of Satisfiability, pages 509–570. IOS press, 2021. - Anders et al. [2024] Markus Anders, Sofia Brenner, and Gaurav Rattan. Satsuma: Structure-based symmetry breaking in sat. arXiv preprint arXiv:2406.13557, 2024. - Ulyantsev et al. [2016] Vladimir Ulyantsev, Ilya Zakirzyanov, and Anatoly Shalyto. Symmetry breaking predicates for sat-based dfa identification. arXiv preprint arXiv:1602.05028, 2016. - Bogaerts et al. [2022] Bart Bogaerts, Stephan Gocht, Ciaran McCreesh, and Jakob Nordström. Certified symmetry and dominance breaking for combinatorial optimisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3698–3707, 2022. - Geirhos et al. [2020] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. - Ye et al. [2024] Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprint arXiv:2402.12715, 2024. - Steinmann et al. [2024] David Steinmann, Felix Divo, Maurice Kraus, Antonia Wüst, Lukas Struppek, Felix Friedrich, and Kristian Kersting. Navigating shortcuts, spurious correlations, and confounders: From origins via detection to mitigation. arXiv preprint arXiv:2412.05152, 2024. - Margeloiu et al. [2021] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289, 2021. - Mahinpei et al. [2021] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. In International Conference on Machine Learning: Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI, volume 1, pages 1–13, 2021. - Havasi et al. [2022] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. 2022. - Raman et al. [2023] Naveen Raman, Mateo Espinosa Zarlenga, Juyeon Heo, and Mateja Jamnik. Do concept bottleneck models obey locality? In XAI in Action: Past, Present, and Future Applications, 2023. - Stammer et al. [2021] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3619–3629, June 2021. - Bontempelli et al. [2021] Andrea Bontempelli, Fausto Giunchiglia, Andrea Passerini, and Stefano Teso. Toward a Unified Framework for Debugging Gray-box Models. In The AAAI-22 Workshop on Interactive Machine Learning, 2021. - Teso et al. [2021] Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive Label Cleaning with Example-based Explanations. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021. - Bontempelli et al. [2023] Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. Concept-level debugging of part-prototype networks. In International Conference on Learning Representations, 2023. - Hyvärinen et al. [2009] Aapo Hyvärinen, Jarmo Hurri, and Patrik O. Hoyer. Independent Component Analysis, pages 151–175. Springer London, London, 2009. ISBN 978-1-84882-491-1. doi: 10.1007/978-1-84882-491-1˙7. URL https://doi.org/10.1007/978-1-84882-491-1_7. - Schölkopf et al. [2021] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021. - Buchholz et al. [2022] Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems, 35:16946–16961, 2022. - Gresele [2023] Luigi Gresele. Learning Identifiable Representations: Independent Influences and Multiple Views. PhD thesis, Eberhard Karls Universität Tübingen Tübingen, 2023. - Hyvarinen et al. [2019] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd international conference on artificial intelligence and statistics, pages 859–868. PMLR, 2019. - Khemakhem et al. [2020] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In AISTATS, 2020. - Lachapelle et al. [2022] Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pages 428–484. PMLR, 2022. - Moran et al. [2021] Gemma E Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable deep generative models via sparse decoding. arXiv preprint arXiv:2110.10804, 2021. - Locatello et al. [2020b] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International conference on machine learning, pages 6348–6359. PMLR, 2020b. - Gresele et al. [2020] Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pages 217–227. PMLR, 2020. - Gresele et al. [2021] Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems, 34:28233–28248, 2021. - Von Kügelgen et al. [2021] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021. - Buchholz et al. [2023] Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. Advances in Neural Information Processing Systems, 36:45419–45462, 2023. - Ahuja et al. [2023] Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International conference on machine learning, pages 372–407. PMLR, 2023. - von Kügelgen et al. [2024] Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2024. - Fokkema et al. [2025] Hidde Fokkema, Tim van Erven, and Sara Magliacane. Sample-efficient learning of concepts with theoretical guarantees: from data to concepts without interventions. arXiv preprint arXiv:2502.06536, 2025. - Reizinger et al. [2024] Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-entropy is all you need to invert the data generating process. arXiv preprint arXiv:2410.21869, 2024. - Lachapelle et al. [2023] Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning. In International Conference on Machine Learning, pages 18171–18206. PMLR, 2023. - Fumero et al. [2023] Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36:27682–27698, 2023. - Zimmermann et al. [2021] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International conference on machine learning, pages 12979–12990. PMLR, 2021. - Roeder et al. [2021] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pages 9030–9039. PMLR, 2021. - Marconato et al. [2025] Emanuele Marconato, Sebastien Lachapelle, Sebastian Weichwald, and Luigi Gresele. All or none: Identifiable linear properties of next-token predictors in language modeling. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL https://openreview.net/forum?id=XCmIlemQP5. - Nielsen et al. [2024] Beatrix Miranda Ginn Nielsen, Luigi Gresele, and Andrea Dittadi. Challenges in explaining representational similarity through identifiability. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024. - Nielsen et al. [2025] Beatrix M. G. Nielsen, Emanuele Marconato, Andrea Dittadi, and Luigi Gresele. When does closeness in distribution imply representational similarity? an identifiability perspective, 2025. URL https://arxiv.org/abs/2506.03784.