## System Architecture Diagram: Visual Reasoning Model with Perception Frontend and Reasoning Backend ### Overview The image is a technical diagram illustrating the architecture of a machine learning model designed for visual reasoning tasks. It is divided into three main panels labeled **a**, **b**, and **c**, each detailing different aspects of the system. The overall flow moves from processing raw visual inputs (perception) to abstract reasoning and answer selection. The diagram uses a combination of block diagrams, mathematical notation, and flow arrows to depict data transformation and logical processes. ### Components/Axes The diagram is segmented into three primary regions: 1. **Panel a (Top):** High-level overview of the complete pipeline, split into a "Perception Frontend" (left) and a "Reasoning Backend" (right). 2. **Panel b (Middle):** A detailed view of the feature disentanglement and attribute representation mechanism, bridging the frontend and backend. 3. **Panel c (Bottom):** A detailed breakdown of the "Rule abduction" and "Rule execution" processes within the Reasoning Backend, separated into numerical and logical rule pathways. **Key Labels and Notation:** * **Inputs:** `Contexts` (denoted as `X^(1,1)`, `X^(1,2)`, ..., `X^(3,2)`) and `Candidates` (`X^(1)` ... `X^(8)`). * **Core Functions:** `f_θ` (Perception Frontend function), `f_Num^φ` (Numerical rule function), `f_Lgc^φ` (Logical rule function). * **Intermediate Representations:** `Ŝ` (SHDR for image panel), `Object_j` with attributes `Type`, `Size`, `Color`, `Num`, `Pos` (collectively `v`), `Attr_k` (attributes across panels). * **Rule Components:** `Rule abduction`, `Rule execution`, `Answer selection`. * **Codebooks:** `Frontend Codebook (C^Front)`, `Backend Codebook (C^Back)`. * **Attribute Vectors:** `v_type`, `v_size`, `v_color`, `v_number`, `v_position`, `c_position`. * **Rule Types:** `2-ary Relation`, `3-ary Relation`. * **Rule Sets:** `R_Num,2`, `R_Num,3`, `R_Lgc,3`. * **Operations:** `Query`, `attention`, `argmax`, `Π sim(·,·)` (product of similarities). * **Outputs:** `x^(ŷ)` (selected answer). ### Detailed Analysis **Panel a: High-Level Pipeline** * **Perception Frontend (Left):** Takes multiple context images and candidate images as input. These are processed by a function `f_θ` to produce `Ŝ` (SHDR for image panel). This leads to "Feature Disentanglement," where objects (`Object_j`, `Object_j+1`) are identified and their attributes (Type, Size, Color, Num, Pos) are extracted into a vector `v`. * **Reasoning Backend (Right):** Receives attributes `v` from "very panels" (likely meaning "across panels"). It operates on an attribute set `Attr_k`, which undergoes "Rule abduction" to infer a rule, followed by "Rule execution" to produce a new attribute `v^(3,3)`. This process iterates (`Attr_k-1`, `Attr_k`, `Attr_k+1`). Finally, "Answer selection" uses the processed attributes (`v^(1)`, `v^(2)`, ..., `v^(8)`) to choose the correct candidate `x^(ŷ)`. **Panel b: Attribute Disentanglement and Representation** * This panel details the transition from the perception frontend to the reasoning backend. * The SHDR `Ŝ` is used to generate an object representation `Ô_j` (with a null symbol `Ø` indicating a slot or placeholder). * This representation is decomposed into disentangled attribute vectors: `v̂_type`, `v̂_size`, `v̂_color`, `v̂_exist`. * These vectors, along with a `Frontend Codebook (C^Front)`, are fed into a "Query" and "attention" mechanism. The attention mechanism also uses a `Backend Codebook (C^Back)`. * The output is a set of "HD attribute representations to backend": `v_type`, `v_size`, `v_color`, `v_number`, `v_position`, `c_position`. **Panel c: Rule Abduction and Execution** * This panel is split vertically into **Numerical rules** (left) and **Logical rules** (right). * **Rule Abduction (Top Half):** * **Numerical Path:** Attribute vectors `v` are input to `f_Num^φ`, producing operator predictions `ÔP_1:M`. These are applied to attribute sets (`V_i^2`, `V_i^3`) within rule modules `R_Num,2` and `R_Num,3` to generate rule hypotheses `r̂_i-1`, `r̂_i`, `r̂_i+1`. A similarity product `Π sim(·,·)` computes unnormalized rule probabilities `s_Num^2` and `s_Num^3`. An `argmax` operation selects the best rule, defined as `[ÔP_1:M, r̄]`. * **Logical Path:** A parallel process uses `f_Lgc^φ` to generate rules for 3-ary relations, resulting in rule `[ÔP_1:M, r̄]` and probability `s_Lgc^3`. * **Rule Execution (Bottom Half):** * The selected rule `[ÔP_1:M, r̄]` is applied to input attributes (e.g., `v^(3,1)`, `v^(3,2)`). * For numerical rules, this occurs in module `R_Num^-1`, outputting `v^(3,3)`. * For logical rules, this occurs in module `R_Lgc^1`, outputting `v^(3,3)`. ### Key Observations 1. **Hierarchical Abstraction:** The system moves from concrete pixel data (`Contexts`, `Candidates`) to disentangled object attributes, and finally to abstract relational rules. 2. **Dual Rule Processing:** The model explicitly separates and processes numerical and logical rules, suggesting it handles different types of reasoning (e.g., arithmetic vs. comparative) with specialized pathways. 3. **Iterative Reasoning:** The notation `Attr_k-1`, `Attr_k`, `Attr_k+1` in Panel a implies the reasoning backend operates over multiple steps or layers of attribute abstraction. 4. **Codebook-Mediated Attention:** Panel b reveals that the translation from frontend features to backend-ready attributes is not direct but is mediated by learned codebooks and an attention mechanism, likely to align feature spaces. 5. **Rule Selection via Probability:** The abduction phase doesn't just generate rules; it scores them (`s_Num^2`, `s_Num^3`, `s_Lgc^3`) using a similarity metric, and selects the most probable one via `argmax`. ### Interpretation This diagram outlines a neuro-symbolic architecture for visual question answering or similar visual reasoning tasks. The **Perception Frontend** acts as a vision backbone that parses images into structured, object-centric representations (disentangled attributes). The **Reasoning Backend** is a symbolic or differentiable logic engine that manipulates these attributes using learned rules. The core innovation appears to be the **Rule Abduction** mechanism. Instead of having a fixed set of rules, the model *infers* the most applicable rule (numerical or logical, 2-ary or 3-ary) from the current context by scoring candidate rules. This makes the system more flexible and capable of handling novel reasoning problems. The separation into numerical and logical streams suggests an inductive bias built into the model to handle these fundamentally different types of relations efficiently. The flow from `v` (raw attributes) to `v̂` (disentangled attributes) to `v` (HD representations) to finally `v^(3,3)` (a predicted attribute after rule execution) shows a complete cycle of perception, reasoning, and prediction. The final "Answer selection" module likely compares the predicted outcome `v^(3,3)` or the state of all candidates against the expected answer format to choose the correct image candidate `x^(ŷ)`. This architecture aims to combine the pattern recognition strengths of neural networks with the interpretability and systematic generalization of symbolic reasoning.