## Behavioural Models and Equations ### Overview The image presents three different behavioral models: Reactive, Sentient, and Intentional. Each model is described with a brief explanation, a mathematical equation, and a further description. The image also includes equations for KL control as inference and expected free energy for POMDPs. ### Components/Axes * **Reactive Behaviour:** * Description: Actions are selected in response to an observed state. * Equation: P(u) = σ(Q | s\_τ) * Example: Q-learning table with states (s1, s2, s3, s4) and actions (up, right, down, left). * **Sentient Behaviour:** * Description: Action selection based on the inferred consequences of action. * Equation: P(u) = σ(-G) * Further Description: Planning as inference under objective constraints or preferences over outcomes. * **Intentional Behaviour:** * Description: Action selection constrained by intended endpoint or goal. * Equation: P(u) = σ(-G - H) * Further Description: Inductive Planning under subjective constraints or preferences over latent states. * **KL (risk sensitive) control as inference:** * Equation: Q(u) = D\_KL [Q(s\_{τ+1} | u) || P(s\_{τ+1} | c)] * Label: Risk * Description: Equivalent to active inference for MDPs * **Expected Free Energy for POMDPs:** * Equation: G(u) = D\_KL [Q(o\_{τ+1} | u) || P(o\_{τ+1} | c)] - E\_{Q\_u} [ln Q(o\_{τ+1} | s\_{τ+1}, u)] * Label: Risk * Label: Ambiguity ### Detailed Analysis or ### Content Details * **Reactive Behaviour - Q-learning Example:** * The Q-learning example is presented as a table. The rows represent states (s1, s2, s3, s4), and the columns represent actions (up, right, down, left). The values in the table represent Q-values. * s1: Up (1.2), Right (0.1), Down (0.0), Left (0.1) * s2: Up (1.1), Right (0.2), Down (0.1), Left (2.4) * s3: Up (0.0), Right (3.3), Down (0.9), Left (0.1) * s4: Up (1.8), Right (0.7), Down (0.3), Left (0.9) ### Key Observations * The image presents a hierarchy of behavioral models, starting from simple reactive behavior to more complex intentional behavior. * Each model is associated with a mathematical equation that captures the underlying principle. * The Q-learning example provides a concrete illustration of reactive behavior. * The equations for KL control and expected free energy provide a more detailed mathematical formulation of the models. ### Interpretation The image provides a conceptual framework for understanding different types of behavior. The models are presented in increasing order of complexity, reflecting the increasing role of internal representations and goals in shaping behavior. The mathematical equations provide a formal language for describing these models, while the Q-learning example provides a concrete illustration of how reactive behavior can be implemented. The KL control and expected free energy equations suggest a deeper connection between these behavioral models and concepts from information theory and decision theory. The image suggests that behavior can be understood as a process of inference and decision-making, where agents use their internal models of the world to select actions that maximize their expected reward or minimize their expected cost.

## Diagram: Behavioral Frameworks in Reinforcement Learning ### Overview The image presents a comparative diagram outlining three behavioral frameworks in reinforcement learning: Reactive Behaviour, Sentient Behaviour, and Intentional Behaviour. Each framework is defined by its action selection process and associated mathematical formulation. The diagram also includes examples of Q-learning and related equations for KL (risk sensitive) control and Expected Free Energy. ### Components/Axes The diagram is structured into three main columns, each representing a behavioral framework. Each column contains: * **Title:** Indicating the type of behavior (Reactive, Sentient, Intentional). * **Definition:** A textual description of the action selection process. * **Mathematical Formulation:** An equation representing the framework. * **Example/Extension:** Further elaboration or a related equation. The bottom of the diagram includes a note stating equivalence to active inference for MDPs. ### Detailed Analysis or Content Details **1. Reactive Behaviour (Left Column)** * **Definition:** "Actions are selected in response to an observed state." * **Equation:** `P(u) = σ(Q | sₜ)` * **Example: Q-learning** * A 2x2 table is presented with rows labeled `s₁`, `s₂`, `s₃`, `s₄` and columns labeled with symbols representing actions (up, right, down, left). * The table contains the following approximate values: * `s₁`: 1.2, 0.1, 0.0, 0.1 * `s₂`: 1.1, 0.2, 2.4, 0.1 * `s₃`: 0.0, 3.3, 0.9, 0.1 * `s₄`: 1.8, 0.7, 0.9, 0.9 * **Equation:** `Q(u) = DKL [Q(sₜ₊₁ | u) || P(sₜ₊₁ | c)]` * **Label:** "Risk" is placed under the equation. * **Note:** "Equivalent to active inference for MDPs" **2. Sentient Behaviour (Center Column)** * **Definition:** "Action selection based on the inferred consequences of action." * **Equation:** `P(u) = σ(–G)` * **Extension:** "Planning as inference under objective constraints or preferences over outcomes." * **Equation:** `G(u) = DKL [Q(oₜ₊₁ | u) || P(oₜ₊₁ | c)] – E Qω [ln Q(oₜ₊₁ | sₜ₊₁ , u)]` * **Labels:** "Risk" and "Ambiguity" are placed under the equation. **3. Intentional Behaviour (Right Column)** * **Definition:** "Action selection constrained by intended endpoint or goal." * **Equation:** `P(u) = σ(–G – H)` * **Extension:** "Inductive Planning under subjective constraints or preferences over latent states." ### Key Observations * The diagram presents a progression from simpler (Reactive) to more complex (Intentional) behavioral frameworks. * Each framework builds upon the previous one, adding layers of inference and constraint. * The use of the Kullback-Leibler divergence (`DKL`) in the equations suggests a focus on information-theoretic approaches to reinforcement learning. * The Q-learning example provides concrete values for a simple state-action space. * The inclusion of "Risk" and "Ambiguity" as separate components in the Sentient Behaviour framework highlights the importance of uncertainty in decision-making. ### Interpretation The diagram illustrates a hierarchical view of behavior in reinforcement learning agents. Reactive behavior represents the most basic level, where actions are directly determined by the current state. Sentient behavior introduces the ability to anticipate consequences, while Intentional behavior adds goal-directedness. The mathematical formulations provide a formal way to represent these different levels of behavior. The Q-learning example demonstrates how the reactive framework can be implemented in practice. The equations for KL control and Expected Free Energy suggest ways to extend the framework to handle risk and ambiguity. The overall message is that intelligent behavior requires a combination of reactivity, anticipation, and goal-directedness, and that these different aspects can be formally modeled using probabilistic and information-theoretic tools. The diagram suggests a progression in complexity, with each level building upon the previous one to create more sophisticated and adaptive agents. The final note about equivalence to active inference suggests a broader theoretical connection to Bayesian brain theory.

\n ## Diagram: Comparison of Behavioural Frameworks in Active Inference ### Overview The image is a technical diagram comparing three paradigms of action selection within the framework of active inference and control as inference: **Reactive Behaviour**, **Sentient Behaviour**, and **Intentional Behaviour**. It presents their core definitions, mathematical formulations, and associated concepts. The diagram is structured into three vertical columns for the behaviour types, with a shared foundational section at the bottom. ### Components/Axes The diagram is organized into three primary vertical panels, each with a light blue background, and a white background section at the bottom. **1. Left Panel: Reactive Behaviour** * **Title:** "Reactive Behaviour" * **Description:** "Actions are selected in response to an observed state" * **Core Formula:** `P(u) = σ(Q | s_τ)` * **Examples Section:** Contains a sub-section titled "Q-learning" with a 4x4 data table. * **Additional Concept:** "KL (risk sensitive) **control as inference**" with the formula `Q(u) = D_KL [Q(s_{τ+1} | u) || P(s_{τ+1} | c)]`. The term `D_KL [...]` is underbraced and labeled "Risk". * **Footer Note:** "Equivalent to active inference for MDPs" **2. Middle Panel: Sentient Behaviour** * **Title:** "Sentient Behaviour" * **Description:** "Action selection based on the inferred consequences of action" * **Core Formula:** `P(u) = σ(-G)` * **Associated Concept:** "**Planning as inference** under objective constraints or preferences over *outcomes*" **3. Right Panel: Intentional Behaviour** * **Title:** "Intentional Behaviour" * **Description:** "Action selection constrained by intended endpoint or goal" * **Core Formula:** `P(u) = σ(-G - H)` * **Associated Concept:** "**Inductive Planning** under subjective constraints or preferences over *latent states*" **4. Bottom Section (Spanning Right Side):** * **Title:** "Expected Free Energy for POMDPs" * **Formula:** `G(u) = D_KL [Q(o_{τ+1} | u) || P(o_{τ+1} | c)] - E_{Q_u} [ln Q(o_{τ+1} | s_{τ+1}, u)]` * The first term `D_KL [...]` is underbraced and labeled "Risk". * The second term `- E_{Q_u} [...]` is underbraced and labeled "Ambiguity". ### Detailed Analysis **Q-Learning Table (Reactive Behaviour Example):** The table is a 4x4 matrix with rows labeled `s₁` to `s₄` (states) and columns indicated by directional arrow icons (↑, →, ↓, ←) representing actions. | State | ↑ | → | ↓ | ← | |-------|---|---|---|---| | s₁ | 1.2 | 0.1 | 0.0 | 0.1 | | s₂ | 1.1 | 0.2 | 0.1 | 2.4 | | s₃ | 0.0 | 3.3 | 0.9 | 0.1 | | s₄ | 1.8 | 0.7 | 0.3 | 0.9 | These values represent Q-values (expected rewards) for taking a specific action in a given state. **Mathematical Formulations:** * `σ` denotes a softmax function, converting values into a probability distribution over actions `u`. * `Q` in the Reactive formula represents the Q-value function. * `G` represents Expected Free Energy, decomposed into Risk and Ambiguity terms in the bottom formula. * `H` in the Intentional formula represents an additional term for subjective constraints or preferences over latent states. * `D_KL` denotes the Kullback-Leibler divergence. * `Q(s_{τ+1} | u)` and `P(s_{τ+1} | c)` represent posterior and prior distributions over next states, respectively. * `Q(o_{τ+1} | u)` and `P(o_{τ+1} | c)` represent posterior and prior distributions over next observations. * `E_{Q_u}[...]` denotes an expectation taken with respect to the distribution `Q_u`. ### Key Observations 1. **Progressive Complexity:** The three behaviour types show a clear progression in the complexity of the action selection rule: from `P(u) = σ(Q)` (Reactive), to `P(u) = σ(-G)` (Sentient), to `P(u) = σ(-G - H)` (Intentional). 2. **Unifying Framework:** All three paradigms are framed within "control as inference," where choosing an action is treated as probabilistic inference. 3. **Risk and Ambiguity:** The decomposition of Expected Free Energy (`G`) into "Risk" (divergence from preferred outcomes) and "Ambiguity" (information gain) is explicitly highlighted as foundational for planning in Partially Observable MDPs (POMDPs). 4. **Conceptual Mapping:** The diagram maps well-known algorithms to these paradigms: Q-learning is an example of Reactive Behaviour, "Planning as inference" aligns with Sentient Behaviour, and "Inductive Planning" aligns with Intentional Behaviour. ### Interpretation This diagram serves as a conceptual taxonomy for understanding different levels of cognitive sophistication in artificial agents, grounded in the mathematics of active inference. * **Reactive Behaviour** represents a stimulus-response mechanism. The agent acts based on cached values (`Q`) associated with the current state (`s_τ`), without explicitly simulating future consequences. The link to "KL control as inference" and "risk" suggests this can be viewed as minimizing a divergence from a desired state distribution, but in a myopic, state-conditioned way. It's equivalent to solving fully observable Markov Decision Processes (MDPs). * **Sentient Behaviour** introduces foresight. Action selection is guided by minimizing Expected Free Energy (`G`), which involves evaluating the *inferred consequences* of actions. This corresponds to "planning as inference" where the agent infers the most likely actions to achieve preferred *outcomes* (`o`). This is suitable for partially observable environments where the agent must reason about future observations. * **Intentional Behaviour** adds a layer of subjective preference or goal-directedness. The additional term `-H` modifies the planning objective to incorporate preferences over *latent states* (`s`), not just observable outcomes. This suggests a form of "inductive planning" where the agent has an internal model or intention (a preferred trajectory in state-space) that guides action selection beyond mere outcome preferences. **Underlying Message:** The diagram argues that complex, goal-directed (intentional) behaviour can be derived as an extension of simpler reactive and sentient mechanisms, all within a unified probabilistic framework. The progression from Q-values to Expected Free Energy to an augmented Free Energy (`G+H`) illustrates how increasingly abstract internal models (of states, outcomes, and latent preferences) enable more sophisticated planning. The explicit breakdown of `G` into Risk and Ambiguity underscores the dual objective in active inference: achieving goals (exploitation) and reducing uncertainty (exploration).

## Diagram Type: Conceptual Diagram ### Overview The diagram illustrates the concept of different types of behavioral strategies in decision-making processes, specifically in the context of reinforcement learning and decision theory. It is divided into three main sections, each representing a different type of behavior: Reactive Behavior, Sentient Behavior, and Intentional Behavior. Additionally, there is a section on Q-learning and a formula for Expected Free Energy in Partially Observable Markov Decision Processes (POMDPs). ### Components/Axes - **Reactive Behavior**: This section explains that actions are selected in response to an observed state. It includes a formula for the probability of an action \( P(u) \) given a state \( s \) and a policy \( \sigma \). - **Sentient Behavior**: This section describes action selection based on the inferred consequences of an action. It includes a formula for the probability of an action \( P(u) \) given a state \( s \) and a policy \( \sigma \). - **Intentional Behavior**: This section explains that action selection is constrained by an intended endpoint or goal. It includes a formula for the probability of an action \( P(u) \) given a state \( s \) and a policy \( \sigma \). - **Q-learning**: This section provides an example of Q-learning, a type of reinforcement learning algorithm. - **Expected Free Energy for POMDPs**: This section includes a formula for the expected free energy in POMDPs, which is a measure of the uncertainty in the decision-making process. ### Detailed Analysis or ### Content Details - **Reactive Behavior**: The formula \( P(u) = \sigma(Q(s | s_t)) \) indicates that the probability of an action \( u \) is determined by the policy \( \sigma \) and the Q-value \( Q(s | s_t) \), which represents the expected return from taking action \( u \) in state \( s \) given the previous state \( s_t \). - **Sentient Behavior**: The formula \( P(u) = \sigma(-G) \) suggests that the probability of an action \( u \) is determined by the policy \( \sigma \) and the negative of the expected return \( G \), which represents the expected cost or loss from taking action \( u \). - **Intentional Behavior**: The formula \( P(u) = \sigma(-G - H) \) indicates that the probability of an action \( u \) is determined by the policy \( \sigma \) and the negative of the sum of the expected return \( G \) and the expected cost \( H \), which represents the total expected cost or loss from taking action \( u \). - **Q-learning**: The example of Q-learning shows a table with values for different states and actions, indicating the expected return for each combination. - **Expected Free Energy for POMDPs**: The formula includes terms for the Q-value, the policy, and the uncertainty in the decision-making process, represented by the KL-divergence term. ### Key Observations - The diagram highlights the differences between reactive, sentient, and intentional behavior in decision-making processes. - It emphasizes the importance of considering both the expected return and the expected cost or loss when selecting actions. - The inclusion of Q-learning and the formula for Expected Free Energy in POMDPs suggests a focus on reinforcement learning and decision theory in the context of complex decision-making processes. ### Interpretation The diagram provides a conceptual framework for understanding different types of behavioral strategies in decision-making processes. It emphasizes the importance of considering both the expected return and the expected cost or loss when selecting actions. The inclusion of Q-learning and the formula for Expected Free Energy in POMDPs suggests a focus on reinforcement learning and decision theory in the context of complex decision-making processes. The diagram can be used to guide research and development in areas such as artificial intelligence, robotics, and operations research.

## Conceptual Diagram: Behavioural Decision-Making Frameworks ### Overview The image presents a comparative framework of three behavioural paradigms (Reactive, Sentient, Intentional) and two decision-making principles (KL control, Expected Free Energy for POMDPs). It combines textual descriptions, mathematical formulas, and a Q-learning example table. ### Components/Axes 1. **Left Panel (Reactive Behaviour)** - Title: "Reactive Behaviour" - Description: "Actions are selected in response to an observed state" - Formula: $ P(u) = \sigma(\mathbf{Q} | s_\tau) $ - Example: Q-learning with a 4x4 state-value table (states $ s_1 $ to $ s_4 $, values 0.0–1.2) - Additional: "KL (risk sensitive) control as inference" with formula $ Q(u) = \underbrace{D_{KL}[Q(s_{\tau+1}|u)||P(s_{\tau+1}|c)]}_{\text{Risk}} $ 2. **Middle Panel (Sentient Behaviour)** - Title: "Sentient Behaviour" - Description: "Action selection based on the inferred consequences of action" - Formula: $ P(u) = \sigma(-\mathbf{G}) $ - Additional: "Planning as inference under objective constraints or preferences over outcomes" 3. **Right Panel (Intentional Behaviour)** - Title: "Intentional Behaviour" - Description: "Action selection constrained by intended endpoint or goal" - Formula: $ P(u) = \sigma(-\mathbf{G} - \mathbf{H}) $ - Additional: "Inductive Planning under subjective constraints or preferences over latent states" 4. **Bottom Section (Expected Free Energy for POMDPs)** - Title: "Expected Free Energy for POMDPs" - Formula: $ G(u) = \underbrace{D_{KL}[Q(o_{\tau+1}|u)||P(o_{\tau+1}|c)]}_{\text{Risk}} \underbrace{-\mathbb{E}_{Q_u}[\ln Q(o_{\tau+1}|s_{\tau+1},u)]}_{\text{Ambiguity}} $ ### Detailed Analysis - **Q-learning Table**: | State | 1.2 | 0.1 | 0.0 | 0.1 | |-------|-----|-----|-----|-----| | $ s_1 $ | 1.1 | 0.2 | 0.1 | 2.4 | | $ s_2 $ | 0.0 | 3.3 | 0.9 | 0.1 | | $ s_3 $ | 1.8 | 0.7 | 0.3 | 0.9 | - **Mathematical Notation**: - $ \sigma $: Sigmoid function (implied but not explicitly labeled) - $ D_{KL} $: Kullback-Leibler divergence (risk term) - $ \mathbb{E} $: Expected value (ambiguity term) ### Key Observations 1. **Hierarchical Structure**: - Reactive < Sentient < Intentional (increasing complexity of constraints) - KL control and Expected Free Energy represent complementary decision-making principles. 2. **Contrast in Constraints**: - Reactive: Observed states only - Sentient: Objective consequences - Intentional: Subjective latent states 3. **Mathematical Relationships**: - Risk (KL divergence) and Ambiguity (negative log-probability) are combined additively in Expected Free Energy. ### Interpretation This diagram illustrates a theoretical taxonomy of decision-making systems, progressing from simple reactive responses to complex goal-directed planning. The inclusion of KL divergence and Expected Free Energy suggests a Bayesian framework for handling uncertainty, where: - **Risk** quantifies distributional mismatch between predictions and outcomes - **Ambiguity** measures epistemic uncertainty in latent states The Q-learning example grounds the theory in reinforcement learning, while the formulas formalize the transition from reactive policies to intentional planning. The absence of explicit numerical trends implies a focus on conceptual relationships rather than empirical data.