# Planning to avoid ambiguous states through Gaussian approximations to non-linear sensors in active inference agents
**Authors**: Wouter M. Kouw
> 0000-0002-0547-4817
institutetext: Bayesian Intelligent Autonomous Systems laboratory TU Eindhoven, Eindhoven, Netherlands email: w.m.kouw@tue.nl
## Abstract
In nature, active inference agents must learn how observations of the world represent the state of the agent. In engineering, the physics behind sensors is often known reasonably accurately and measurement functions can be incorporated into generative models. When a measurement function is non-linear, the transformed variable is typically approximated with a Gaussian distribution to ensure tractable inference. We show that Gaussian approximations that are sensitive to the curvature of the measurement function, such as a second-order Taylor approximation, produce a state-dependent ambiguity term. This induces a preference over states, based on how accurately the state can be inferred from the observation. We demonstrate this preference with a robot navigation experiment where agents plan trajectories.
Keywords: Active inference Free energy minimization Bayesian filtering Non-linear sensing Control systems Planning Navigation
## 1 Introduction
In nature, intelligent agents build a model to infer the causes of their sensations [2]. In engineering, we are able to utilize knowledge of the relevant physics to structure such a model. In particular, we often know how sensors measure states of the world. For example, we know how radar measures relative velocity and distance [19]. Measurement functions that are non-linear transformations of state variables pose challenges to state estimation, which are often dealt with using Gaussian approximations of the transformed variables [8, 14]. We show that for certain Gaussian approximations, an active inference agent will prefer to avoid states because it already knows that state estimation will be difficult.
Active inference agents are based on free energy functionals that rank policies on explorative and goal-directed behaviour [5, 7, 6, 16]. The expected free energy functional can be understood through its decomposition into a cross-entropy term between states and observations given action ("ambiguity"), and a Kullback-Leibler divergence between the posterior predictive and a goal prior distribution ("risk") [7, 18, 4]. We show that Gaussian approximations of a non-linear observation function that are itself linear in the covariance matrix, e.g., first-order Taylor and the unscented transform [9], lead to ambiguity terms that are constant over states. This echoes an earlier finding that agents with a linear Gaussian state-space model exhibit a constant ambiguity term [10]. However, utilizing a second-order Taylor approximation induces a non-constant ambiguity term. Under this model, the agent will avoid states where the non-linear measurement function curves strongly. Our contributions are:
- Analysis of ambiguity in expected free energy functions under three different Gaussian approximations.
- An experiment where a robot must plan a trajectory and navigate to a goal prior distribution, testing the effect of the ambiguity term.
## 2 Problem statement
We want to plan a trajectory for a robot across a plane. The robot’s state at time $k$ is its planar position and time derivatives, $x_k∈ℝ^D_x$ . The robot does not sense position directly, but has to infer it from noisy measurements $y_k∈ℝ^D_y$ , produced by a sensor through a non-linear mapping $g:ℝ^D_x→ℝ^D_y$ and measurement noise $v_k∈R^D_y$ . It accepts control inputs $u_k∈ℝ^D_u$ and moves according to linear dynamics with a transition matrix $A∈ℝ^D_x× D_x$ , control matrix $B∈ℝ^D_x× D_u$ and process noise $e_k∈ℝ^D_x$ . Overall, we consider robot systems described with discrete-time state-space models of the form:
$$
\displaystyle x_k \displaystyle=Ax_k-1+Bu_k+e_k , \displaystyle e_k∼N(0,Q) , \displaystyle y_k \displaystyle=g(x_k)+v_k , \displaystyle v_k∼N(0,R) , \tag{1}
$$
where $Q,R$ are noise covariance matrices.
The goal is to find a sequence of $T$ controls $\bar{u}_k=u_k+1,\dots u_k+T$ that produces future states close to a desired state $x_*$ . Agents must plan every time-step. The challenge is that errors in state estimation may cause drastic changes in the planned trajectory, which can lead an agent astray.
#### 2.0.1 Example
Consider a robot with position and velocity states that must move from position $x_0=(0,-1)$ to $x_*=(0,1)$ . Its state transition, control and process noise covariance matrices are given by:
$$
\displaystyle A=\begin{bmatrix}1&0&Δ t&0\\
0&1&0&Δ t\\
0&0&1&0\\
0&0&0&1\end{bmatrix}, B=\begin{bmatrix}0&0\\
0&0\\
Δ t&0\\
0&Δ t\end{bmatrix}, Q=\begin{bmatrix}σ^2_1\frac{Δ t^3
}{3}&0&σ^2_1\frac{Δ t^2}{2}&0\\
0&σ^2_2\frac{Δ t^3}{3}&0&σ^2_2\frac{Δ t^2}{2}
\\
σ^2_1\frac{Δ t^2}{2}&0&σ^2_1Δ t&0\\
0&σ^2_2\frac{Δ t^2}{2}&0&σ^2_2Δ t\end{bmatrix}, \tag{3}
$$
for $Δ t=0.5$ , $σ_1=σ_2=0.1$ . Measurements are produced by a sensor station at $(0,0)$ that reports relative angle $φ_k∈[-π π]$ and relative distance $d_k∈[0,∞)$ . The mapping and measurement noise covariance matrix are:
$$
\displaystyle g(x_k)=\begin{bmatrix}φ_k\\
d_k\end{bmatrix}=\begin{bmatrix}√{x_1k^2+x_2k^2}\\
\arctan(x_1k,x_2k)\end{bmatrix} , R=\begin{bmatrix}ρ^2_1&0\\
0&ρ^2_2\end{bmatrix} , \tag{4}
$$
where $ρ_1=ρ_2=0.001$ . Suppose it uses an extended Kalman filter (first-order Taylor approximation) for state estimation and a finite-horizon model-predictive control objective of the form:
$$
\displaystyle J_k(\bar{u}_k)=∑_t=k+1^k+T\big{(}(A\hat{x}_t-1+
Bu_t)-x_*\big{)}^∫ercalC\big{(}(A\hat{x}_t-1+Bu_t)-x_*\big{
)}+η u_t^2 , \tag{5}
$$
where $\hat{x}$ is the mean state, $\hat{x}_t=A\hat{x}_t-1+Bu_t$ , $C$ is a cost matrix (ones for position, zeros for velocity) and $η$ a regularization parameter. Minimizing this objective every time-step produces the control sequence $\bar{u}_k^MPC=\arg\min J_k(\bar{u}_k)$ . Such an agent will first plan a trajectory moving directly forward, as described in Figure 1 (left). However, as it approaches the sensor station, its state estimate become progressively more inaccurate and it makes increasingly more drastic adjustments to the control plan (see $k=5$ in Figure 1 middle). Figure 1 (right) shows the executed trajectory over a trial of 10 steps, demonstrating that the agent lost track of the robot’s state and did not successfully reach the target.
<details>
<summary>extracted/5864169/figures/cart2polar-mpc-trajectory-k1.png Details</summary>
### Visual Description
## Diagram: Path Planning and Sensor Range Visualization
### Overview
This image is a 2D Cartesian coordinate plot illustrating a path planning scenario. It displays a sensor station at the origin, a state estimate at the bottom, a goal state at the top, and a series of planned states connecting them. Concentric circles centered at the origin likely represent sensor range, uncertainty, or signal strength zones.
### Components/Axes
* **X-axis:** Labeled "position x". Range from -2 to 2, with major ticks at -2, -1, 0, 1, 2.
* **Y-axis:** Labeled "position y". Range from -1.5 to 3, with major ticks at -1, 0, 1, 2, 3.
* **Legend (Top-Left):**
* Black triangle: "sensor station"
* Green circle: "start state"
* Red circle: "goal state"
* Blue line with blue circle: "system states"
* Purple circle: "state estimate"
* Orange line with orange circle: "planned states"
### Detailed Analysis
#### 1. Spatial Elements and Positioning
* **Sensor Station:** A black triangle is positioned exactly at the origin (0, 0).
* **State Estimate:** A purple circle is positioned at (0, -1).
* **Goal State:** A red circle is positioned at (0, 1).
* **Planned States:** An orange line connects the state estimate (0, -1) to the goal state (0, 1). Along this line, there are intermediate orange circles at approximately:
* (0, -0.75)
* (0, -0.25)
* (0, 0.25)
* (0, 0.5)
* (0, 0.75)
* **Concentric Circles:** Four concentric circles are centered at (0, 0).
* **Inner Circle:** Filled with a reddish-brown color, radius approximately 0.75.
* **Second Circle:** Filled with light orange, radius approximately 1.25.
* **Third Circle:** Filled with very light orange, radius approximately 2.0.
* **Outer Circle:** Outlined in faint grey, radius approximately 2.9.
#### 2. Trend Verification
* **Path Trend:** The "planned states" form a perfectly vertical, linear path along the Y-axis, moving from the state estimate at y = -1 to the goal state at y = 1.
* **Uncertainty/Range Trend:** The concentric circles expand outward from the sensor station at (0, 0), suggesting a gradient of influence or uncertainty that increases with distance from the origin.
### Key Observations
* **Legend Discrepancy:** There is a notable discrepancy between the legend and the plotted data. The legend includes "start state" (green) and "system states" (blue), but neither green nor blue elements are present in the plot.
* **Path Linearity:** The planned path is strictly linear, suggesting a direct movement vector toward the goal, passing directly through the sensor station.
* **Visual Hierarchy:** The "goal state" (red) and "state estimate" (purple) are larger than the intermediate "planned states" (orange), emphasizing the start and end points of the trajectory.
### Interpretation
This diagram represents a classic robotics or control theory problem: **Path Planning under Sensor Constraints.**
* **The Scenario:** An agent (represented by the state estimate) is attempting to navigate from a position at (0, -1) to a goal at (0, 1).
* **The Sensor:** The sensor station at (0, 0) is the focal point of the concentric circles. These circles likely represent a "sensor field" or "uncertainty zone." The fact that the path passes directly through the center of these circles suggests the agent is moving through the sensor's field of view or range.
* **The Missing Data:** The absence of the "start state" (green) and "system states" (blue) suggests this might be a snapshot of a specific phase of a simulation where only the *estimate* and the *plan* are relevant, or perhaps the "state estimate" has superseded the "start state."
* **Peircean Investigative Note:** The concentric circles are not uniform in color; the transition from reddish-brown to light orange suggests a decay in signal quality or an increase in positional uncertainty as the agent moves away from the sensor station. The path is direct, implying that the sensor station is either the target or a waypoint that must be traversed to reach the goal.
</details>
<details>
<summary>extracted/5864169/figures/cart2polar-mpc-trajectory-k5.png Details</summary>
### Visual Description
## [Chart/Diagram Type]: 2D Trajectory and State Estimation Plot
### Overview
This image is a 2D coordinate plot visualizing the trajectory of an autonomous system. It displays the system's movement from a starting position, through a series of state estimates, and along a planned path. The plot includes a fixed sensor station, a goal state, and concentric uncertainty ellipses (covariance regions) indicating the system's confidence in its position or the environment.
### Components/Axes
* **Axes:**
* **X-axis:** Labeled "position x", ranging from -2 to 2.
* **Y-axis:** Labeled "position y", ranging from -1.5 to 3.5.
* **Legend (Top-Left):**
* **Black triangle:** "sensor station"
* **Green circle:** "start state"
* **Red circle:** "goal state"
* **Blue line/circle:** "system states"
* **Purple circle:** "state estimate"
* **Orange line/circle:** "planned states"
### Detailed Analysis
The plot maps the following spatial data points (approximate coordinates):
* **Sensor Station (Black Triangle):** Located at the origin (0, 0).
* **Start State (Green Circle):** Located at (0, -1).
* **System States (Blue Line/Circles):**
* The path begins at the start state (0, -1).
* It moves upward and slightly right to approximately (0.1, -0.8).
* It continues to approximately (0.2, -0.4).
* **State Estimate (Purple Line/Circles):**
* The path continues from the last blue point (~0.2, -0.4).
* It moves to approximately (0.2, -0.2).
* **Goal State (Red Circle):** Located at approximately (0.35, 0.05).
* **Planned States (Orange Line/Circles):**
* The path begins near the goal state at approximately (0.2, 0.05).
* It moves upward to (0.2, 0.3).
* It continues upward to (0.1, 0.8).
* It terminates at approximately (0.1, 1.0).
* **Uncertainty Ellipses:**
* There are multiple concentric, semi-transparent beige/orange ellipses.
* These are centered roughly at (0, 1).
* The ellipses represent increasing levels of uncertainty, with the largest ellipse extending from roughly x = -1.5 to 1.5 and y = -0.5 to 3.0.
### Key Observations
* **Trajectory Flow:** The system moves from the bottom (start) toward the sensor station (origin), then pivots.
* **Divergence:** The "Planned States" (orange) move away from the "Goal State" (red) and head directly into the region of highest uncertainty (the center of the concentric ellipses).
* **Uncertainty Concentration:** The uncertainty ellipses are centered at (0, 1), which is directly "above" the sensor station. The planned path terminates exactly in the center of this high-uncertainty region.
* **Spatial Relationship:** The "Goal State" is positioned very close to the sensor station (0,0), while the "Planned States" move away from this goal toward the higher-uncertainty region.
### Interpretation
This diagram likely represents a path-planning algorithm in a robotics or control theory context, possibly involving **Active Perception** or **Exploration**.
* **The "Why":** The system does not simply move to the "Goal State" (red). Instead, it moves to the goal, then immediately plans a path (orange) that leads away from the goal and into the region of highest uncertainty (the center of the ellipses).
* **Peircean Investigative Reading:**
* The "Goal State" may be a temporary waypoint or a location where the system has already gathered sufficient information.
* The movement into the high-uncertainty region (the center of the ellipses) suggests the system is performing an "information-gathering" maneuver. It is intentionally moving into an area where its state estimate is less certain (or where the environment is less known) to reduce that uncertainty.
* The concentric ellipses represent the covariance of the system's position estimate; the system is prioritizing moving toward the center of this uncertainty, likely to improve its localization or map the environment in that specific area.
</details>
<details>
<summary>extracted/5864169/figures/cart2polar-mpc-trajectory-trial.png Details</summary>
### Visual Description
## 2D Scatter/Line Plot: State Estimation Trajectory
### Overview
This image displays a 2D Cartesian plot representing a state estimation scenario, likely from a control theory or robotics context (e.g., a Kalman Filter visualization). It compares the "system states" (the ground truth trajectory) against "state estimates" (the filter's belief). The plot includes a start state, a goal state, and uncertainty regions (covariance ellipses) surrounding the state estimates.
### Components/Axes
* **X-axis:** Labeled "position x". The scale ranges from -2 to 2, with major ticks at integer intervals.
* **Y-axis:** Labeled "position y". The scale ranges from -1 to 3, with major ticks at integer intervals.
* **Legend (Top-Left):**
* **Black triangle:** Labeled "sensor station" (Note: This element is absent from the plot area).
* **Green circle:** Labeled "start state".
* **Red circle:** Labeled "goal state".
* **Blue line with blue circles:** Labeled "system states".
* **Purple line with purple circles:** Labeled "state estimates".
* **Uncertainty Regions:** Large, semi-transparent, concentric purple circles centered on the purple "state estimates" markers.
### Detailed Analysis
#### 1. Start and Goal States
* **Start State:** Located at coordinates (0, -1). Marked with a green circle.
* **Goal State:** Located at coordinates (0, 1). Marked with a red circle.
#### 2. System States (Blue Line)
* **Trend:** The blue line represents the actual trajectory of the system. It originates at the start state (0, -1) and moves generally upward and to the left, diverging from the vertical axis.
* **Data Points (Approximate):**
* (0, -1) [Start]
* (-0.5, -0.8)
* (-0.6, -0.4)
* (-0.4, 0.4)
* (-0.5, 0.9)
* (-0.6, 1.4)
* (-0.6, 1.9) [End]
#### 3. State Estimates (Purple Line)
* **Trend:** The purple line represents the estimated trajectory. It originates at the start state (0, -1) and moves generally upward and slightly to the right, before oscillating and clustering near the center.
* **Data Points (Approximate):**
* (0, -1) [Start]
* (0, -0.8)
* (0, -0.6)
* (0, -0.2)
* (0, 0)
* (0.2, -0.2)
* (0.3, 0.1)
* (0.2, 0.1)
* (0.1, 0.3)
* (0.1, 0.4)
* (0.1, 0.7)
* (0, 0.7) [End]
#### 4. Uncertainty Regions
* The purple circles represent the uncertainty (covariance) of the state estimates.
* The circles are largest in the middle of the trajectory (around y=0 to y=0.5), indicating higher uncertainty in the estimate during that phase.
* The circles are smaller near the start state (0, -1) and the end of the purple path (0, 0.7), indicating higher confidence in the estimate at those points.
### Key Observations
* **Divergence:** There is a significant and increasing divergence between the "system states" (blue) and the "state estimates" (purple). The system moves into the top-left quadrant, while the estimates remain clustered near the center-top.
* **Goal Proximity:** The state estimates (purple) appear to be converging toward the "goal state" (red circle at 0, 1), whereas the actual system states (blue) are moving away from the goal toward the top-left.
* **Missing Data:** The legend includes a "sensor station" (black triangle), but no such marker is visible anywhere on the plot. This suggests either an error in the visualization generation or that the sensor station is located outside the plotted bounds.
### Interpretation
This visualization demonstrates a state estimation process where the filter is failing to accurately track the true system state.
* **Filter Performance:** The "state estimates" (purple) are clearly not tracking the "system states" (blue). The filter appears to be "attracted" to the goal state (red), or perhaps the system is moving in a way that the filter's internal model does not account for (e.g., unmodeled dynamics or external disturbances).
* **Uncertainty Analysis:** The large, overlapping purple circles indicate that the filter is aware of its high uncertainty during the middle of the trajectory. The fact that the uncertainty remains large even as the estimates diverge from the true path suggests the filter may be under-confident or that the measurement noise is high.
* **Peircean Investigative Note:** The absence of the "sensor station" is a critical anomaly. If the sensor station is the source of the data for the state estimates, its absence might explain the poor tracking performance—perhaps the sensor is out of range or the data is not being received, leading the filter to rely on a default model that drifts toward the goal.
</details>
Figure 1: (Left) Planned trajectory at $k=1$ , from start to goal directly over the sensor station. (Middle) Planned trajectory at $k=5$ showing a mismatch between true and estimated state resulting in a strong adjustment to the planned trajectory. (Right) Executed trajectory over a trial of $10$ steps demonstrates the agent losing track of the robot when it approaches the sensor station.
## 3 Agent specification
### 3.1 Probabilistic Model
The agent’s model will have Gaussian prior distributions over states and controls,
$$
\displaystyle p(x_0)=N(x_0\>|\>m_0,S_0) , N(
u_k\>|\>0,η^-1I) , \tag{6}
$$
with mean $m_0$ , covariance matrix $S_0$ , precision $η$ and identity matrix $I$ . The agent’s state transition will also be expressed as a Gaussian distribution:
$$
p(x_k\>|\>x_k-1,u_k)=N(x_k\>|\>Ax_k-1+Bu_k,Q) . \tag{7}
$$
Let the marginal state $x_k$ be Gaussian distributed, i.e., $p(x_k)=N(x_k\>|\>m_k,S_k)$ . We restrict our attention to approximations of the nonlinear sensor $g(x_k)$ that produce Gaussian joint distributions over states and observations [14], i.e.,
$$
\displaystyle p(y_k,x_k)≈N\Big{(}\begin{bmatrix}x_k\\
y_k\end{bmatrix}\>|\>\begin{bmatrix}m_k\\
μ_k\end{bmatrix},\begin{bmatrix}S_k&Γ_k\\
Γ_k^∫ercal&Σ_k\end{bmatrix}\Big{)} . \tag{8}
$$
From the joint, we obtain a conditional distribution of observations given states:
$$
\displaystyle p(y_k\>|\>x_k)≈N\big{(}y_k\>|\>μ_k+
Γ_k^∫ercalS_k^-1(x_k-m_k),Σ_k-Γ_kS_k^-1
Γ_k^∫ercal\big{)} . \tag{9}
$$
This distribution is linear in $x_k$ , and will allow for exact Bayesian filtering. But note that the parameters $μ_k$ , $Γ_k$ and $Σ_k$ may be nonlinear functions of $x_k$ , depending on the type of Gaussian approximation (specifics treated in Section 4), and may thus capture more of the effect of $g(x_k)$ .
### 3.2 Inferring states
We assume that, when inferring states, the agent has observed the system output $y_k=\hat{y}_k$ and input $u_k=\hat{u}_k$ . Let $D_k\triangleq\{\hat{y}_i,\hat{u}_i\}_i=1^k$ refer to data observed thus far. Given the known executed control, state estimation follows the general Bayesian filtering equations [14]. Firstly, the prior predictive distribution is given by:
$$
\displaystyle p(x_k| \hat{u}_k,D_k-1)=∫ p(x_k|x_
k-1,\hat{u}_k) p(x_k-1|D_k-1)
dx_k-1=N(x_k| \bar{m}_k,\bar{S}_k) . \tag{10}
$$
with $\bar{m}_k\triangleq Am_k-1+B\hat{u}_k$ and $\bar{S}_k\triangleq AS_k-1A^∫ercal+Q$ . This prediction is corrected by the observation through Bayes’ rule [14],
$$
\displaystyle p(x_k\>|\>D_k)=\frac{p(\hat{y}_k\>|\>x_k)}{p(
\hat{y}_k\>|\>D_k-1)}p(x_k\>|\>\hat{u}_k,D_k-1)=N(x_k\>|\>m_k,S_k) , \tag{11}
$$
with $m_k=\bar{m}_k+Γ_kΣ_k^-1(\hat{y}_k-μ_k)$ and $S_k=\bar{S}_k-Γ_kΣ_k^-1Γ_k$ .
### 3.3 Inferring controls
We will discuss the inference procedure first for a single step into the future, and then generalize to a finite horizon of length $T$ . Predictions for the future state and observation are made by unrolling the generative model to $t=k+1$ :
$$
\displaystyle p(y_t,x_t,u_t\>|\>D_k)=p(y_t\>|\>x_t)p(x_
t\>|\>u_t;D_k)p(u_t) . \tag{12}
$$
We will use an expected free energy functional to infer a posterior distribution over the control $u_t$ [13]:
$$
\displaystyleF_k[q]=∫ q(y_t\>|\>x_t)∫ q(x_t,u_t)\ln
\frac{q(x_t,u_t)}{p(y_t,x_t,u_t\>|\>D_k)}d(u_t
,x_t)dy_t . \tag{13}
$$
The variational model is specified to be:
$$
\displaystyle q(y_t\>|\>x_t)\triangleq p(y_t\>|\>x_t) , q(x_t,
u_t)\triangleq p(x_t\>|\>u_t;D_k)q(u_t) . \tag{14}
$$
Constraining $q(y_t\>|\>x_t)$ to the Gaussian approximation defined in Eq. 9 allows us to study deterministic approximations in an expected free energy minimization context. Given this variational model, Eq. 13 may be re-arranged to:
$$
\displaystyleF_k[q] \displaystyle=∫ p(y_t\>|\>x_t)∫ p(x_t\>|\>u_t;D_k)q(
u_t)\ln\frac{p(x_t\>|\>u_t;D_k)q(u_t)}{p(y_t,x_t,u_t
;D_k)}d(x_t,u_t)dy_t \displaystyle=∫q(u_t)\big{(}∫ p(y_t,x_t\>|\>u_t;D
_k)\ln\frac{p(x_t\>|\>u_t;D_k)q(u_t)}{p(y_t,x_t\>|\>u
_t;D_k)p(u_t)}d(y_t,x_t)\big{)}du_t \displaystyle=∫q(u_t)\big{(}\ln\frac{q(u_t)}{p(u_t)}+
\underbrace{∫p(y_t,x_t|u_t;D_k)\ln\frac{p(x_t\>|\>u
_t;D_k)}{p(y_t,x_t|u_t;D_k)}d(y_t,x
_t)}_J_k(u_t)\big{)}du_t. \tag{15}
$$
We refer to $J_k(u_t)$ as the expected free energy function as it depends on the value of $u_t$ not on its distribution. Under $J_k(u_t)=\ln(1/\exp(-J_k(u_t)))$ , the expected free energy functional can be concisely expressed as:
$$
\displaystyleF_k[q] \displaystyle=∫ q(u_t)\ln\frac{q(u_t)}{p(u_t)\exp≤ft(-J_
k(u_t)\right)}du_t . \tag{18}
$$
The above is a Kullback-Leibler divergence, which is minimal when
$$
\displaystyle q^*(u_t)∝ p(u_t)\exp≤ft(-J_k(u_t)
\right) . \tag{19}
$$
The proportionality is due to the implicit constraint A more rigorous treatment would define a Lagrangian with normalization and marginalization constraints [17]. However, such a treatment is inconsequential when resorting to MAP estimation, as will be pursued later in the paper. that $q^*(u_t)$ should integrate to $1$ . To work out the expectation in Eq. 17, we first decompose the joint over states and observations into
$$
\displaystyle p(y_t,x_t\>|\>u_t;D_k)=p(x_t\>|\>y_t,u_t
;D_k)p(y_t) , \tag{20}
$$
and then intervene on the marginal distribution over $y_t$ with a distribution reflecting desired future observations (a.k.a. goal prior) [11]:
$$
\displaystyle p(y_t)→ p(y_t\>|\>y_*)=N(y_t\>|\>μ
_*,Σ_*) . \tag{21}
$$
The next step involves applying Bayes’ rule in the inverse direction:
$$
\displaystyle\frac{1}{p(x_t\>|\>y_t,u_t;D_k)}=\frac{p(y_t
\>|\>u_t;D_k)}{p(y_t\>|\>x_t)p(x_t\>|\>u_t;D_
k)} , \tag{22}
$$
where the marginal prediction for the future observation is:
$$
\displaystyle p(y_t\>|\>u_t;D_k) \displaystyle=∫ p(y_t\>|\>x_t) p(x_t\>|\>u_t;D_k)
dx_t \displaystyle=∫N(\begin{bmatrix}x_t\\
y_t\end{bmatrix}\>|\>\begin{bmatrix}\bar{m}_t\\
μ_t\end{bmatrix},\begin{bmatrix}\bar{S_t}&Γ_t\\
Γ_t^∫ercal&Σ_t\end{bmatrix})dx_t=N(y_
t\>|\>μ_t,Σ_t) . \tag{23}
$$
Note that $μ_t$ and $Σ_t$ depend on $u_t$ through $\bar{m}_t$ . Plugging Eqs. 21 and 22 into Eq. 20 yields:
$$
\displaystyleJ_k(u_t) \displaystyle=∫p(y_t,x_t|u_t;D_k)\ln\frac{p(x_t|u_
t;D_k)}{p(y_t\>|\>y_*)}\frac{p(y_t\>|\>u_t;D_k
)}{p(y_t\>|\>x_t)p(x_t\>|\>u_t;D_k)} d(y_t,x_t) \displaystyle=\underbrace{∫p(y_t,x_t\>|\>u_t;D_k)\Big
{[}-\ln\frac{p(y_t,x_t\>|\>u_t;D_k)}{p(x_t\>|\>u_t;
D_k)}\Big{]}d(y_t,x_t)}_ambiguity \displaystyle +\underbrace{∫\Big{[}∫ p(y_t,x_t\>|\>u_t;
D_k)dx_t\Big{]}\ln\frac{p(y_t\>|\>u_t;D_
k)}{p(y_t\>|\>y_*)} dy_t}_risk. \tag{25}
$$
"Risk" refers to the Kullback-Leibler (KL) divergence between predicted and desired future observations. The inner integral in the risk term leads to a Gaussian distribution (Eq. 24) and the KL divergence between Gaussians is [3]:
$$
\displaystyleE_p(y_{t\>|\>u_t;D_k)}\Big{[}\ln\frac{p(
y_t\>|\>u_t;D_k)}{p(y_t\>|\>y_*)}\Big{]}=\frac{1}{2}\Big{(
}\ln\frac{|Σ_*|}{|Σ_t|}-D_y+tr\big{(}Σ_*^-1(
Σ_t+Ψ_*)\big{)}\Big{)} . \tag{27}
$$
where $Ψ_*\triangleq(μ_*-μ_t\big{)}\big{(}μ_*-μ_t)^∫ercal$ . "Ambiguity" refers to the conditional entropy of the future observations given the future states.
**Lemma 1**
*Ambiguity, as defined in Eq. 26, for a generative model described in Eq. 12 and a variational distribution described in Eq. 14, is:
$$
\displaystyleE_p(y_{t,x_t|u_t;D_k)}\big{[}-\ln
\frac{p(y_t,x_t|u_t;D_k)}{p(x_t|u_t;D_k)}
\big{]}=\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln|Σ_t-Γ^
∫ercal_t\bar{S}_t^-1Γ_t|. \tag{28}
$$*
The proof is in Appendix 0.A. Note that the first term does not depend on the state $x_t$ . Plugging Eqs. 27 and 28 into the expected free energy function (Eq. 26) produces:
$$
\displaystyleJ_k(u_t)=\frac{1}{2}\Big{(}\ln\frac{|Σ_*
|}{|Σ_t|}+D_y\ln(2π)+tr\big{(}Σ_*^-1(Σ
_t+Ψ_*)\big{)}+\ln\big{|}Σ_t-Γ^∫ercal_t
\bar{S}_t^-1Γ_t\big{|}\Big{)} . \tag{29}
$$
Note that $Γ_t$ , $Σ_t$ and $Ψ_*$ depend on $u_t$ . The above steps can be generalized to a longer time horizon $t=k+1,\dots k+T$ . Because the prior is independent over time, $p(\bar{u})=∏_t=1^Tp(u_t)$ (see Eq. 6), the function $J_k(\bar{u})$ factorizes to a sum of recursive expected free energy functions $∑_t=1^TJ_k(u_t)$ .
We are interested in the most probable value under the approximate control posterior, i.e., the MAP estimate:
$$
\displaystyle\hat{u} \displaystyle=\underset{\bar{u}∈U}{\arg\max} q^*(\bar{u})=
\underset{\bar{u}∈U}{\arg\min}∑_t=k+1^k+TJ_t(u_t)-\ln p(u_t) , \tag{30}
$$
where $U⊂ℝ^T$ is the space of affordable controls over $T$ steps. Constraints such as motor force limits can be imposed during optimization.
## 4 Gaussian approximations
We discuss the three most popular Gaussian approximations to non-linear transformations of Gaussian random variables: the first and second-order Taylor series approximations (used in extended Kalman filters) and the unscented transform (used in the unscented Kalman filter) [9, 8] [14, Ch. 5].
The first-order Taylor series approximation effectively linearizes the non-linear observation function $g(x_t)$ . Since ambiguity is known to be constant over states under a linear observation function [10], it is no surprise that the first-order Taylor also leads to an ambiguity term that is constant over states.
**Theorem 4.1**
*Let $G_x(\bar{m}_t)$ be the Jacobian of $g$ with respect to $x_t$ , evaluated at $\bar{m}_t$ . Under a first -order Taylor approximation, the parameters $Σ_t,Γ_t$ are:
$$
\displaystyleΣ_t=G_x(\bar{m}_t)\bar{S}_tG_x(\bar{m}_t)^
∫ercal+R , Γ_t=\bar{S}_tG_x(\bar{m}_t)^∫ercal . \tag{31}
$$
With these parameters, the ambiguity term does not depend on the state $x_t$ :
$$
\displaystyleE_p(y_{t,x_t\>|\>u_t;D_k)}\big{[}-\ln
\frac{p(y_t,x_t\>|\>u_t;D_k)}{p(x_t\>|\>u_t;D_k)}\big{]}=\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln|R| . \tag{32}
$$*
The proof is in Appendix 0.B. Perhaps surprisingly, under the second-order Taylor approximation, the ambiguity term varies as a function of the state $x_t$ .
**Theorem 4.2**
*Let $G_xx^(i)(\bar{m}_t)$ be the Hessian of the $i$ -th element of the non-linear observation function evaluated at $\bar{m}_t$ and $e_i$ be a canonical basis vector. The parameters $Σ_t,Γ_t$ computed through a second -order Taylor approximation are:
$$
\displaystyleΣ_t \displaystyle=G_x(\bar{m}_t)\bar{S}_tG_x(\bar{m}_t)^∫ercal+
\frac{1}{2}∑_i=1^D_y∑_j=1^D_ye_ie_j^∫ercaltr
\big{(}G_xx^(i)(\bar{m}_t)\bar{S}_tG_xx^(j)(\bar{m}_t)\bar{S}_t
\big{)}+R \displaystyleΓ_t \displaystyle=\bar{S}_tG_x(\bar{m}_t)^∫ercal . \tag{33}
$$
With these parameters, the ambiguity term depends on $x_t$ through:
$$
\displaystyleE_p(y_{t,x_t\>|\>u_t;D_k)}\big{[}-\ln
\frac{p(y_t,x_t\>|\>u_t;D_k)}{p(x_t\>|\>u_t;D_k)}\big{]}= \displaystyle \frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln\Big{|}
\frac{1}{2}∑_i=1^D_y∑_j=1^D_ye_ie_j^∫ercaltr
\big{(}G_xx^(i)(\bar{m}_t)\bar{S}_tG_xx^(j)(\bar{m}_t)\bar{S}_t
\big{)}+R \Big{|} . \tag{34}
$$*
The proof is in Appendix 0.C.
Interestingly, the ambiguity is also constant for the unscented transform.
**Theorem 4.3**
*Define $2D_x+1$ sigma points as:
$$
\displaystyleχ_0\triangleq\bar{m}_t, χ_i\triangleq\bar{m}_t
+√{D_x+λ}\big{[}√{\bar{S}_t}\big{]}_i, χ_D_{x+i}
\triangleq\bar{m}_t-√{D_x+λ}\big{[}√{\bar{S}_t}\big{]}
_i, \tag{35}
$$
where $i=1,\dots D_x$ , $[·]_i$ denotes the $i$ -th column of a matrix, and $√{S}$ denotes the matrix square root such that $√{S}√{S}=S$ . The parameter $λ\triangleqα^2(D_x+κ)-D_x$ depends on free parameters $α$ and $κ$ . Define $2D_x+1$ weights as:
$$
\displaystyle w_0\triangleq\frac{λ}{D_x+λ}+(1-α^2+β
) , w_i\triangleq\frac{1}{D_x+λ} , \tag{36}
$$
for $i=1,\dots 2D_x$ and $β$ as an additional free parameter. Under these sigma points and weights, the parameters $μ_t$ , $Σ_t$ , $Γ_t$ are [14, Eq. 5.89]:
$$
\displaystyleμ_t \displaystyle=\frac{λ}{D_x+λ}g(χ_0)+∑_i=1^2D_x
\frac{1}{2(D_x+λ)}g(χ_i) , Γ_t=∑_i=0^2D_x
w_i(χ_i-\bar{m}_t)(g(χ_i)-μ_t)^∫ercal , \displaystyle Σ_t=∑_i=0^2D_xw_i(g(χ_i)-μ_t)(g(
χ_i)-μ_t)^∫ercal+R . \tag{37}
$$
Then, the ambiguity is independent of the state:
$$
\displaystyleE_p(y_{t,x_t\>|\>u_t;D_k)} \displaystyle\big{[}-\ln\frac{p(y_t,x_t\>|\>u_t;D_k)}{p(x_t
\>|\>u_t;D_k)}\big{]}=\frac{D_y}{2}\ln(2π\textit{e})+\frac{
1}{2}\ln|R| . \tag{38}
$$*
The proof can be found in the Appendix 0.D. This result is conjectured to hold for other Gaussian approximations that are linear in their estimate of the covariance matrix, for example the Gauss-Hermite approximation [14, Ch. 6].
## 5 Experiments
Our experiment is as described in Section 2.0.1, with the nonlinear observation function $g(·)$ measuring relative angle and distance to a base station. Examples of sensors include Hall effect and ultrasound sensors. The robot starts at $x_0=[0 -1 0 0]$ and must reach $x_*=[0 1 0 0]$ . The agent’s state prior distribution’s parameters were $m_0=[0 -1 0 0]$ and $S_0=0.5I$ . Its control prior precision was set to a tiny value, $η=1.0· 10^-8$ , so as to best study the effects of ambiguity and risk. It was given a goal prior of $m_*=g(x_*)$ and $S_*=0.5I$ .
We will compare three agents Details and code at: https://github.com/biaslab/IWAI2024-ambiguity: firstly, an agent that uses the first-order Taylor approximation, referred to as EFE1. Secondly, an agent with a second-order Taylor approximation, referred to as EFE2. Thirdly, an agent with a second-order Taylor approximation but with only the risk term included, referred to as EFER. The difference between EFER and EFE2 reflects the effect of the ambiguity term, while the difference between EFE1 and EFE2 reflects the effect of the second-order Gaussian approximation. Figure 2 plots the value of the control objective function at every position in state-space, under a state covariance matrix of $S_t=I$ . States close to the sensor station are red and will lead to high values under the control objective. Note that the area around the sensor station increases from EFE1 to EFER due to the curvature of the relative distance sensor. The white markers are the approximate minimizers for this choice of $S_t$ matrix. Comparing EFER and EFE2, we can see that ambiguity increases the cost of being close to the sensor station.
<details>
<summary>extracted/5864169/figures/cart2polar-ambiguity-contourf.png Details</summary>
### Visual Description
## Contour Plots: Comparative Analysis of EFE1, EFER, and EFE2
### Overview
The image displays three side-by-side contour plots (heatmaps) labeled "EFE1", "EFER", and "EFE2". These plots visualize a scalar field over a two-dimensional domain defined by axes $X_1$ and $X_2$. Each plot features a distinct "swirl" or vortex-like pattern centered near the origin $(0,0)$ and a single white circular marker positioned at varying heights along the $X_2$ axis.
### Components/Axes
* **X-Axis ($X_1$):** Horizontal axis ranging from approximately -1.5 to 1.5, with major ticks at -1, 0, and 1.
* **Y-Axis ($X_2$):** Vertical axis ranging from -0.5 to 2.0, with major ticks at -0.5, 0.0, 0.5, 1.0, 1.5, and 2.0.
* **Legends:** Each plot includes a vertical color bar on the right side, mapping colors to numerical values.
* **EFE1:** Scale ranges from -10 (dark purple) to 15 (dark red).
* **EFER:** Scale ranges from 5 (dark purple) to 30 (dark red).
* **EFE2:** Scale ranges from 5 (dark purple) to 30 (dark red).
* **Markers:** A white dot is present in each plot, positioned at $X_1 = 0$.
* **EFE1:** $X_2 \approx 1.2$
* **EFER:** $X_2 \approx 1.3$
* **EFE2:** $X_2 \approx 1.7$
### Detailed Analysis
#### 1. EFE1
* **Trend:** The plot exhibits a large, low-value region (dark purple, $\approx -10$) occupying the upper portion of the domain. A high-value "swirl" (orange/red, $\approx 10-15$) originates from the left side ($X_1 < 0$) and spirals into a central point near $(0,0)$.
* **Marker:** The white dot is located at $(0, 1.2)$, situated within the low-value dark purple region.
#### 2. EFER
* **Trend:** The color distribution is similar to EFE1 but shifted to a positive scale (5 to 30). A large dark purple region (value $\approx 5$) dominates the top half. A high-intensity vortex (dark red, $\approx 30$) is concentrated at the origin $(0,0)$.
* **Marker:** The white dot is located at $(0, 1.3)$, slightly higher than in EFE1, still within the low-value region.
#### 3. EFE2
* **Trend:** This plot is visually nearly identical to EFER. It features the same large dark purple region at the top and the same high-intensity vortex at the origin.
* **Marker:** The white dot is located at $(0, 1.7)$, significantly higher than in the previous two plots, moving further away from the central vortex.
### Key Observations
* **Vortex Feature:** All three plots share a common topological feature: a high-value vortex centered at $(0,0)$. This suggests a non-convex landscape or a potential field with a singularity at the origin.
* **Marker Progression:** The white dot moves vertically upward across the three plots ($1.2 \rightarrow 1.3 \rightarrow 1.7$). This implies a sequential process, such as an optimization path, a parameter sweep, or a time-series evolution of a system state.
* **Scale Discrepancy:** EFE1 uses a different color scale (including negative values) compared to EFER and EFE2, which use identical positive scales. This suggests EFE1 may represent a different function, a different phase of the system, or a normalized version of the data.
### Interpretation
The data suggests these plots represent the evaluation of a cost function or a potential field in a 2D space. The "swirl" pattern is characteristic of optimization landscapes where a global or local optimum is surrounded by a "valley" or "ridge" that spirals toward the center.
The progression of the white dot indicates that the system is moving away from the central vortex (the high-value region) as the index progresses from EFE1 to EFE2. Given the visual similarity between EFER and EFE2, they likely represent the same underlying model or state, whereas EFE1 represents a distinct state or a different mathematical formulation. The white dot likely represents the current "position" or "state" of an agent or algorithm being tracked across these different configurations.
</details>
Figure 2: Value under three EFE functions over a plane: EFE1 is risk and ambiguity under a first-order Taylor approximation, EFER is risk only under a second-order Taylor approximation and EFE2 is both risk and ambiguity under a second-order Taylor approximation. White markers indicate minimizers. Note that each EFE function induces a different preference over states.
We ran 100 Monte Carlo experiments. Figure 3 plots the average trajectory of $T=30$ steps taken by the EFE1, EFER and EFE2 agents. Ribbons indicate the standard error of the mean at every time-point. Note that all agents avoid the sensor station, with EFE2 taking the widest curve (EFE1 and EFER turn at $x_1=1.0$ while EFE2 turns at $x_1=1.5$ ). EFE1 and EFER lose track of the robot in a number of experiments (like the model predictive controller in Sec. 2.0.1), leading to a more volatile average trajectory. EFE2 has the smoothest average trajectory, indicating that the ambiguity term helps planning.
<details>
<summary>extracted/5864169/figures/botnav-cart2polar-EFE1.png Details</summary>
### Visual Description
## Chart: EFE1 Trajectory Plot
### Overview
This image displays a 2D state-space plot, titled "EFE1," representing the trajectory of a system and an associated estimate as they move from a starting point to a goal point. The plot includes a fixed sensor location and shaded regions representing uncertainty (variance) for both the system and the estimate.
### Components/Axes
* **Title:** "EFE1" (centered at the top).
* **X-Axis:** Labeled "X1", with a scale ranging from -0.5 to 2.0.
* **Y-Axis:** Labeled "X2", with a scale ranging from -1.5 to 2.5.
* **Legend (Position: Top-Right):**
* **Black Triangle:** "sensor"
* **Green Circle:** "start"
* **Red Circle:** "goal"
* **Blue Line with Blue Circles:** "system"
* **Purple Line with Purple Circles:** "estimate"
### Detailed Analysis
#### Spatial Grounding and Markers
* **Sensor:** Located at coordinates (0, 0).
* **Start:** Located at coordinates (0, -1).
* **Goal:** Located at coordinates (0, 1).
#### Trajectory Trends
* **System (Blue Line):**
* **Trend:** The line originates at the "start" (0, -1), curves smoothly to the right, reaching a maximum X1 value of approximately 1.1, then curves upward and back to the left, terminating near the "goal" (0, 1).
* **Uncertainty:** The blue shaded region follows the path of the line. It is relatively narrow and consistent, indicating low variance in the system's state.
* **Estimate (Purple Line):**
* **Trend:** The line originates near the "start" (0, -1). It follows a path similar to the "system" line but is consistently offset to the right. It reaches a maximum X1 value of approximately 1.2 before curving back toward the "goal" (0, 1).
* **Uncertainty:** The purple shaded region is significantly wider and more irregular (jagged) than the blue region. This indicates a much higher level of uncertainty or variance in the estimation process compared to the system's actual state.
### Key Observations
* **Systematic Offset:** The "estimate" line consistently tracks to the right of the "system" line, suggesting a persistent bias or lag in the estimation model.
* **Uncertainty Disparity:** The "estimate" uncertainty (purple) is substantially larger than the "system" uncertainty (blue). The purple shaded area is particularly wide at the beginning and the peak of the trajectory, suggesting the estimator is less confident during the initial phase and the turn.
* **Navigation:** Both trajectories appear to be navigating around the "sensor" located at (0, 0), forming a "C" or "U" shape rotated 90 degrees clockwise.
### Interpretation
This plot is characteristic of control theory, robotics, or active inference simulations (EFE likely stands for "Expected Free Energy").
* **The Relationship:** The "system" represents the ground truth or the intended path, while the "estimate" represents the agent's internal model or filter output attempting to track that path.
* **The Discrepancy:** The fact that the "estimate" is offset from the "system" and possesses higher uncertainty suggests that the agent is operating under imperfect information or a noisy observation model.
* **Peircean Investigative View:** The "sensor" at (0,0) acts as a landmark or potential obstacle. The trajectories curve away from this point, implying the agent is maintaining a safety margin or is being influenced by the sensor's position. The widening of the purple uncertainty region at the peak of the curve (where X1 is highest) suggests that the agent's predictive model struggles most when the system is furthest from the start/goal axis, indicating a potential non-linearity in the system dynamics that the estimator is struggling to resolve.
</details>
<details>
<summary>extracted/5864169/figures/botnav-cart2polar-EFER.png Details</summary>
### Visual Description
## 2D Line Plot: EFER
### Overview
The image displays a 2D coordinate plot titled "EFER". It visualizes two distinct trajectories or data series (blue and purple) plotted against axes labeled X1 and X2. The plot illustrates the path of these series as they evolve from an upper starting position toward a common terminal point near the bottom center. The plot includes shaded regions representing variance or uncertainty, as well as specific marker symbols.
### Components/Axes
* **Title:** "EFER" (centered at the top).
* **X-axis:** Labeled "X1". The scale ranges from -0.5 to 2.0, with major ticks at 0.0, 0.5, 1.0, 1.5, and 2.0.
* **Y-axis:** Labeled "X2". The scale ranges from -1 to 2, with major ticks at -1, 0, 1, and 2.
* **Data Series 1 (Blue):** A smooth, continuous line with blue circular markers. It is accompanied by a light blue shaded region representing the variance/confidence interval.
* **Data Series 2 (Purple):** A jagged, erratic line with purple circular markers. It is accompanied by a light purple shaded region representing the variance/confidence interval.
* **Reference Markers:**
* **Green Circle:** Located at approximately (0, -1).
* **Purple Circle:** Located at approximately (0, 1).
* **Black Triangle:** Located at approximately (0, 0), pointing to the right.
### Detailed Analysis
#### Trend Verification
* **Blue Series:** The line exhibits a smooth, parabolic-like arc. It begins at approximately (-0.3, 1.0), curves upward to a peak near (0.3, 1.5), and then descends in a consistent, smooth arc, crossing the X1=1.0, X2=0.5 coordinate, and terminating at approximately (0, -1.0).
* **Purple Series:** The line exhibits high volatility. It begins near (0, 1.6) and moves erratically in the upper-left quadrant. It then follows a downward trajectory that roughly parallels the blue line, eventually converging toward the same terminal point near (0, -0.9).
#### Spatial Grounding & Data Points
* **Upper Region (X2 > 0):** The purple series shows significant deviation from the blue series. The purple shaded region is wide and irregular, indicating high uncertainty or exploration in this phase.
* **Lower Region (X2 < 0):** Both lines converge tightly toward the green circle at (0, -1). The shaded regions narrow significantly as they approach this point, suggesting that the system becomes more deterministic or stable as it reaches the target.
* **Markers:**
* The **Purple Circle** at (0, 1) sits slightly below the starting point of the purple line.
* The **Black Triangle** at (0, 0) acts as a central reference point, positioned between the start and end of the trajectories.
* The **Green Circle** at (0, -1) serves as the clear destination or "goal" state for both trajectories.
### Key Observations
* **Convergence:** Despite the high variance and erratic path of the purple series, both data series converge to the same coordinate (0, -1).
* **Volatility Differential:** The purple series is significantly more "noisy" than the blue series. This is visually represented by the jagged line and the wide, irregular purple shaded region.
* **Stability:** The blue series represents a stable, optimized path, while the purple series represents a less stable, potentially exploratory or stochastic process.
### Interpretation
The data suggests this is a visualization of a reinforcement learning agent or a control system trajectory.
* **The Blue Line** likely represents the "mean" or "optimal" policy/trajectory, characterized by smoothness and low variance.
* **The Purple Line** likely represents a "noisy" or "exploratory" policy, where the agent is testing different actions (hence the jaggedness and wide shaded region) before eventually converging on the optimal path.
* **The Goal:** The convergence at the green circle (0, -1) indicates that both the optimal and exploratory agents successfully reach the target state. The plot demonstrates the transition from high-uncertainty exploration (top of the graph) to high-certainty exploitation (bottom of the graph). The "EFER" title may refer to an algorithm or metric related to "Expected Future..." or a specific reinforcement learning framework.
</details>
<details>
<summary>extracted/5864169/figures/botnav-cart2polar-EFE2.png Details</summary>
### Visual Description
## Chart: EFE2
### Overview
This image displays a 2D coordinate plot titled "EFE2". It visualizes two distinct trajectories (represented by blue and purple lines with circular data points) plotted against an X1 and X2 axis. Additionally, three distinct markers (a red circle, a black triangle, and a green circle) are positioned along the vertical axis at X1 ≈ 0. The plot includes shaded regions around the lines, representing uncertainty or variance in the data.
### Components/Axes
* **Title:** "EFE2" (centered at the top).
* **X-Axis (Horizontal):** Labeled "X1". The scale ranges from -0.5 to 2.0, with major ticks every 0.5 units.
* **Y-Axis (Vertical):** Labeled "X2". The scale ranges from -1 to 2, with major ticks every 1 unit.
* **Data Series:**
* **Blue Series:** A line connecting blue circular markers, surrounded by a light blue shaded region.
* **Purple Series:** A line connecting purple circular markers, surrounded by a light purple shaded region.
* **Markers (positioned at X1 ≈ 0):**
* **Red Circle:** Positioned at X2 ≈ 1.0.
* **Black Triangle:** Positioned at X2 ≈ 0.0.
* **Green Circle:** Positioned at X2 ≈ -1.0.
### Detailed Analysis
The trajectories follow a curved path, starting from the left and looping toward the right before returning to the bottom-left.
**Trend Verification:**
Both the blue and purple lines exhibit a consistent trend:
1. **Start:** Both originate at approximately (-0.4, 1.5).
2. **Ascent:** They curve upward, reaching a peak at approximately (0.4, 1.8).
3. **Descent/Loop:** They curve downward and to the right, reaching a maximum X1 extension of approximately 1.4.
4. **Termination:** Both lines converge toward the bottom-left, ending at approximately (0, -1), which coincides with the location of the green circle marker.
**Data Points & Uncertainty:**
* **Shaded Regions:** The shaded areas represent the variance or confidence interval. The uncertainty is widest during the peak of the arc (around X1 = 0.5 to 1.0) and narrows significantly as the lines approach the terminal point at (0, -1).
* **Overlap:** The blue and purple lines are nearly identical in trajectory, with the purple line appearing slightly offset or "inside" the blue line at the peak of the curve.
### Key Observations
* **Convergence:** Both trajectories terminate precisely at the location of the green circle (0, -1).
* **Vertical Alignment:** The three markers (Red Circle, Black Triangle, Green Circle) are perfectly vertically aligned at X1 = 0.
* **Trajectory Shape:** The paths resemble a "C" shape rotated 90 degrees clockwise, suggesting a system moving from a starting state, performing an action or arc, and returning to a specific target state.
* **Uncertainty:** The variance (shaded regions) is highest in the middle of the trajectory, suggesting the system is most "uncertain" about its position when it is furthest from the start and end points.
### Interpretation
This plot is characteristic of control theory, reinforcement learning, or active inference simulations (often associated with "EFE" or Expected Free Energy).
* **The Trajectories:** The lines likely represent the predicted path of an agent or system state over time. The fact that they loop suggests a movement away from a starting point to explore or perform a task, followed by a return to a goal state.
* **The Markers:** The markers at X1=0 likely represent specific environmental features or states. The green circle at (0, -1) acts as a "goal" or "terminal state" because both trajectories terminate there. The red circle and black triangle may represent obstacles or alternative states that the trajectories avoid.
* **The Shaded Regions:** These represent the epistemic uncertainty of the model. The widening of the shaded area in the middle of the arc suggests that the model is less certain about the state of the system during the "mid-flight" portion of the trajectory compared to the start and end points.
* **Conclusion:** The system demonstrates a high degree of confidence in its start and end states, with a controlled, predictable arc between them, despite a slight variance in the path taken.
</details>
Figure 3: Trajectories of agents under three EFE functions, averaged over 100 Monte Carlo samples (ribbon is standard deviation of the mean). The robot starts at the green marker and must reach the red goal marker. All agents avoid the sensor station, with EFE2 taking the widest curve and having the smoothest average trajectory.
## 6 Discussion
One could argue that our analysis is more about model selection than inference, as each Gaussian approximation essentially constitutes a different generative model. In that sense, the experiments only indicate that richer approximations of nonlinear functions lead to better performance, which is not surprising. However, the result is more subtle than that since the unscented transform is richer than the first-order Taylor (produces a more accurate mean estimate [8]) but apparently still leads to constant ambiguity. No, the approximation must be sensitive to how the covariance matrix of the joint distribution over states and observations changes as a function of $g$ ’s curvature. It would be interesting to extend this work with parameter estimation, such as inferring the process noise covariance matrix using a Wishart distribution [15], or the state transition matrix with a Matrix-Normal distribution [1, 12].
## 7 Conclusion
We examined active inference agents with linear Gaussian distributed dynamics and a non-linear measurement function. We found that the first-order Taylor series and unscented transform approximations to the non-linearly transformed states lead to expected free energy functions with ambiguity terms that are constant over states. A second-order Taylor approximation leads to a state-dependent ambiguity term, inducing a preference over states.
#### 7.0.1 Acknowledgements
The author gratefully acknowledges financial support from the Eindhoven Artificial Intelligence Systems Institute (EAISI) at TU Eindhoven.
#### 7.0.2
The authors have no competing interests to declare that are relevant to the content of this article.
## Appendix 0.A Appendix: proof of Lemma 1
**Proof**
*The cross-entropy is split into two entropies that simplify according to:
$$
\displaystyleE_p(y_{t,x_t\>|\>u_t;D_k)}\big{[}-\ln
\frac{p(y_t,x_t\>|\>u_t;D_k)}{p(x_t\>|\>u_t;D_k)}\big{]} \displaystyle=-∫N\big{(}\begin{bmatrix}x_t\\
y_t\end{bmatrix}\>|\>\begin{bmatrix}\bar{m}_t\\
μ_t\end{bmatrix},\begin{bmatrix}\bar{S_t}&Γ_t\\
Γ_t^∫ercal&Σ_t\end{bmatrix}\big{)}\lnN\big{(}
\begin{bmatrix}x_t\\
y_t\end{bmatrix}\>|\>\begin{bmatrix}\bar{m}_t\\
μ_t\end{bmatrix},\begin{bmatrix}\bar{S_t}&Γ_t\\
Γ_t^∫ercal&Σ_t\end{bmatrix}\big{)} d(y_t,x_t) \displaystyle +∫N(x_t\>|\>\bar{m}_t,\bar{S}_t)\ln
N(x_t\>|\>\bar{m}_t,\bar{S}_t)dx_t \displaystyle=\frac{D_x+D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln\big{|}
\begin{bmatrix}\bar{S_t}&Γ_t\\
Γ^∫ercal_t&Σ_t\end{bmatrix}\big{|}-\Big{(}\frac{D_x}{2}
\ln(2π\textit{e})+\frac{1}{2}\ln|\bar{S}_t|\Big{)} \displaystyle=\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln\big{(}|\bar{S_t}|·|Σ_t-Γ^∫ercal_t\bar{S}^-1_tΓ_t|\big{)}
-\frac{1}{2}\ln|\bar{S}_t| \displaystyle=\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln|Σ_t-
Γ^∫ercal_t\bar{S}_t^-1Γ_t| . \tag{39}
$$*
## Appendix 0.B Appendix: proof of Theorem 1
**Proof**
*Plugging $Σ_t$ , $Γ_t$ from (31) into the result from Lemma 1, yields:
$$
\displaystyle\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln|Σ_t-
Γ_t^∫ercal\bar{S}_t^-1Γ| \displaystyle =\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln|G_x(
\bar{m}_t)\bar{S}_tG_x(\bar{m}_t)^∫ercal+R-G_x(\bar{m}_t)\bar
{S}_t^∫ercal\bar{S}_t^-1\bar{S}_tG_x(\bar{m}_t)^∫ercal| \displaystyle =\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln|R| . \tag{43}
$$
The cancellation is due to $\bar{S}_t$ being symmetric, i.e., $\bar{S}_t^∫ercal\bar{S}_t^-1=\bar{S}_t\bar{S}_t^-1=I$ .*
## Appendix 0.C Appendix: proof of Theorem 2
**Proof**
*Plugging $Σ_t$ , $Γ_t$ from (33) into the result from Lemma 1, yields:
$$
\displaystyle\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln\Big{|}
Σ_t-Γ_t^∫ercal\bar{S}_t^-1Γ \Big{|}=\frac{D
_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln\Big{|} G_x(\bar{m}_t)\bar{
S}_tG_x(\bar{m}_t)^∫ercal \displaystyle +\frac{1}{2}∑_i=1^D_y∑_j=1^D_ye_ie_j^
∫ercaltr\big{(}G_xx^(i)(\bar{m}_t)\bar{S}_tG_xx^(j)(
\bar{m}_t)\bar{S}_t\big{)}+R-G_x(\bar{m}_t)\bar{S}_t^∫ercal\bar
{S}_t^-1\bar{S}_tG_x(\bar{m}_t)^∫ercal\Big{|} \displaystyle =\frac{D_y}{2}\ln(2π\textit{e})+\frac{1}{2}\ln\Big{|}
\frac{1}{2}∑_i=1^D_y∑_j=1^D_ye_ie_j^∫ercaltr
\big{(}G_xx^(i)(\bar{m}_t)\bar{S}_tG_xx^(j)(\bar{m}_t)\bar{S}_t
\big{)}+R \Big{|} . \tag{45}
$$
The covariance matrix $\bar{S}_t$ is symmetric, i.e., $\bar{S}_t^∫ercal\bar{S}_t^-1=\bar{S}_t\bar{S}_t^-1=I$ . Note that the Hessian $G_xx^(i)(\bar{m}_t)$ depends on the inferred mean of the predicted state $\bar{m}_t$ , meaning that ambiguity is not constant over state-space.*
## Appendix 0.D Appendix: proof of Theorem 3
**Proof**
*Plugging $μ_t$ , $Σ_t$ , $Γ_t$ from (4.3) into the log-determinant term from the result in Lemma 1, gives:
$$
\displaystyle\frac{1}{2}\ln\big{|}Σ_t-Γ_t^∫ercal\bar{S}_t
^-1Γ\big{|}=\frac{1}{2}\ln\Big{|}∑_i^\prime=0^2D_xw_i^
\prime(g(χ_i^\prime)-μ_t)(g(χ_i^\prime)-μ_t)^
∫ercal+R \displaystyle-\Big{(}∑_i=0^2D_xw_i(χ_i-\bar{m}_t)(g(χ
_i)-μ_t)^∫ercal\Big{)}^∫ercal\bar{S}_t^-1\Big{(}∑_
j=0^2D_xw_j(χ_j-\bar{m}_t)(g(χ_j)-μ_t)^
∫ercal\Big{)}\Big{|} . \tag{47}
$$*
The second term can be re-arranged to:
$$
\displaystyle\Big{(}∑_i=0^2D_xw_i(χ_i-\bar{m}_t)(g(χ_i)-μ_t)^∫ercal\Big{)}^∫ercal\bar{S}_t^-1\Big{(}∑_j
=0^2D_xw_j(χ_j-\bar{m}_t)(g(χ_j)-μ_t)^
∫ercal\Big{)} \displaystyle=∑_i=0^2D_x∑_j=0^2D_xw_i(g(χ_i)-μ_t)(χ_i-\bar{m}_t)^∫ercal\bar{S}_t^-1w_j(χ_j-
\bar{m}_t)(g(χ_j)-μ_t)^∫ercal . \tag{48}
$$
Note that for $j=0$ , $(χ_j-\bar{m}_t)=(\bar{m}_t-\bar{m}_t)=0$ . Let $D_λ=D_x+λ$ . For $j≥ 1$ :
$$
\displaystyle(χ_i-\bar{m}_t)^∫ercal\bar{S}_t^-1w_j(χ_j-\bar{m}_t) \displaystyle =(\bar{m}_t+(-1)^i√{D_λ}\big{[}√{\bar{S
}_t}\big{]}_i-\bar{m}_t)^∫ercal\bar{S}_t^-1\frac{1}{D_
λ}(\bar{m}_t+(-1)^j√{D_λ}\big{[}√{\bar{S}_t}
\big{]}_j-\bar{m}_t) \displaystyle =\frac{1}{D_λ}(√{D_λ})^2(-1)^i+j\big{[}
√{\bar{S}_t}\big{]}_i^∫ercal\bar{S}_t^-1\big{[}√{\bar{S}_t}\big{]}_j \displaystyle =(-1)^i+j\big{[}√{\bar{S}_t}\big{]}_i^∫ercal\bar
{S}_t^-1\big{[}√{\bar{S}_t}\big{]}_j . \tag{49}
$$
Furthermore, note that column selection $\big{[}·\big{]}_i$ is equivalent to right-multiplication with a canonical basis vector $e_i$ ;
$$
\displaystyle\big{[}√{\bar{S}_t}\big{]}_i^∫ercal\bar{S}_t^-1
\big{[}√{\bar{S}_t}\big{]}_j=\big{(}√{\bar{S}_t}e_i\big{)}^
∫ercal\bar{S}_t^-1\big{(}√{\bar{S}_t}e_j\big{)}=e_i^
∫ercal√{\bar{S}_t}^∫ercal\bar{S}_t^-1√{\bar{S}_t}e_j
. \tag{52}
$$
Since $\bar{S}_t$ is a normal matrix, the eigendecomposition $\bar{S}_t=VΩ V^-1$ generates an orthonormal eigenvector matrix $V$ , implying $V^-1=V^⊤$ , and a diagonal matrix of eigenvalues $Ω$ . This means that $√{\bar{S}_t}=VΩ^1/2V^-1$ , and that:
$$
\displaystyle√{\bar{S}_t}^∫ercal\bar{S}_t^-1√{\bar{S}_t} \displaystyle=\big{(}VΩ^1/2V^-1\big{)}^∫ercalVΩ^-1V^-1
\big{(}VΩ^1/2V^-1\big{)} \displaystyle=VΩ^1/2V^-1VΩ^-1V^-1VΩ^1/2V^-1 \displaystyle=VV^-1=I. \tag{53}
$$
Therefore, $e_i^∫ercalIe_j$ will be $1$ for all $i=j$ and $0 0$ for $i≠ j$ . We can thus identify two cases in the double sum in (48), one of which is always $0 0$ :
$$
\displaystyle∑_i=0^2D_x∑_j=0^2D_x \displaystyle w_i(g(χ_i)-μ_t)(χ_i-\bar{m}_t)^
∫ercal\bar{S}_t^-1w_j(χ_j-\bar{m}_t)(g(χ_j)-μ_t)^∫ercal \displaystyle=∑_i=0^2D_x∑_j=iw_i(g(χ_i)-μ_t)(-1)^(i+j) 1 (g(χ_j)-μ_t)^∫ercal \displaystyle +∑_i=0^2D_x∑_j≠ iw_i(g(χ_i)-μ
_t)(-1)^(i+j) 0 (g(χ_j)-μ_t)^∫ercal \displaystyle=∑_i=0^2D_xw_i(g(χ_i)-μ_t)(g(χ_i)-
μ_t)^∫ercal , \tag{56}
$$
where the $(-1)^(i+j)$ drops out because for $i=j$ , $i+j$ will always be even. One may now recognize that (0.D) has two terms that cancel each other:
$$
\displaystyle\frac{1}{2}\ln|∑_i^\prime=0^2D_xw_i^\prime(g(χ_i^\prime)-μ_t)(g(χ_i^\prime)-μ_t)^∫ercal+R
-∑_i=0^2D_xw_i \displaystyle(g(χ_i)-μ_t)(g(χ_i)-μ_t)^∫ercal| \displaystyle=\frac{1}{2}\ln|R| . \tag{59}
$$
Using this result and Lemma 1, we have proven Theorem 3.
## References
- [1] Barber, D., Chiappa, S.: Unified inference for variational Bayesian linear Gaussian state-space models. Advances in Neural Information Processing Systems 19 (2006)
- [2] Conant, R.C., Ross Ashby, W.: Every good regulator of a system must be a model of that system. International Journal of Systems Science 1 (2), 89–97 (1970)
- [3] Cover, T.M.: Elements of information theory. John Wiley & Sons (1999)
- [4] Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., Friston, K.: Active inference on discrete state-spaces: A synthesis. Journal of Mathematical Psychology 99, 102447 (2020)
- [5] Friston, K.: The free-energy principle: a unified brain theory? Nature Reviews Neuroscience 11 (2), 127–138 (2010)
- [6] Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., Pezzulo, G.: Active inference: a process theory. Neural Computation 29 (1), 1–49 (2017)
- [7] Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T., Pezzulo, G.: Active inference and epistemic value. Cognitive Neuroscience 6 (4), 187–214 (2015)
- [8] Gustafsson, F., Hendeby, G.: Some relations between extended and unscented Kalman filters. IEEE Transactions on Signal Processing 60 (2), 545–555 (2011)
- [9] Julier, S.J., Uhlmann, J.K.: Unscented filtering and nonlinear estimation. Proceedings of the IEEE 92 (3), 401–422 (2004)
- [10] Koudahl, M.T., Kouw, W.M., de Vries, B.: On epistemics in expected free energy for linear Gaussian state space models. Entropy 23 (12), 1565 (2021)
- [11] van de Laar, T., Koudahl, M., van Erp, B., de Vries, B.: Active inference and epistemic value in graphical models. Frontiers in Robotics and AI 9, 794464 (2022)
- [12] Luttinen, J.: Fast variational Bayesian linear state-space model. In: European Conference on Machine Learning. pp. 305–320. Springer (2013)
- [13] Millidge, B., Tschantz, A., Buckley, C.L.: Whence the expected free energy? Neural Computation 33 (2), 447–482 (2021)
- [14] Särkkä, S.: Bayesian filtering and smoothing, vol. 3. Cambridge University Press (2013)
- [15] Sarkka, S., Nummenmaa, A.: Recursive noise adaptive Kalman filtering by variational Bayesian approximations. IEEE Transactions on Automatic Control 54 (3), 596–600 (2009)
- [16] Schwartenbeck, P., Passecker, J., Hauser, T.U., FitzGerald, T.H., Kronbichler, M., Friston, K.J.: Computational mechanisms of curiosity and goal-directed exploration. eLife 8, e41703 (2019)
- [17] Şenöz, İ., van de Laar, T., Bagaev, D., de Vries, B.: Variational message passing and local constraint manipulation in factor graphs. Entropy 23 (7), 807 (2021)
- [18] Tschantz, A., Seth, A.K., Buckley, C.L.: Learning action-oriented models through active inference. PLOS Computational Biology 16 (4), e1007805 (2020)
- [19] Zamiri-Jafarian, Y., Plataniotis, K.N.: A Bayesian surprise approach in designing cognitive radar for autonomous driving. Entropy 24 (5), 672 (2022)