## Diagram: SymDQN Decision Process
### Overview
The image illustrates a decision-making process within a Reinforcement Learning (RL) framework, specifically utilizing a "SymDQN" (Symmetry-aware Deep Q-Network) approach. The diagram depicts a state-space representation (a 5x5 grid), a set of potential action outcomes (a diamond-shaped icon cluster), and the mathematical filtering process used to select the optimal action based on rewards and Q-values.
### Components/Axes
* **Left Region (Grid World):** A 5x5 grid containing various symbols.
* The grid contains empty squares, circles, crosses (X), and a plus sign.
* A specific cell at row 3, column 2 is highlighted with a thicker border, likely representing the current agent position.
* **Center Region (Action Outcomes):** A diamond-shaped cluster of four icons representing the potential next states resulting from four possible actions (Up, Left, Right, Down).
* **Top:** Empty square.
* **Left:** Circle pattern.
* **Right:** X pattern.
* **Bottom:** X pattern.
* **Right Region (Decision Logic):** Two stacked rectangular boxes containing mathematical vectors.
* **Top Box:** Contains reward vector `r` and the identified `best actions`.
* **Bottom Box:** Contains Q-value vector `q`, the `filter q` operation, and the final selected `action`.
* **Connectivity:** Arrows indicate the flow of information from the grid to the action outcomes, and from the action outcomes/SymDQN module to the decision logic boxes.
### Detailed Analysis
#### 1. Grid State
The 5x5 grid serves as the environment state.
* **Agent Position:** Row 3, Column 2 (thick border).
* **Other notable features:**
* Row 1, Col 5: X
* Row 2, Col 5: Circle
* Row 3, Col 3: Circle
* Row 4, Col 3: Circle
* Row 4, Col 4: Plus
* Row 4, Col 5: X
* Row 5, Col 4: X
#### 2. Decision Logic (Right Boxes)
The logic follows a specific filtering process to determine the optimal action:
* **Reward Vector (`r`):** `[0, 1, 1, -1]`
* This vector represents the immediate reward associated with each of the four actions (indices 0, 1, 2, 3).
* **Best Actions:** The indices with the maximum reward (1) are `[1, 2]`.
* **Q-Value Vector (`q`):** `[0.34, -0.2, -0.13, 0.3]`
* This vector represents the predicted future value of taking each action.
* **Filtering Process (`filter q`):**
* The algorithm applies a mask to the Q-values based on the `best actions` identified from the reward vector.
* Indices not in `best actions` (0 and 3) are set to negative infinity (`-∞`).
* Resulting vector: `[-∞, -0.2, -0.13, -∞]`
* **Action Selection:**
* The agent selects the action corresponding to the maximum value in the filtered vector.
* Comparing `-0.2` (index 1) and `-0.13` (index 2), `-0.13` is the higher value.
* **Final Action:** `2`
### Key Observations
* **Symmetry Awareness:** The "SymDQN" label suggests the algorithm is designed to exploit symmetries in the environment. The filtering process effectively restricts the agent's choice to only those actions that yield the maximum immediate reward, regardless of the Q-values of other actions.
* **Logic Flow:** The diagram demonstrates a "constrained optimization" approach. Even though index 0 and index 3 have high Q-values (0.34 and 0.3 respectively), they are ignored because their immediate reward (`r`) is lower than the maximum.
* **Action Selection:** The final selection of `action = 2` is a tie-break between two equally rewarding actions (1 and 2) based on which has the higher Q-value.
### Interpretation
This diagram visualizes how a SymDQN agent prioritizes immediate reward signals to constrain its policy search. By calculating `best actions` based on the reward vector `r` and using that to filter the Q-values, the agent ensures it does not pursue high-Q-value actions that are suboptimal in terms of immediate reward. This is a method to enforce safety or goal-directed behavior in reinforcement learning, ensuring the agent focuses on the most promising immediate paths before considering long-term value estimates.
