## Diagram: Actor-Critic Debate Process
### Overview
The image illustrates a process involving an Actor and a Critic, where they engage in a "debate" to refine a model. The diagram shows the flow of information and decision-making based on the relative quality of trajectories.
### Components/Axes
* **Actor:** Represented by an orange robot icon.
* **Critic:** Represented by a blue robot icon.
* **States:** Represented by rectangular boxes labeled with z subscript a (orange) and z subscript c (blue/green/red). The superscript (t-1) and (t) denote time steps.
* **Decision Points:** Represented by circles labeled with Δy (green) and Δ!y (red).
* **Preference Data:** Represented by a cylinder, indicating a data storage.
* **Train Models:** Represented by orange and blue robots with arrows pointing to slightly different versions of themselves.
* **Arrows:** Indicate the flow of information and decision-making.
* **Text Labels:** Describe the processes and conditions.
### Detailed Analysis
1. **Initial States:**
* The Actor starts with state z subscript a at time (t-1), enclosed in an orange box.
* The Critic starts with state z subscript c at time (t-1), enclosed in a blue box.
2. **Debate Process:**
* The Actor's state z subscript a (t-1) and the Critic's state z subscript c (t-1) lead to a "Natural Debate".
* The Actor is "Critic Guided Towards y", resulting in state z subscript a (t) in an orange box and z subscript c (t) in a green box.
* The Critic is "Critic Guided Away From y", resulting in state z subscript a (t) in an orange box and z subscript c (t) in a red box.
3. **Decision Points:**
* The "Natural Debate" results in states z subscript a (t) in an orange box and z subscript c (t) in a blue box.
* The "Critic Guided Towards y" path leads to Δy (green).
* The "Critic Guided Away From y" path leads to Δ!y (red).
4. **Conditional Logic:**
* If Δy is greater than or equal to ε, then (z subscript c (t), z subscript c (t)) from the "Critic Guided Towards y" path is added to the "Preference Data".
* "elif" (else if) Δ!y is greater than or equal to ε, then (z subscript c (t), z subscript c (t)) from the "Critic Guided Away From y" path is added to the "Preference Data".
5. **Training:**
* The "Preference Data" is used to "Train Models", resulting in updated Actor (orange) and Critic (blue) models.
* The updated Critic model feeds back into the initial state of the Critic.
### Key Observations
* The diagram illustrates an iterative process where the Actor and Critic interact and refine their models based on a debate and relative quality of trajectories.
* The decision points Δy and Δ!y determine which data is used to update the models.
* The use of "elif" suggests that only one of the two conditions (Δy >= ε or Δ!y >= ε) can be true at a time.
### Interpretation
The diagram depicts a reinforcement learning process where an Actor and a Critic engage in a form of adversarial training. The "Natural Debate" represents the initial interaction between the Actor and Critic. The "Critic Guided Towards y" and "Critic Guided Away From y" paths represent different strategies or outcomes based on the Critic's guidance. The decision points Δy and Δ!y, along with the threshold ε, determine which trajectories are considered valuable and used to update the models. This process aims to improve the Actor's performance by incorporating feedback from the Critic, leading to more refined models. The use of different colors (orange, blue, green, red) helps to visually distinguish the different states and paths in the process.
