## Diagram: Comparison of Iterative Training Methodologies for LLMs
### Overview
This image displays a comparative diagram of five distinct iterative training methodologies for Large Language Models (LLMs). Each row represents a specific framework (RFT/STaR/ReST^EM, V-STaR, MATH-SHEPHERD, Self-Rewarding, and ReST-MCTS*). The diagram illustrates the workflow from input data $\{x_i\}$ through model generation, selection/reward processing, and final training to produce an updated model $M_{t+1}$.
### Components/Axes
The diagram is organized into five horizontal rows, separated by dashed lines. Each row follows a left-to-right flow:
* **Input Data:** Represented by a blue cylinder labeled $\{x_i\}$.
* **Model State:** Represented by a green rectangle labeled $M_t$.
* **Generation Method:** A pink/orange rectangle positioned above $M_t$ (e.g., CoT, BoN, MCTS*).
* **Output Data:** A blue cylinder containing generated outputs $\{y_i^1, \dots, y_i^N\}$.
* **Processing/Selection:** A central block (blue, yellow, or grey) that filters or evaluates the outputs.
* **Final Training Data:** A blue cylinder containing selected outputs $\{y_i^m, \dots, y_i^n\}$.
* **Training Method:** An arrow label indicating the training algorithm (SFT, DPO, PPO).
* **Updated Model:** A green rectangle labeled $M_{t+1}$.
---
### Detailed Analysis
#### 1. RFT/STaR/ReST^EM
* **Generation:** $M_t$ uses **CoT** (Chain of Thought).
* **Process:** The output $\{y_i^1, \dots, y_i^N\}$ passes through a **Z** block (Selection).
* **Training:** The selected data $\{y_i^m, \dots, y_i^n\}$ is used for **SFT** (Supervised Fine-Tuning) to update to $M_{t+1}$.
#### 2. V-STaR
* **Generation:** $M_t$ uses **CoT**.
* **Process:** The output $\{y_i^1, \dots, y_i^N\}$ splits into two paths:
1. To a **Z** block for selection.
2. To a **DPO** (Direct Preference Optimization) process, which feeds into a **V** (Value) block.
* **Training:** The selected data is used for **SFT** to update to $M_{t+1}$.
#### 3. MATH-SHEPHERD
* **Generation:** $M_t$ uses **BoN** (Best-of-N).
* **Process:** The output $\{y_i^1, \dots, y_i^N\}$ splits into two paths:
1. **PRM** (Process Reward Model), guided by **HE/SE** (likely Heuristic/Search Evaluation).
2. **ORM** (Outcome Reward Model).
* **Training:** The selected data $\{y_i^m, \dots, y_i^n\}$ is used for **PPO** (Proximal Policy Optimization) to update to $M_{t+1}$.
#### 4. Self-Rewarding
* **Generation:** $M_t$ uses **CoT**.
* **Process:** The output $\{y_i^1, \dots, y_i^N\}$ is fed back into $M_t$ to generate rewards $\{r_i^1, \dots, r_i^N\}$.
* **Training:** The combined data $\{x_i, y_i^m, \dots, y_i^n\}$ is used for **DPO** to update to $M_{t+1}$.
#### 5. ReST-MCTS*
* **Generation:** $M_t$ uses **MCTS*** (Monte Carlo Tree Search).
* **Process:** The output $\{y_i^1, \dots, y_i^N\}$ is evaluated by a **Value Model** and a **PRM** (Process Reward Model).
* **Training:** The selected data $\{y_i^m, \dots, y_i^n\}$ is used for **SFT** to update to $M_{t+1}$.
---
### Key Observations
* **Commonality:** All five methods utilize an iterative loop where the model $M_t$ generates data, which is then filtered or evaluated to create a refined dataset for training $M_{t+1}$.
* **Training Algorithms:**
* **SFT** is the most common training method (used in RFT/STaR/ReST^EM, V-STaR, and ReST-MCTS*).
* **DPO** is used in V-STaR and Self-Rewarding.
* **PPO** is unique to MATH-SHEPHERD in this set.
* **Evaluation Complexity:** There is a clear progression in the complexity of the "middle" selection/evaluation blocks, moving from simple selection (Z) to complex reward modeling (PRM/ORM/Value Model).
### Interpretation
This diagram demonstrates the evolution of iterative LLM training strategies.
* **Early Approaches (RFT/STaR):** Focus on simple filtering (Z) of Chain-of-Thought outputs, relying on the model's initial generation capabilities.
* **Preference-Based Alignment (V-STaR, Self-Rewarding):** These methods introduce DPO, shifting the focus from simple imitation (SFT) to preference optimization, where the model learns to distinguish between better and worse outputs.
* **Reward Modeling (MATH-SHEPHERD, ReST-MCTS*):** These methods introduce explicit reward models (PRM/ORM/Value Model) to guide the training process. MATH-SHEPHERD specifically uses PPO, suggesting a reinforcement learning approach to optimize the policy based on process-level or outcome-level rewards.
* **Search-Based Generation (ReST-MCTS*):** The inclusion of MCTS* indicates a shift toward search-based reasoning, where the model explores a tree of possibilities rather than just generating a single chain of thought, requiring a Value Model to guide the search.
