## Diagram: Comparison of Iterative Training Methodologies for LLMs ### Overview This image displays a comparative diagram of five distinct iterative training methodologies for Large Language Models (LLMs). Each row represents a specific framework (RFT/STaR/ReST^EM, V-STaR, MATH-SHEPHERD, Self-Rewarding, and ReST-MCTS*). The diagram illustrates the workflow from input data $\{x_i\}$ through model generation, selection/reward processing, and final training to produce an updated model $M_{t+1}$. ### Components/Axes The diagram is organized into five horizontal rows, separated by dashed lines. Each row follows a left-to-right flow: * **Input Data:** Represented by a blue cylinder labeled $\{x_i\}$. * **Model State:** Represented by a green rectangle labeled $M_t$. * **Generation Method:** A pink/orange rectangle positioned above $M_t$ (e.g., CoT, BoN, MCTS*). * **Output Data:** A blue cylinder containing generated outputs $\{y_i^1, \dots, y_i^N\}$. * **Processing/Selection:** A central block (blue, yellow, or grey) that filters or evaluates the outputs. * **Final Training Data:** A blue cylinder containing selected outputs $\{y_i^m, \dots, y_i^n\}$. * **Training Method:** An arrow label indicating the training algorithm (SFT, DPO, PPO). * **Updated Model:** A green rectangle labeled $M_{t+1}$. --- ### Detailed Analysis #### 1. RFT/STaR/ReST^EM * **Generation:** $M_t$ uses **CoT** (Chain of Thought). * **Process:** The output $\{y_i^1, \dots, y_i^N\}$ passes through a **Z** block (Selection). * **Training:** The selected data $\{y_i^m, \dots, y_i^n\}$ is used for **SFT** (Supervised Fine-Tuning) to update to $M_{t+1}$. #### 2. V-STaR * **Generation:** $M_t$ uses **CoT**. * **Process:** The output $\{y_i^1, \dots, y_i^N\}$ splits into two paths: 1. To a **Z** block for selection. 2. To a **DPO** (Direct Preference Optimization) process, which feeds into a **V** (Value) block. * **Training:** The selected data is used for **SFT** to update to $M_{t+1}$. #### 3. MATH-SHEPHERD * **Generation:** $M_t$ uses **BoN** (Best-of-N). * **Process:** The output $\{y_i^1, \dots, y_i^N\}$ splits into two paths: 1. **PRM** (Process Reward Model), guided by **HE/SE** (likely Heuristic/Search Evaluation). 2. **ORM** (Outcome Reward Model). * **Training:** The selected data $\{y_i^m, \dots, y_i^n\}$ is used for **PPO** (Proximal Policy Optimization) to update to $M_{t+1}$. #### 4. Self-Rewarding * **Generation:** $M_t$ uses **CoT**. * **Process:** The output $\{y_i^1, \dots, y_i^N\}$ is fed back into $M_t$ to generate rewards $\{r_i^1, \dots, r_i^N\}$. * **Training:** The combined data $\{x_i, y_i^m, \dots, y_i^n\}$ is used for **DPO** to update to $M_{t+1}$. #### 5. ReST-MCTS* * **Generation:** $M_t$ uses **MCTS*** (Monte Carlo Tree Search). * **Process:** The output $\{y_i^1, \dots, y_i^N\}$ is evaluated by a **Value Model** and a **PRM** (Process Reward Model). * **Training:** The selected data $\{y_i^m, \dots, y_i^n\}$ is used for **SFT** to update to $M_{t+1}$. --- ### Key Observations * **Commonality:** All five methods utilize an iterative loop where the model $M_t$ generates data, which is then filtered or evaluated to create a refined dataset for training $M_{t+1}$. * **Training Algorithms:** * **SFT** is the most common training method (used in RFT/STaR/ReST^EM, V-STaR, and ReST-MCTS*). * **DPO** is used in V-STaR and Self-Rewarding. * **PPO** is unique to MATH-SHEPHERD in this set. * **Evaluation Complexity:** There is a clear progression in the complexity of the "middle" selection/evaluation blocks, moving from simple selection (Z) to complex reward modeling (PRM/ORM/Value Model). ### Interpretation This diagram demonstrates the evolution of iterative LLM training strategies. * **Early Approaches (RFT/STaR):** Focus on simple filtering (Z) of Chain-of-Thought outputs, relying on the model's initial generation capabilities. * **Preference-Based Alignment (V-STaR, Self-Rewarding):** These methods introduce DPO, shifting the focus from simple imitation (SFT) to preference optimization, where the model learns to distinguish between better and worse outputs. * **Reward Modeling (MATH-SHEPHERD, ReST-MCTS*):** These methods introduce explicit reward models (PRM/ORM/Value Model) to guide the training process. MATH-SHEPHERD specifically uses PPO, suggesting a reinforcement learning approach to optimize the policy based on process-level or outcome-level rewards. * **Search-Based Generation (ReST-MCTS*):** The inclusion of MCTS* indicates a shift toward search-based reasoning, where the model explores a tree of possibilities rather than just generating a single chain of thought, requiring a Value Model to guide the search.