## Diagram: Algorithm Flowcharts ### Overview The image presents a series of flowcharts, each representing a different algorithm or method. The algorithms are RFT/STaR/ReST^EM, V-STaR, MATH-SHEPHERD, Self-Rewarding, and ReST-MCTS*. Each flowchart illustrates the sequence of operations and data transformations within the respective algorithm. ### Components/Axes Each flowchart consists of the following components: - **Input Data:** Represented as a blue cylinder labeled "{x_i}". - **Model (M_t):** Represented as a green rectangle. - **Intermediate Data:** Represented as blue cylinders labeled with variations of "{y_i¹, ..., y_i^N}" or "{y_i^m, ..., y_iⁿ}". - **Processing Steps:** Represented as light blue, light orange, or light red rectangles with labels such as "Z", "PRM", "ORM", "Value Model", "CoT", "BoN", "HE/SE", "DPO", "PPO", "SFT", "MCTS*". - **Selection Steps:** Indicated by the word "select" and an arrow. - **Output Data:** Represented as a green rectangle labeled "M_t+1". The flow direction is generally from left to right, indicating the sequence of operations. ### Detailed Analysis **1. RFT/STaR/ReST^EM** - Input: {x_i} - Process: CoT (light red rectangle) -> M_t (green rectangle) - Intermediate Data: {y_i¹, ..., y_i^N} - Process: Z (light blue rectangle) -> select {y_i^m, ..., y_iⁿ} - Process: SFT (light green rectangle) -> M_t+1 **2. V-STaR** - Input: {x_i} - Process: CoT (light red rectangle) -> M_t (green rectangle) - Intermediate Data: {y_i¹, ..., y_i^N} - Process: Z (light blue rectangle) -> select {y_i^m, ..., y_iⁿ} - Process: DPO (light gray rectangle) and V (light gray rectangle) - Process: SFT (light green rectangle) -> M_t+1 **3. MATH-SHEPHERD** - Input: {x_i} - Process: BoN (light red rectangle) -> M_t (green rectangle) - Intermediate Data: {y_i¹, ..., y_i^N} - Process: HE/SE (light red rectangle) -> PRM (light orange rectangle) -> select {y_i^m, ..., y_iⁿ} - Process: ORM (light yellow rectangle) - Process: PPO (light green rectangle) -> M_t+1 **4. Self-Rewarding** - Input: {x_i} - Process: CoT (light red rectangle) -> M_t (green rectangle) - Intermediate Data: {y_i¹, ..., y_i^N} - Process: M_t (green rectangle) - Intermediate Data: {r_i¹, ..., r_i^N} - Process: select {x_i, y_i^m, ..., y_iⁿ} - Process: DPO (light gray rectangle) -> M_t+1 **5. ReST-MCTS*** - Input: {x_i} - Process: MCTS* (light red rectangle) -> M_t (green rectangle) - Intermediate Data: {y_i¹, ..., y_i^N} - Process: Value Model (light red rectangle) -> PRM (light orange rectangle) -> select {y_i^m, ..., y_iⁿ} - Process: SFT (light green rectangle) -> M_t+1 ### Key Observations - All algorithms start with input data {x_i} and produce an output M_t+1. - The algorithms differ in their intermediate processing steps and data transformations. - Several algorithms use "CoT" (Chain of Thought) as an initial processing step. - "PRM" (likely Prompt Rewriting Module) appears in MATH-SHEPHERD and ReST-MCTS*. - "DPO" (Direct Preference Optimization) appears in V-STaR and Self-Rewarding. - "SFT" (Supervised Fine-Tuning) is a common final step before producing M_t+1. ### Interpretation The diagram illustrates the architectural differences between several reinforcement learning and language model training algorithms. The flowcharts highlight the key components and data flow within each algorithm, emphasizing the different strategies used for processing input data, generating intermediate representations, and optimizing the final model. The presence of common elements like "CoT" and "SFT" suggests shared underlying principles, while the unique processing steps indicate distinct approaches to problem-solving and model training. The diagram is useful for comparing and contrasting these algorithms at a high level.