## Algorithm Flow Diagram: PPO vs. GRPO ### Overview The image presents a comparative diagram illustrating the flow of two reinforcement learning algorithms: Proximal Policy Optimization (PPO) and Grouped Robust Policy Optimization (GRPO). The diagram highlights the key components and data flow within each algorithm, emphasizing the differences in their architectures and processes. ### Components/Axes * **Title:** PPO (top), GRPO (bottom) * **Blocks:** Representing models, computations, and data. * **Arrows:** Indicating the flow of data and processes. * **Labels:** * `q`: Input to the Policy Model * `Policy Model`: A core component in both algorithms. * `o`: Output of the Policy Model in PPO. `o1`, `o2`, ..., `oG` in GRPO. * `Reference Model`: Used in both algorithms. * `Reward Model`: Used in both algorithms. * `Value Model`: Used in PPO. * `r`: Reward signal in PPO. `r1`, `r2`, ..., `rG` in GRPO. * `v`: Value function in PPO. * `GAE`: Generalized Advantage Estimation in PPO. * `Group Computation`: Process specific to GRPO. * `A`: Advantage function in PPO. `A1`, `A2`, ..., `AG` in GRPO. * `KL`: Kullback-Leibler divergence. * **Legend (Right Side):** * Yellow: Trained Models * Blue: Frozen Models ### Detailed Analysis **PPO (Top)** 1. **Input:** `q` feeds into the `Policy Model`. 2. **Policy Model:** The output `o` from the `Policy Model` is fed into the `Reference Model`, `Reward Model`, and `Value Model`. The `Policy Model` is colored yellow, indicating it is a "Trained Model". 3. **Reference and Reward Models:** The outputs of the `Reference Model` and `Reward Model` are combined using an operation denoted by a circle with a plus sign inside (⊕), along with a `KL` divergence term, to produce `r`. The `Reference Model` and `Reward Model` are colored blue, indicating they are "Frozen Models". 4. **Value Model:** The `Value Model` outputs `v`. The `Value Model` is colored yellow, indicating it is a "Trained Model". 5. **GAE:** `r` and `v` are fed into `GAE` (Generalized Advantage Estimation). 6. **Advantage:** The output of `GAE` is `A`. 7. **Feedback Loop:** There is a feedback loop from `A` back into the `GAE` and the `Policy Model`. **GRPO (Bottom)** 1. **Input:** `q` feeds into the `Policy Model`. 2. **Policy Model:** The output of the `Policy Model` is a set of values `o1`, `o2`, ..., `oG`. The `Policy Model` is colored yellow, indicating it is a "Trained Model". 3. **Reference and Reward Models:** The outputs `o1`, `o2`, ..., `oG` are fed into the `Reference Model` and `Reward Model`. The `Reference Model` and `Reward Model` are colored blue, indicating they are "Frozen Models". 4. **Reward Signals:** The outputs of the `Reference Model` and `Reward Model`, along with a `KL` divergence term, produce a set of reward signals `r1`, `r2`, ..., `rG`. 5. **Group Computation:** The reward signals `r1`, `r2`, ..., `rG` are fed into a `Group Computation` block. 6. **Advantage:** The output of the `Group Computation` block is a set of advantage functions `A1`, `A2`, ..., `AG`. 7. **Feedback Loop:** There is a feedback loop from `A1`, `A2`, ..., `AG` back into the `Policy Model`. ### Key Observations * **Model Types:** Both PPO and GRPO use a `Policy Model`, `Reference Model`, and `Reward Model`. PPO also uses a `Value Model`. * **Frozen vs. Trained Models:** The `Reference Model` and `Reward Model` are "Frozen Models" (blue), while the `Policy Model` and `Value Model` (in PPO) are "Trained Models" (yellow). * **Grouped Structure:** GRPO introduces a grouped structure for observations, rewards, and advantages (`oG`, `rG`, `AG`), along with a `Group Computation` step. * **KL Divergence:** Both algorithms incorporate a Kullback-Leibler (KL) divergence term. * **Feedback Loops:** Both algorithms have feedback loops from the advantage function(s) back to the `Policy Model`. ### Interpretation The diagram illustrates the architectural differences between PPO and GRPO. GRPO introduces a grouped structure, allowing it to handle multiple observations, rewards, and advantages simultaneously. This suggests that GRPO is designed for environments or tasks where considering groups of data points is beneficial. The use of "Frozen Models" for the `Reference Model` and `Reward Model` in both algorithms implies that these models are pre-trained or fixed during the training process, potentially to provide a stable reference or reward signal. The feedback loops in both algorithms are characteristic of reinforcement learning, allowing the policy to be iteratively improved based on the observed advantages.

## Diagram: PPO vs. GRPO Model Architectures ### Overview The image presents a comparative diagram illustrating the architectures of two reinforcement learning algorithms: Proximal Policy Optimization (PPO) and Group Proximal Policy Optimization (GRPO). The diagram visually contrasts the data flow and model interactions within each algorithm. The top section details PPO, while the bottom section details GRPO. Both sections share similar components but differ in how they handle groups of data. ### Components/Axes The diagram consists of several key components: * **PPO Section (Top):** * `q`: Input query. * **Policy Model:** Processes `q` and outputs `o` (observation). * **Reference Model:** Receives `o` and contributes to the calculation of `KL` divergence. * **Reward Model:** Receives `o` and outputs `r` (reward). * **Value Model:** Receives `o` and outputs `v` (value). * **GAE:** Generalized Advantage Estimation, receives `r` and `v`. * **A:** Action. * `KL`: Kullback-Leibler divergence. * **GRPO Section (Bottom):** * `q`: Input query. * **Policy Model:** Processes `q` and outputs `o1` to `oG` (observations). * **Reference Model:** Receives `o1` to `oG` and contributes to the calculation of `KL` divergence. * **Reward Model:** Receives `o1` to `oG` and outputs `r1` to `rG` (rewards). * **Group Computation:** Processes `r1` to `rG`. * **A1` to `AG`: Actions. * `KL`: Kullback-Leibler divergence. * **Labels:** * "PPO" - Label for the top section. * "GRPO" - Label for the bottom section. * "Trained Models" - Label for the output of the PPO section. * "Frozen Models" - Label for the output of the GRPO section. ### Detailed Analysis or Content Details **PPO Section:** The PPO section shows a single-path data flow. The input `q` goes to the Policy Model, which generates an observation `o`. This observation is then fed into the Reference, Reward, and Value Models. The Reward Model outputs `r`, and the Value Model outputs `v`. These are used by the GAE to produce an action `A`. The KL divergence is calculated using the Reference Model and is combined with `r` via addition. The arrows indicate a sequential flow of information. **GRPO Section:** The GRPO section demonstrates a grouped approach. The input `q` goes to the Policy Model, which generates a series of observations `o1` through `oG`. These observations are fed into the Reference, Reward Models, producing rewards `r1` through `rG`. These rewards are then processed by the "Group Computation" block, which outputs actions `A1` through `AG`. The KL divergence is calculated using the Reference Model. The dotted line between the PPO and GRPO sections suggests a conceptual link or comparison. ### Key Observations * The GRPO architecture introduces a grouping mechanism, processing multiple observations and rewards in parallel before generating actions. * The PPO architecture operates on a single observation-reward-action cycle. * The "Group Computation" block in GRPO is a key differentiator, suggesting a method for aggregating information across multiple observations. * The labels "Trained Models" and "Frozen Models" indicate that PPO models are continuously updated during training, while GRPO models are fixed after an initial training phase. ### Interpretation The diagram illustrates a fundamental difference in how PPO and GRPO handle data during reinforcement learning. PPO employs a standard policy gradient approach, updating its policy based on individual experiences. GRPO, on the other hand, leverages a group-based approach, potentially improving sample efficiency and stability by aggregating information across multiple observations. The "Frozen Models" label for GRPO suggests that the algorithm might be designed for scenarios where computational resources are limited or where a pre-trained model needs to be adapted to a new environment without extensive retraining. The dotted line between the two sections highlights the conceptual relationship, suggesting GRPO can be seen as an extension or modification of the PPO algorithm. The use of KL divergence in both models indicates a common goal of preventing drastic policy changes during training. The diagram is a high-level architectural overview and does not provide specific details about the implementation of the models or the group computation process.

\n ## Technical Diagram: Comparison of PPO and GRPO Reinforcement Learning Architectures ### Overview The image is a technical flowchart diagram comparing two reinforcement learning algorithm architectures: **PPO** (Proximal Policy Optimization) and **GRPO** (Group Relative Policy Optimization). The diagram is divided into two horizontal sections separated by a dashed line. The top section illustrates the PPO workflow, and the bottom section illustrates the GRPO workflow. A legend on the right side defines the color-coding for model types. ### Components/Axes **Legend (Right Side):** * **Trained Models:** Represented by yellow-filled boxes with black borders. * **Frozen Models:** Represented by blue-filled boxes with black borders. **PPO Section (Top Half):** * **Input:** A single input labeled `q` (likely representing a query or state). * **Core Models:** * `Policy Model` (Trained, Yellow): Takes `q` as input, produces output `o`. * `Reference Model` (Frozen, Blue): Receives `o`. * `Reward Model` (Frozen, Blue): Receives `o`. * `Value Model` (Trained, Yellow): Receives `o`. * **Outputs & Computations:** * `o`: Output from the Policy Model. * `KL`: Kullback-Leibler divergence, computed between the Policy Model and the Reference Model. * `⊕`: A summation or combination symbol, combining the KL term and the output from the Reward Model. * `r`: Scalar reward signal, output from the Reward Model (after combination with KL). * `v`: Value estimate, output from the Value Model. * `GAE`: Generalized Advantage Estimation, a computation block that takes `r` and `v` as inputs. * `A`: The final advantage estimate, output from the GAE block. * **Flow:** The diagram shows a feedback loop where the advantage `A` is used to update the `Policy Model` and `Value Model`. **GRPO Section (Bottom Half):** * **Input:** A single input labeled `q`. * **Core Models:** * `Policy Model` (Trained, Yellow): Takes `q` as input, produces a *group* of outputs: `o₁`, `o₂`, ..., `o_G`. * `Reference Model` (Frozen, Blue): Receives the group of outputs. * `Reward Model` (Frozen, Blue): Receives the group of outputs. * **Outputs & Computations:** * `o₁, o₂, ..., o_G`: A group of `G` outputs from the Policy Model. * `KL`: Kullback-Leibler divergence, computed between the Policy Model and the Reference Model. The arrow points directly to the Policy Model, suggesting a direct regularization term. * `r₁, r₂, ..., r_G`: A group of scalar reward signals, each corresponding to an output `o_i`, from the Reward Model. * `Group Computation`: A processing block that takes the group of rewards `r₁...r_G` as input. * `A₁, A₂, ..., A_G`: A group of advantage estimates, output from the Group Computation block. * **Flow:** The diagram shows a feedback loop where the group of advantages `A₁...A_G` is used to update the `Policy Model`. ### Detailed Analysis **PPO Architecture Flow:** 1. A single query `q` is fed into the trained `Policy Model`. 2. The Policy Model generates a single output `o`. 3. This output `o` is simultaneously fed into three models: the frozen `Reference Model`, the frozen `Reward Model`, and the trained `Value Model`. 4. The `Reference Model` helps compute a `KL` divergence penalty against the current policy. 5. The `Reward Model` produces a reward signal `r`. The diagram indicates this `r` is combined with the `KL` term (via the `⊕` symbol). 6. The `Value Model` produces a value estimate `v`. 7. The combined reward `r` and value `v` are fed into the `GAE` (Generalized Advantage Estimation) module. 8. The GAE module computes the final advantage estimate `A`. 9. The advantage `A` is used in the loss function to update the parameters of the `Policy Model` and the `Value Model` (indicated by the curved feedback arrows). **GRPO Architecture Flow:** 1. A single query `q` is fed into the trained `Policy Model`. 2. The Policy Model generates a *group* of `G` different outputs: `o₁, o₂, ..., o_G`. 3. This entire group of outputs is fed into the frozen `Reference Model` and the frozen `Reward Model`. 4. The `Reference Model` computes a `KL` divergence penalty directly against the `Policy Model`. 5. The `Reward Model` produces a corresponding group of scalar rewards: `r₁, r₂, ..., r_G`. 6. This group of rewards is processed by a `Group Computation` block. This likely involves normalizing or comparing rewards within the group to compute relative advantages. 7. The Group Computation block outputs a group of advantages: `A₁, A₂, ..., A_G`. 8. These group advantages are used in the loss function to update the parameters of the `Policy Model` (indicated by the curved feedback arrow). Notably, there is no separate Value Model in the GRPO diagram. ### Key Observations 1. **Output Granularity:** The most fundamental difference is that PPO processes a single output (`o`) per query, while GRPO processes a group of `G` outputs (`o₁...o_G`) per query. 2. **Advantage Calculation:** PPO uses the classic GAE method combining a reward signal and a value estimate. GRPO replaces this with a "Group Computation" step that operates on multiple rewards, suggesting it calculates advantages based on the relative performance within the generated group. 3. **Model Architecture:** PPO explicitly includes a trained `Value Model` to estimate state values (`v`). GRPO does not show a Value Model, implying its advantage estimation is derived differently, likely from the group rewards. 4. **KL Divergence Application:** In PPO, the KL term is combined with the reward `r`. In GRPO, the KL arrow points directly to the Policy Model, suggesting it may be applied as a direct regularization term in the policy loss. 5. **Color-Coding Consistency:** The legend is applied consistently. The `Policy Model` and `Value Model` (in PPO) are yellow (Trained). The `Reference Model` and `Reward Model` are blue (Frozen) in both architectures. ### Interpretation This diagram contrasts two approaches to policy optimization in reinforcement learning, likely from a language model fine-tuning context (given the "query" `q` and "output" `o` terminology). * **PPO** represents the standard, well-established approach. It relies on a separate value function (the Value Model) to estimate the "goodness" of a state, which is crucial for calculating low-variance advantage estimates via GAE. This is effective but requires training and maintaining an additional model. * **GRPO** appears to be a proposed variant that eliminates the need for a separate value function. Instead of predicting absolute value, it generates multiple responses (`o₁...o_G`) for a single prompt and computes advantages *relative* to the group. The "Group Computation" likely normalizes the rewards (e.g., subtracting the mean, dividing by standard deviation) to determine which outputs in the batch were better or worse than average. This relative advantage (`A_i`) is then used for policy update. The core innovation suggested by GRPO is shifting from absolute advantage estimation (requiring a value model) to relative advantage estimation within a generated batch. This could potentially simplify training (one less model to train) and might offer more stable learning signals by focusing on comparative performance. The direct application of the KL penalty in GRPO also hints at a potentially different formulation of the policy loss objective. The diagram effectively highlights this architectural shift from a single-sample, value-dependent process to a multi-sample, group-relative process.

## Diagram: Comparison of PPO and GRPO Algorithms ### Overview The diagram illustrates the architectural differences between Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) in reinforcement learning. It highlights the flow of data, model interactions, and computational steps for each algorithm. ### Components/Axes - **Key Components**: - **Policy Model**: Generates actions (`o`) from input (`q`). - **Reference Model**: Provides a baseline for comparison (frozen in both PPO and GRPO). - **Reward Model**: Computes rewards (`r` or `r1...rG`) (frozen in GRPO, trained in PPO). - **Value Model**: Estimates state values (`v`) (trained in PPO only). - **GAE (Generalized Advantage Estimation)**: Combines KL divergence and reward signals to compute advantages (`A`). - **Group Computation**: Aggregates rewards (`r1...rG`) into a group action (`AG`) for GRPO. - **Legend**: - **Trained Models** (yellow): Policy, Reward, Value Models (PPO) and Policy Model (GRPO). - **Frozen Models** (blue): Reference and Reward Models (GRPO). ### Detailed Analysis 1. **PPO Workflow**: - Input `q` → Policy Model → Output `o`. - `o` is processed by: - Reference Model (frozen, blue). - Reward Model (trained, yellow). - Value Model (trained, yellow). - Outputs feed into GAE, which computes advantage `A`. 2. **GRPO Workflow**: - Input `q` → Policy Model → Multiple outputs (`o1...oG`). - Each `oi` is processed by: - Reference Model (frozen, blue). - Reward Model (frozen, blue). - Rewards (`r1...rG`) are aggregated via Group Computation to produce `AG`. ### Key Observations - **Model Training Status**: - PPO trains Policy, Reward, and Value Models (yellow). - GRPO freezes Reference and Reward Models (blue) but trains the Policy Model (yellow). - **Output Handling**: - PPO uses a single output (`o`) for advantage computation. - GRPO processes multiple outputs (`o1...oG`) and aggregates rewards into a group action (`AG`). - **Computational Steps**: - PPO relies on GAE for advantage estimation. - GRPO uses Group Computation to handle multi-output scenarios. ### Interpretation - **PPO’s Strengths**: - Trains multiple models (Policy, Reward, Value) for fine-grained policy updates. - Uses GAE to balance exploration and exploitation via KL divergence and reward signals. - **GRPO’s Innovations**: - Freezes Reference and Reward Models to stabilize training. - Processes multiple outputs (`o1...oG`) to handle complex, grouped actions (e.g., multi-step tasks). - Group Computation enables efficient scaling for high-dimensional action spaces. - **Trade-offs**: - PPO’s training of Reward and Value Models may increase computational cost but improve policy adaptability. - GRPO’s frozen models reduce training complexity but may limit reward signal flexibility. This diagram underscores the design philosophies of PPO (dynamic, multi-model training) and GRPO (stability via frozen components with group-based optimization).