## Diagram: GRPO Process Flow ### Overview The image is a diagram illustrating the GRPO process flow, starting with a GSM8K sample and culminating in a reward system based on format, math accuracy, and diversity. The diagram shows a series of transformations and aggregations, involving embeddings and cosine similarity calculations. ### Components/Axes * **Input:** GSM8K Sample * **Rollout:** Process applied to the sample, influenced by Qwen2.5 (1.5B parameters). * **Res 1, Res 2, Res n:** Represent different residual blocks or processing stages. * **Ebd 1, Ebd 2, Ebd n:** Represent embeddings corresponding to the residual blocks. * **Embedding:** Label indicating the transformation from Res to Ebd. * **Ebd Avg:** Average of the embeddings. * **Cos Similarity:** Cosine similarity calculation applied after averaging embeddings. * **+0.5, +0.1, +0.3:** Reward values associated with different aspects. * **Format Reward, Math Acc Reward, Diversity Reward:** Components of the reward system. * **GTE-Qwen2:** Model used for embedding. * **GRPO:** Overall process name. ### Detailed Analysis or ### Content Details 1. **GSM8K Sample:** The process begins with a GSM8K sample. 2. **Rollout:** The sample undergoes a rollout process, influenced by Qwen2.5 (1.5B). 3. **Residual Blocks and Embeddings:** * There are 'n' parallel paths, each consisting of a residual block (Res) followed by an embedding (Ebd). * The paths are labeled as Res 1 -> Ebd 1, Res 2 -> Ebd 2, and Res n -> Ebd n. 4. **Embedding Averaging:** * The embeddings from each path (Ebd 1, Ebd 2, ..., Ebd n) are averaged to produce "Ebd Avg". 5. **Cosine Similarity:** * Cosine similarity is calculated after the embedding averaging step. 6. **Reward System:** * The reward system consists of three components: Format Reward, Math Acc Reward, and Diversity Reward. * These rewards are combined (indicated by "+" symbols). 7. **Reward Values:** * Format Reward is associated with a value of +0.5 (red). * Math Acc Reward is associated with a value of +0.1 (green). * Diversity Reward is associated with a value of +0.3 (yellow). 8. **Model Attribution:** * The embedding process is attributed to GTE-Qwen2. ### Key Observations * The diagram illustrates a parallel processing approach with multiple residual blocks and embeddings. * Embedding averaging and cosine similarity calculations are key steps in the process. * The reward system combines multiple factors, with Format Reward having the highest associated value (+0.5). ### Interpretation The diagram describes a process (GRPO) for evaluating and rewarding the performance of a model (likely Qwen2.5) on the GSM8K dataset. The model generates multiple outputs (rollouts), which are then processed through residual blocks and converted into embeddings. These embeddings are averaged, and cosine similarity is calculated, possibly to measure the similarity between different outputs. The final reward is a combination of format correctness, mathematical accuracy, and diversity, suggesting that the goal is to generate solutions that are not only correct but also varied in their approach. The higher weight given to "Format Reward" suggests that the output format is a critical aspect of the evaluation.