## Line Chart: On-Policy GRPO using πk ### Overview The image is a line chart that displays the performance of "On-Policy GRPO with πref swap" over iterations. The y-axis represents "Pass@1", a performance metric, and the x-axis represents "Iteration". The chart includes vertical dashed lines indicating specific swap events. ### Components/Axes * **Title:** On-Policy GRPO using πk * **X-axis:** * Label: Iteration * Scale: 0 to 1000, with markers at 0, 200, 400, 600, 800, and 1000. * **Y-axis:** * Label: Pass@1 * Scale: 0.10 to 0.45, with markers at 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, and 0.45. * **Legend:** Located in the top-left corner. * Blue Line: On-Policy GRPO with πref swap * **Vertical Dashed Lines:** Three vertical dashed lines, colored light red, are present at approximately x=250, x=550, and x=750. * The first line at x=250 is labeled "grpo-plus-v1-l1". * The second line at x=550 is labeled "grpo-plus-v1-l1-swap-1". * The third line at x=750 is labeled "grpo-plus-v1-l1-swap-2". ### Detailed Analysis * **On-Policy GRPO with πref swap (Blue Line):** * Trend: Initially, the line slopes upward from approximately (0, 0.21) to (100, 0.37). It then plateaus around 0.38-0.40 until the first vertical line. After the first vertical line, the line drops to approximately 0.28, then rises again to approximately 0.39 before dropping sharply to approximately 0.10 at the second vertical line. After the second vertical line, the line rises sharply to approximately 0.38, then plateaus around 0.41-0.44 until the third vertical line. After the third vertical line, the line drops to approximately 0.39, then rises again to approximately 0.44. * Data Points: * (0, 0.21) * (100, 0.37) * (200, 0.39) * (300, 0.36) * (400, 0.28) * (500, 0.38) * (550, 0.10) * (600, 0.38) * (700, 0.41) * (800, 0.39) * (900, 0.41) * (1000, 0.44) ### Key Observations * The "On-Policy GRPO with πref swap" performance, as measured by "Pass@1", generally increases over iterations, but experiences significant drops at or near the "swap" events indicated by the vertical dashed lines. * The performance recovers after each swap event, suggesting the algorithm adapts. ### Interpretation The chart illustrates the impact of "πref swap" events on the performance of the "On-Policy GRPO" algorithm. The drops in "Pass@1" at each swap indicate a temporary disruption in performance, possibly due to the change in the reference policy. However, the subsequent recovery suggests that the algorithm is able to adapt and continue learning, eventually reaching a higher performance level. The swap events are likely part of an exploration strategy, where the algorithm is forced to explore new regions of the policy space.

## Line Chart: On-Policy GRPO using πk ### Overview This image presents a line chart illustrating the performance of an On-Policy GRPO algorithm using πk, measured by the Pass@1 metric, across iterations. The chart shows fluctuations in performance, with distinct dips coinciding with labeled "swap" events. ### Components/Axes * **Title:** "On-Policy GRPO using πk" - positioned at the top-center of the chart. * **X-axis:** "Iteration" - ranging from 0 to 1000, with tick marks at approximately 0, 200, 400, 600, 800, and 1000. Specific labels are: "grpo-plus-v1-1", "grpo-plus-v1-1-swap-1", "grpo-plus-v1-1-swap-2", "grpo-plus-v1-1-swap-3". * **Y-axis:** "Pass@1" - ranging from 0.10 to 0.45, with tick marks at 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, and 0.45. * **Data Series:** A single line labeled "On-Policy GRPO with πref swap" - colored in a dark blue. * **Vertical Dashed Lines:** Four vertical dashed lines, colored in red and teal, are present. These lines are positioned at approximately iterations 200, 400, 600, and 800, and are labeled with "grpo-plus-v1-1-swap-1", "grpo-plus-v1-1-swap-2", "grpo-plus-v1-1-swap-3". ### Detailed Analysis The line representing "On-Policy GRPO with πref swap" starts at approximately (0, 0.22). It then exhibits a steep upward trend, reaching a peak of approximately (200, 0.41). Following this peak, the line declines sharply to a low of approximately (400, 0.30), coinciding with the first dashed line. It then rises again to a peak of approximately (500, 0.42), before declining to a low of approximately (600, 0.36) at the second dashed line. The line then rises to a peak of approximately (700, 0.43), and declines to a low of approximately (800, 0.38) at the third dashed line. Finally, the line shows a slight upward trend, stabilizing around (900, 0.42) and (1000, 0.42). Here's a more detailed breakdown of approximate data points: * (0, 0.22) * (100, 0.35) * (200, 0.41) * (300, 0.38) * (400, 0.30) * (500, 0.42) * (600, 0.36) * (700, 0.43) * (800, 0.38) * (900, 0.42) * (1000, 0.42) ### Key Observations The chart demonstrates a cyclical pattern in the Pass@1 metric. Each cycle consists of an increase in performance, followed by a sharp decline coinciding with a "swap" event. The magnitude of the performance decline appears to be relatively consistent across the observed swaps. The overall trend suggests that the algorithm is capable of learning and improving, but is periodically disrupted by the swap events. ### Interpretation The data suggests that the "πref swap" mechanism, while potentially beneficial in some contexts, introduces instability into the learning process. The periodic dips in performance indicate that the swaps disrupt the algorithm's current state, requiring it to re-adapt. The consistent pattern of decline following each swap suggests that the swap process itself may be a source of inefficiency. The algorithm appears to recover from these disruptions, but the recovery process introduces a cyclical pattern in performance. Further investigation is needed to understand the underlying cause of these disruptions and to explore strategies for mitigating their impact. The chart highlights a trade-off between exploration (through swaps) and exploitation (through continued learning). The optimal balance between these two strategies likely depends on the specific characteristics of the environment and the algorithm's learning parameters.

## Line Chart: On-Policy GRPO using π_k ### Overview The image displays a line chart tracking the performance of a reinforcement learning algorithm, specifically "On-Policy GRPO with π_ref swap," over 1000 training iterations. The chart plots the "Pass@1" metric against the iteration number, showing significant volatility with an overall upward trend. Four vertical dashed lines mark specific iteration points, each annotated with a label below the x-axis. ### Components/Axes * **Chart Title:** "On-Policy GRPO using π_k" (centered at the top). * **Y-Axis:** * **Label:** "Pass@1" (rotated vertically on the left). * **Scale:** Linear, ranging from 0.10 to 0.45. Major tick marks are at 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, and 0.45. * **X-Axis:** * **Label:** "Iteration" (centered at the bottom). * **Scale:** Linear, ranging from 0 to 1000. Major tick marks are at 0, 200, 400, 600, 800, and 1000. * **Legend:** * **Position:** Top-left corner of the plot area. * **Content:** A blue line with a circle marker labeled "On-Policy GRPO with π_ref swap". * **Data Series:** A single blue line with circular markers at each data point. * **Annotations:** Four vertical, red, dashed lines extending from the x-axis to the top of the plot. Each is labeled with blue text below the x-axis: 1. At approximately Iteration 250: `grpo-plus-v1-11` 2. At approximately Iteration 350: `grpo-plus-v1-11-swap-1` 3. At approximately Iteration 500: `grpo-plus-v1-11-swap-2` 4. At approximately Iteration 700: `grpo-plus-v1-11-swap-3` ### Detailed Analysis **Trend Verification:** The blue line shows a general upward trend from the start to the end of the plotted iterations, but with high volatility. It rises sharply initially, enters a period of fluctuation, experiences a dramatic drop, recovers, and then continues a more gradual, fluctuating ascent. **Approximate Data Points (Pass@1 vs. Iteration):** * **Start (Iteration ~0):** ~0.21 * **Initial Rise:** Rapid increase to ~0.35 by Iteration ~50. * **First Plateau/Fluctuation:** Hovers between ~0.38 and ~0.40 from Iteration ~100 to ~250. * **First Annotation (`grpo-plus-v1-11` at ~250):** Value is ~0.38. * **Post-250 Fluctuation:** Dips to ~0.36, recovers to ~0.41, then drops sharply to ~0.28 around Iteration ~380. * **Second Annotation (`grpo-plus-v1-11-swap-1` at ~350):** Value is on a downward slope, approximately ~0.36. * **Recovery and Second Peak:** Recovers to ~0.41 by Iteration ~450. * **Third Annotation (`grpo-plus-v1-11-swap-2` at ~500):** This coincides with the most dramatic feature—a precipitous drop to the chart's minimum value of ~0.10. * **Post-500 Recovery:** Extremely sharp recovery back to ~0.42 by Iteration ~550. * **Fourth Annotation (`grpo-plus-v1-11-swap-3` at ~700):** Value is near the chart's maximum, approximately ~0.44. * **Final Segment:** After a dip to ~0.39 around Iteration ~750, the line trends upward with fluctuations, ending at approximately ~0.44 at Iteration ~1000. ### Key Observations 1. **Extreme Volatility at Swap-2:** The most notable event is the catastrophic drop in performance (Pass@1 from ~0.41 to ~0.10) at the iteration marked `grpo-plus-v1-11-swap-2` (~500). This is immediately followed by an equally sharp recovery. 2. **Correlation with Annotations:** Performance dips are observed around or shortly after each annotated "swap" event (especially at ~380 and ~500), suggesting these events (likely policy or reference model swaps) introduce instability. 3. **Overall Positive Trend:** Despite the severe mid-training collapse, the algorithm demonstrates resilience, recovering and ultimately achieving a higher Pass@1 score (~0.44) at the end of the run than at any prior point. 4. **Performance Range:** The Pass@1 metric varies widely, from a low of ~0.10 to a high of ~0.44, indicating high sensitivity to the training process or the specific swap events. ### Interpretation This chart visualizes the training dynamics of an on-policy reinforcement learning algorithm (GRPO) that involves periodic swaps of a reference policy (π_ref). The "Pass@1" metric likely measures task success rate. The data suggests that the **policy swap events are critical points of instability**. The swap at iteration 500 (`swap-2`) caused a near-total collapse in performance, which could indicate a severe mismatch between the new reference policy and the current agent policy, or a disruptive change in the optimization landscape. However, the system's ability to rapidly recover from this collapse and continue improving is a sign of robustness. The **upward trend** implies that, despite these disruptive events, the learning process is effective over the long term. The final performance is the highest observed, suggesting the swaps, while destabilizing in the short term, may ultimately be beneficial for escaping local optima or adapting the policy. The pattern of "dip and recover" after each swap (most dramatically after `swap-2`) is a key characteristic of this training run. An investigator would want to examine the algorithmic details of the "swap" operation and the conditions at iteration 500 to understand the cause of the extreme drop.

## Line Graph: On-Policy GRPO using π_k ### Overview The image depicts a line graph tracking the performance of an "On-Policy GRPO with π_ref swap" algorithm over 1000 iterations. The y-axis measures "Pass@1" (a metric likely representing task success rate), while the x-axis represents training iterations. The graph includes a blue data line and four red dashed vertical markers at specific iteration points. ### Components/Axes - **Title**: "On-Policy GRPO using π_k" - **X-axis**: "iteration" (0 to 1000, linear scale) - **Y-axis**: "Pass@1" (0.10 to 0.45, linear scale) - **Legend**: Located in the top-right corner, labeled "On-Policy GRPO with π_ref swap" (blue line) - **Red Dashed Lines**: Four vertical markers at iterations 200, 400, 600, and 800, labeled: - "gpo-plus-v1-11-swap-1" (200) - "gpo-plus-v1-11-swap-2" (400) - "gpo-plus-v1-11-swap-3" (600) - "gpo-plus-v1-11-swap-4" (800) ### Detailed Analysis - **Blue Line (On-Policy GRPO with π_ref swap)**: - Starts at ~0.20 at iteration 0. - Rises sharply to ~0.40 by iteration 100. - Peaks at ~0.45 near iteration 200 (coinciding with "gpo-plus-v1-11-swap-1"). - Drops abruptly to ~0.10 at iteration 400 ("gpo-plus-v1-11-swap-2"). - Recovers to ~0.40 by iteration 600, then fluctuates between ~0.38–0.44 until iteration 1000. - Notable instability at iteration 400 (sharp dip) and minor dips at 600 and 800. ### Key Observations 1. **Initial Growth**: Rapid improvement in performance during early iterations (0–200). 2. **Catastrophic Drop**: A 70% performance drop at iteration 400 ("gpo-plus-v1-11-swap-2"), suggesting a critical failure or parameter adjustment. 3. **Recovery and Stability**: Partial recovery after iteration 400, with sustained performance (~0.38–0.44) in later iterations. 4. **Red Markers**: Align with labeled "swap" events, indicating potential hyperparameter changes or training phases. ### Interpretation The graph demonstrates the GRPO algorithm's sensitivity to parameter swaps (π_ref). The catastrophic drop at iteration 400 ("gpo-plus-v1-11-swap-2") suggests that the swap introduced instability, possibly due to overfitting or misalignment with the policy. The recovery phase implies adaptive adjustments, but the persistent fluctuations highlight challenges in maintaining stability during training. The red markers likely denote experimental interventions, with the 400-iteration swap being the most disruptive. This pattern underscores the importance of careful hyperparameter tuning in reinforcement learning algorithms.