## Chart Type: Line Graphs ### Overview The image contains two line graphs comparing the performance of different models on two tasks: "Pass @N (2B)" and "Solving Hard Questions". The x-axis represents the number of samples (N) on a logarithmic scale (base 2), and the y-axis represents accuracy or success rate. The graphs compare the performance of models SFT, RFT, ORM-RL, and PAV-RL (in the first graph) and ORM and PAV (in the second graph). ### Components/Axes **Graph (a): Pass @N (2B)** * **Title:** Pass @N (2B) * **X-axis:** N (Number of samples). Scale: 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, 2⁷ * **Y-axis:** Accuracy. Scale: 0.2, 0.3, 0.4, 0.5 * **Legend (top-right):** * SFT (light blue, dashed line with circle markers) * RFT (light blue, dashed line with x markers) * ORM-RL (light red, dashed line with square markers) * PAV-RL (dark orange, solid line with star markers) **Graph (b): Solving Hard Questions** * **Title:** Solving Hard Questions * **X-axis:** N (Number of samples). Scale: 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, 2⁷, 2⁸ * **Y-axis:** Success Rate on Problems Unsolved by SFT @256. Scale: 0.00, 0.05, 0.10, 0.15 * **Legend (top-right):** * ORM (light green, dashed line with square markers) * PAV (light red, solid line with star markers) ### Detailed Analysis **Graph (a): Pass @N (2B)** * **SFT (light blue, dashed line with circle markers):** Starts at approximately 0.18 accuracy at N=2¹ and increases to approximately 0.42 at N=2⁷. The trend is generally upward. * **RFT (light blue, dashed line with x markers):** Starts at approximately 0.17 accuracy at N=2¹ and increases to approximately 0.40 at N=2⁷. The trend is generally upward. * **ORM-RL (light red, dashed line with square markers):** Starts at approximately 0.20 accuracy at N=2¹ and increases to approximately 0.39 at N=2⁷. The trend is generally upward. * **PAV-RL (dark orange, solid line with star markers):** Starts at approximately 0.28 accuracy at N=2¹ and increases to approximately 0.50 at N=2⁷. The trend is generally upward. **Graph (b): Solving Hard Questions** * **ORM (light green, dashed line with square markers):** Starts at approximately 0.00 at N=2¹ and increases to approximately 0.025 at N=2⁸. The trend is slightly upward. * **PAV (light red, solid line with star markers):** Starts at approximately 0.02 at N=2¹, increases sharply to approximately 0.08 at N=2⁴, and then continues to increase to approximately 0.15 at N=2⁸. The trend is upward. ### Key Observations * In Graph (a), PAV-RL consistently outperforms the other models (SFT, RFT, ORM-RL) in terms of accuracy. * In Graph (b), PAV significantly outperforms ORM in terms of the success rate on problems unsolved by SFT. * Both graphs show an increase in performance (accuracy or success rate) as the number of samples (N) increases. * The shaded regions around the lines likely represent confidence intervals or standard deviations, indicating the uncertainty in the performance estimates. ### Interpretation The data suggests that the PAV-RL model is more effective than SFT, RFT, and ORM-RL for the "Pass @N (2B)" task. Additionally, the PAV model is more successful at solving hard questions that SFT cannot solve, compared to the ORM model. The increasing performance with larger N values indicates that all models benefit from more data. The confidence intervals provide a measure of the reliability of these conclusions. The "Solving Hard Questions" graph indicates that PAV is significantly better at addressing problems that SFT struggles with.

## Charts: Model Performance Comparison ### Overview The image presents two line charts comparing the performance of different models (SFT, RFT, ORM-RL, PAV-RL, ORM, PAV) across varying values of 'N'. The left chart (a) displays "Pass @N (2B)" which appears to be an accuracy metric, while the right chart (b) shows "Success Rate on Problems Unsolved by SFT @256". Both charts use a logarithmic scale for the x-axis (N). ### Components/Axes **Chart (a): Pass @N (2B)** * **X-axis:** N, ranging from 2¹ to 2⁷ (approximately 2 to 128). * **Y-axis:** Accuracy, ranging from 0.15 to 0.5. * **Data Series:** * SFT (Light Blue) * RFT (Pale Blue) * ORM-RL (Light Orange) * PAV-RL (Dark Orange) * **Legend:** Located in the top-left corner. **Chart (b): Solving Hard Questions** * **X-axis:** N, ranging from 2¹ to 2⁸ (approximately 2 to 256). * **Y-axis:** Success Rate on Problems Unsolved by SFT @256, ranging from 0 to 0.15. * **Data Series:** * ORM (Light Green) * PAV (Dark Orange) * **Legend:** Located in the top-left corner. ### Detailed Analysis or Content Details **Chart (a): Pass @N (2B)** * **SFT (Light Blue):** The line slopes upward, starting at approximately 0.17 at N=2¹ and reaching approximately 0.35 at N=2⁷. * **RFT (Pale Blue):** The line also slopes upward, starting at approximately 0.18 at N=2¹ and reaching approximately 0.42 at N=2⁷. * **ORM-RL (Light Orange):** The line slopes upward more steeply than SFT and RFT, starting at approximately 0.22 at N=2¹ and reaching approximately 0.45 at N=2⁷. * **PAV-RL (Dark Orange):** The line has the steepest upward slope, starting at approximately 0.25 at N=2¹ and reaching approximately 0.5 at N=2⁷. **Chart (b): Solving Hard Questions** * **ORM (Light Green):** The line is relatively flat, starting at approximately 0.02 at N=2¹ and reaching approximately 0.03 at N=2⁸. * **PAV (Dark Orange):** The line slopes upward, starting at approximately 0.02 at N=2¹ and reaching approximately 0.15 at N=2⁸. The increase is more pronounced from N=2⁵ onwards. ### Key Observations * In Chart (a), PAV-RL consistently outperforms all other models across all values of N. ORM-RL performs better than RFT and SFT. * In Chart (b), PAV demonstrates a significant increase in success rate as N increases, while ORM remains relatively stable. * The shaded areas around the lines in both charts represent confidence intervals or standard deviations, indicating the variability in the results. ### Interpretation The data suggests that models utilizing Reinforcement Learning (RL) – specifically ORM-RL and PAV-RL – demonstrate superior performance in the "Pass @N (2B)" task (Chart a) compared to models trained with Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RFT). PAV-RL consistently achieves the highest accuracy. Chart (b) highlights the ability of the PAV model to solve problems that are initially unsolved by the SFT model, and this ability increases with larger values of N. This suggests that PAV benefits from increased computational resources or a larger search space. The relatively flat performance of ORM indicates it does not scale as effectively as PAV in solving these "hard" problems. The logarithmic scale of the x-axis (N) is crucial. It indicates that the performance gains are more significant at lower values of N, and the rate of improvement diminishes as N increases. This could be due to diminishing returns or the inherent difficulty of the task. The confidence intervals suggest that the observed differences in performance are statistically significant, but there is still some degree of uncertainty. The two charts together suggest a trade-off between overall accuracy (Chart a) and the ability to solve particularly challenging problems (Chart b).

## [Chart Type]: Dual Line Charts Comparing Model Performance ### Overview The image contains two side-by-side line charts, labeled (a) and (b), comparing the performance of different methods (likely language model training or decoding strategies) on two distinct metrics as a function of the number of samples or attempts, denoted by N. The charts use a logarithmic scale for the x-axis (N). Both charts include shaded regions around the lines, indicating confidence intervals or variance. ### Components/Axes **Chart (a): "Pass @N (2B)"** * **Title:** Pass @N (2B) * **Y-axis:** Label: "Accuracy". Scale: Linear, from 0.2 to 0.5, with major ticks at 0.2, 0.3, 0.4, 0.5. * **X-axis:** Label: "N". Scale: Logarithmic base 2. Ticks: 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, 2⁷. * **Legend:** Located in the bottom-right corner. Contains four entries: 1. `SFT` - Light blue line with circle markers. 2. `RFT` - Blue line with 'x' markers. 3. `ORM-RL` - Red-orange line with square markers. 4. `PAV-RL` - Orange line with star markers. **Chart (b): "Solving Hard Questions"** * **Title:** Solving Hard Questions * **Y-axis:** Label: "Success Rate on Problems Unsolved by SFT @256". Scale: Linear, from 0.00 to 0.15, with major ticks at 0.00, 0.05, 0.10, 0.15. * **X-axis:** Label: "N". Scale: Logarithmic base 2. Ticks: 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, 2⁷, 2⁸. * **Legend:** Located in the top-left corner. Contains two entries: 1. `ORM` - Green line with square markers. 2. `PAV` - Orange line with star markers. ### Detailed Analysis **Chart (a) Data Points & Trends:** * **Trend Verification:** All four lines show a clear upward trend, indicating that accuracy increases as N increases. The slope is steepest for PAV-RL and ORM-RL. * **PAV-RL (Orange, Stars):** Highest performance across all N. Starts at ~0.28 (N=2¹) and rises steadily to ~0.50 (N=2⁷). * **ORM-RL (Red-Orange, Squares):** Second highest. Starts at ~0.20 (N=2¹) and rises to ~0.42 (N=2⁷). * **RFT (Blue, 'x'):** Third. Starts at ~0.18 (N=2¹) and rises to ~0.45 (N=2⁷). Note: At N=2⁷, it appears to slightly surpass ORM-RL. * **SFT (Light Blue, Circles):** Lowest performance. Starts at ~0.15 (N=2¹) and rises to ~0.45 (N=2⁷), converging with RFT at the highest N. **Chart (b) Data Points & Trends:** * **Trend Verification:** Both lines show an upward trend. PAV exhibits a pronounced, accelerating (convex) curve, while ORM shows a shallow, near-linear increase. * **PAV (Orange, Stars):** Shows dramatic improvement. Starts at ~0.02 (N=2¹), increases slowly until N=2³ (~0.06), then accelerates sharply, reaching ~0.15 at N=2⁸. * **ORM (Green, Squares):** Shows modest, steady improvement. Starts at ~0.00 (N=2¹) and rises linearly to only ~0.03 at N=2⁸. ### Key Observations 1. **Performance Hierarchy:** In chart (a), the order from best to worst is consistently PAV-RL > ORM-RL ≈ RFT > SFT for most N, with convergence between RFT and SFT at high N. 2. **Scaling Advantage:** PAV(-RL) demonstrates superior scaling properties in both charts. Its advantage over other methods becomes significantly more pronounced as N increases, especially in the "Solving Hard Questions" metric. 3. **Hard Problem Specialization:** Chart (b) reveals a massive performance gap between PAV and ORM on problems that the baseline SFT method fails to solve even with 256 samples. PAV's success rate on these hard problems is approximately 5 times higher than ORM's at N=2⁸. 4. **Variance:** The shaded confidence intervals are relatively narrow for all lines, suggesting the reported trends are statistically stable. ### Interpretation These charts likely evaluate methods for improving the problem-solving capability of a 2-billion parameter (2B) language model through techniques like Reinforcement Learning (RL) or different sampling strategies. "Pass @N" measures the probability that at least one of N generated samples is correct. The data strongly suggests that the **PAV** (and its RL variant PAV-RL) method is significantly more effective than the alternatives (**SFT**, **RFT**, **ORM**). Its key advantage is not just in overall accuracy (chart a), but specifically in its ability to solve *hard problems* that stump the baseline model (chart b). The accelerating curve in chart (b) indicates that PAV's benefit is not linear; it becomes disproportionately more valuable as you are allowed more attempts (higher N). This implies PAV is better at exploring the solution space or has learned a more robust reasoning strategy. The close performance of RFT and SFT at high N in chart (a) suggests that simple fine-tuning (SFT) and perhaps Rejection Fine-Tuning (RFT) have similar asymptotic limits, which PAV and ORM-RL surpass. The charts collectively argue for the superiority of the PAV approach, particularly for challenging tasks where brute-force sampling (increasing N) is computationally expensive.

## Line Graphs: Performance Comparison Across Methods ### Overview The image contains two line graphs comparing the performance of different methods (SFT, RFT, ORM-RL, PAV-RL) on two tasks: "Pass @N (2B)" and "Solving Hard Questions". Both graphs show performance metrics (accuracy and success rate) as a function of problem size (N), with N increasing exponentially from 2¹ to 2⁷ (graph a) and 2⁸ (graph b). Confidence intervals are represented by shaded regions around each line. --- ### Components/Axes #### Graph (a): Pass @N (2B) - **X-axis**: N (problem size), logarithmic scale from 2¹ to 2⁷. - **Y-axis**: Accuracy (0.0 to 0.5). - **Legend**: - SFT: Dashed blue line with circle markers. - RFT: Dotted blue line with cross markers. - ORM-RL: Dashed orange line with square markers. - PAV-RL: Solid orange line with star markers. - **Shading**: Confidence intervals (light gray for SFT/RFT, light orange for ORM-RL/PAV-RL). #### Graph (b): Solving Hard Questions - **X-axis**: N (problem size), logarithmic scale from 2¹ to 2⁸. - **Y-axis**: Success Rate on Problems Unsolved by SFT @256 (0.0 to 0.15). - **Legend**: - ORM: Dashed green line with square markers. - PAV: Solid orange line with star markers. - **Shading**: Confidence intervals (light green for ORM, light orange for PAV). --- ### Detailed Analysis #### Graph (a): Pass @N (2B) - **PAV-RL**: - Starts at ~0.25 accuracy at N=2¹, rising steadily to ~0.5 at N=2⁷. - Confidence interval widens slightly at larger N. - **ORM-RL**: - Begins at ~0.15, increases to ~0.4 at N=2⁷. - Confidence interval remains narrow. - **SFT**: - Starts at ~0.1, reaches ~0.35 at N=2⁷. - Confidence interval widens significantly at larger N. - **RFT**: - Starts at ~0.1, peaks at ~0.3 at N=2⁶, then plateaus. - Confidence interval narrows at smaller N but widens at larger N. #### Graph (b): Solving Hard Questions - **PAV**: - Starts at ~0.02 at N=2¹, rises sharply to ~0.15 at N=2⁸. - Confidence interval widens at larger N. - **ORM**: - Starts at ~0.01, increases gradually to ~0.02 at N=2⁸. - Confidence interval remains narrow. --- ### Key Observations 1. **PAV-RL/PAV** consistently outperforms other methods in both tasks, with the largest gap observed at larger N (e.g., N=2⁷ in graph a, N=2⁸ in graph b). 2. **ORM-RL** shows moderate improvement in graph (a) but minimal progress in graph (b), suggesting task-specific limitations. 3. **SFT/RFT** underperform PAV-RL in graph (a) but show comparable trends to ORM-RL in graph (b). 4. Confidence intervals widen for most methods as N increases, indicating greater uncertainty at larger problem sizes. --- ### Interpretation The data demonstrates that **PAV-based methods** (PAV-RL and PAV) are significantly more effective than alternatives in both tasks, particularly as problem size grows. This suggests that PAV's approach (likely involving problem decomposition or hierarchical reasoning) scales better with complexity. - **Graph (a)**: PAV-RL's steady improvement implies robustness in handling increasing problem sizes, while SFT/RFT's plateauing performance highlights limitations in generalization. - **Graph (b)**: PAV's sharp rise indicates superior ability to solve harder questions, whereas ORM's stagnation suggests it struggles with tasks requiring advanced reasoning beyond its training. The widening confidence intervals at larger N across most methods imply that performance becomes less predictable as problem complexity increases, emphasizing the need for further research into scalable reasoning frameworks.