## Chart: Pass@K Performance Comparison ### Overview The image presents three line charts comparing the "pass@K" performance of different language models: DS-R1-Qwen-32B, Qwen3-8B, and GPT-OSS-20B. Each chart plots "pass@K" (y-axis) against "K" (x-axis) for two methods: "Vanilla N" and "TopM". The charts show how the performance changes as K increases. ### Components/Axes * **Titles:** * Left Chart: DS-R1-Qwen-32B * Middle Chart: Qwen3-8B * Right Chart: GPT-OSS-20B * **X-axis:** * Label: K * Scale: 1, 3, 5, 7, 9, 11, 13, 15 * **Y-axis:** * Label: pass@K * Scale: 40, 45, 50, 55, 60 * **Legend:** Located in the center-left of the image. * Vanilla N (Orange Line with Triangle Markers) * TopM (Blue Line with Circle Markers) ### Detailed Analysis **Chart 1: DS-R1-Qwen-32B** * **Vanilla N (Orange):** The line starts at approximately 49 at K=1, rises sharply to approximately 56 at K=5, and then gradually increases to approximately 61 at K=15. * **TopM (Blue):** The line starts at approximately 49 at K=1, rises sharply to approximately 58 at K=5, and then gradually increases to approximately 63 at K=15. **Chart 2: Qwen3-8B** * **Vanilla N (Orange):** The line starts at approximately 42 at K=1, rises sharply to approximately 49 at K=5, and then gradually increases to approximately 53 at K=15. * **TopM (Blue):** The line starts at approximately 42 at K=1, rises sharply to approximately 52 at K=5, and then gradually increases to approximately 55 at K=15. **Chart 3: GPT-OSS-20B** * **Vanilla N (Orange):** The line starts at approximately 40 at K=1, rises sharply to approximately 54 at K=5, and then gradually increases to approximately 60 at K=15. * **TopM (Blue):** The line starts at approximately 43 at K=1, rises sharply to approximately 56 at K=5, and then gradually increases to approximately 61 at K=15. ### Key Observations * In all three charts, the "TopM" method (blue line) consistently outperforms the "Vanilla N" method (orange line) across all values of K. * The performance gain from increasing K diminishes as K gets larger. The curves flatten out after K=9. * The DS-R1-Qwen-32B model achieves the highest "pass@K" scores, followed by GPT-OSS-20B, and then Qwen3-8B. ### Interpretation The charts demonstrate the impact of the "TopM" sampling method on the "pass@K" performance of different language models. The data suggests that "TopM" consistently improves performance compared to the "Vanilla N" method. The diminishing returns as K increases indicate that there is a point beyond which increasing the number of samples (K) provides little additional benefit. The relative performance of the models suggests that DS-R1-Qwen-32B is the most effective among the three, followed by GPT-OSS-20B and Qwen3-8B. This could be due to differences in model size, architecture, or training data.

## Chart: pass@K vs. K for Different Models ### Overview The image presents three line charts, each depicting the relationship between 'pass@K' (y-axis) and 'K' (x-axis) for different language models: DS-R1-Owen-32B, Qwen3-8B, and GPT-OSS-20B. Each chart compares two data series: "Vanilla N" and "TopM". The charts visually demonstrate how the 'pass@K' metric changes as the value of 'K' increases for each model and sampling method. ### Components/Axes * **X-axis Label:** "K" - ranging from 1 to 15, with markers at integer values. * **Y-axis Label:** "pass@K" - ranging from approximately 40 to 65, with markers at intervals of 5. * **Chart Titles:** * "DS-R1-Owen-32B" (Top-left chart) * "Qwen3-8B" (Top-center chart) * "GPT-OSS-20B" (Top-right chart) * **Legend:** Located in the top-left corner of each chart. * "Vanilla N" - represented by an orange line with a circular marker. * "TopM" - represented by a blue line with a circular marker. ### Detailed Analysis or Content Details **Chart 1: DS-R1-Owen-32B** * **Vanilla N:** The line slopes upward, starting at approximately 50 at K=1, reaching around 58 at K=7, and plateauing around 62-63 from K=9 onwards. Approximate data points: (1, 50), (3, 53), (5, 56), (7, 58), (9, 61), (11, 62), (13, 62), (15, 63). * **TopM:** The line also slopes upward, but starts higher at approximately 53 at K=1, reaches around 63 at K=7, and plateaus around 64-65 from K=9 onwards. Approximate data points: (1, 53), (3, 58), (5, 61), (7, 63), (9, 64), (11, 64), (13, 65), (15, 65). **Chart 2: Qwen3-8B** * **Vanilla N:** The line slopes upward, starting at approximately 43 at K=1, reaching around 54 at K=7, and plateauing around 56-57 from K=9 onwards. Approximate data points: (1, 43), (3, 47), (5, 51), (7, 54), (9, 56), (11, 56), (13, 57), (15, 57). * **TopM:** The line slopes upward, starting at approximately 42 at K=1, reaching around 55 at K=7, and plateauing around 57-58 from K=9 onwards. Approximate data points: (1, 42), (3, 46), (5, 50), (7, 55), (9, 57), (11, 57), (13, 58), (15, 58). **Chart 3: GPT-OSS-20B** * **Vanilla N:** The line slopes upward, starting at approximately 41 at K=1, reaching around 52 at K=7, and plateauing around 55-56 from K=9 onwards. Approximate data points: (1, 41), (3, 45), (5, 49), (7, 52), (9, 55), (11, 55), (13, 56), (15, 56). * **TopM:** The line slopes upward, starting at approximately 44 at K=1, reaching around 56 at K=7, and plateauing around 58-60 from K=9 onwards. Approximate data points: (1, 44), (3, 48), (5, 52), (7, 56), (9, 58), (11, 59), (13, 60), (15, 60). ### Key Observations * In all three charts, "TopM" consistently outperforms "Vanilla N" across all values of K. * The performance gains from "Vanilla N" to "TopM" appear to diminish as K increases, with the lines flattening out and converging at higher K values. * DS-R1-Owen-32B generally achieves the highest 'pass@K' values, followed by GPT-OSS-20B, and then Qwen3-8B. * The rate of improvement in 'pass@K' with increasing K is steepest at lower K values (1-7) for all models and sampling methods. ### Interpretation The charts demonstrate the impact of different sampling methods ("Vanilla N" and "TopM") on the 'pass@K' metric for three different language models. 'pass@K' likely represents the percentage of times the model successfully completes a task when considering the top K possible outputs. The consistent outperformance of "TopM" suggests that selecting from the most probable outputs (TopM) leads to higher success rates compared to a more naive sampling approach ("Vanilla N"). The diminishing returns observed at higher K values indicate that beyond a certain point, increasing the number of considered outputs does not significantly improve performance. This suggests that the most informative outputs are concentrated within the top K possibilities, and exploring beyond that point yields diminishing benefits. The differences in overall 'pass@K' values between the models suggest varying levels of capability. DS-R1-Owen-32B appears to be the most capable model, followed by GPT-OSS-20B and Qwen3-8B, based on this metric. The charts provide valuable insights into the trade-offs between sampling methods and model performance, which can inform the selection of appropriate configurations for specific applications.

## Line Charts: Model Performance Comparison (pass@K) ### Overview The image displays three horizontally arranged line charts comparing the performance of two methods ("Vanilla N" and "TopM") across three different language models. The metric is "pass@K," plotted against the parameter "K." All charts show an increasing trend for both methods as K increases. ### Components/Axes * **Titles (Top-Center of each subplot):** * Left Chart: `DS-R1-Qwen-32B` * Middle Chart: `Qwen3-8B` * Right Chart: `GPT-OSS-20B` * **Y-Axis Label (Leftmost chart):** `pass@K` * **X-Axis Label (Centered below all charts):** `K` * **Legend (Bottom-right corner of the leftmost chart):** * `Vanilla N`: Orange line with plus (`+`) markers. * `TopM`: Blue line with circle (`o`) markers. * **Axis Scales:** * **X-Axis (K):** Linear scale from 1 to 15, with major ticks at every odd number (1, 3, 5, 7, 9, 11, 13, 15). * **Y-Axis (pass@K):** Linear scales differ per chart: * DS-R1-Qwen-32B: ~48 to ~63 * Qwen3-8B: ~42 to ~55 * GPT-OSS-20B: ~40 to ~61 ### Detailed Analysis **1. DS-R1-Qwen-32B (Left Chart)** * **Trend:** Both lines show a steep initial rise that gradually flattens. The `TopM` (blue) line is consistently above the `Vanilla N` (orange) line. * **Approximate Data Points:** * **K=1:** Vanilla N ≈ 49, TopM ≈ 49 (nearly identical start). * **K=3:** Vanilla N ≈ 55, TopM ≈ 56. * **K=7:** Vanilla N ≈ 59, TopM ≈ 62. * **K=15:** Vanilla N ≈ 63, TopM ≈ 63 (converge at the highest K). **2. Qwen3-8B (Middle Chart)** * **Trend:** Similar logarithmic growth pattern. The performance gap between `TopM` and `Vanilla N` is smaller here than in the first chart. * **Approximate Data Points:** * **K=1:** Vanilla N ≈ 43, TopM ≈ 44. * **K=5:** Vanilla N ≈ 49, TopM ≈ 51. * **K=11:** Vanilla N ≈ 53, TopM ≈ 54. * **K=15:** Vanilla N ≈ 55, TopM ≈ 55 (very close convergence). **3. GPT-OSS-20B (Right Chart)** * **Trend:** Consistent upward trend. `TopM` maintains a clear and steady lead over `Vanilla N` across the entire range of K. * **Approximate Data Points:** * **K=1:** Vanilla N ≈ 40, TopM ≈ 43. * **K=5:** Vanilla N ≈ 53, TopM ≈ 55. * **K=9:** Vanilla N ≈ 57, TopM ≈ 59. * **K=15:** Vanilla N ≈ 61, TopM ≈ 61 (converge at the final point). ### Key Observations 1. **Universal Superiority of TopM:** In all three models and at nearly every measured value of K, the `TopM` method yields a higher `pass@K` score than the `Vanilla N` method. 2. **Diminishing Returns:** The rate of improvement for `pass@K` slows as K increases for both methods, following a characteristic logarithmic curve. 3. **Convergence at High K:** For the DS-R1-Qwen-32B and Qwen3-8B models, the performance of the two methods becomes nearly indistinguishable at the highest measured K value (K=15). For GPT-OSS-20B, they converge at the final point. 4. **Model-Specific Scaling:** The absolute `pass@K` values and the size of the performance gap between methods vary by model. The DS-R1-Qwen-32B model shows the largest initial gap, while the Qwen3-8B model shows the smallest. ### Interpretation This data demonstrates the effectiveness of the `TopM` method over the `Vanilla N` baseline for improving the pass@K metric across diverse model architectures and sizes. The `pass@K` metric typically measures the probability that at least one of K generated samples is correct. The consistent advantage of `TopM` suggests it is a more reliable sampling or ranking strategy for generating correct outputs, especially when the budget for attempts (K) is low to moderate. The convergence at high K indicates that given enough attempts, the simpler `Vanilla N` method can eventually match the performance of the more sophisticated `TopM` method. Therefore, the primary value of `TopM` lies in **efficiency**—achieving higher success rates with fewer generations, which translates to reduced computational cost and latency. The variation across models implies that the benefit of `TopM` may depend on the underlying model's capabilities or training, being most pronounced in the DS-R1-Qwen-32B model in this comparison.

## Line Graphs: Model Performance Comparison Across Datasets ### Overview The image contains three line graphs comparing the performance of two models ("Vanilla N" and "TopM") across three datasets: "DS-R1-Qwen-32B", "Qwen3-8B", and "GPT-OSS-20B". Each graph plots "pass@K" (y-axis) against "K" (x-axis), showing how performance improves as K increases. Both models exhibit similar upward trends, with "TopM" consistently outperforming "Vanilla N" slightly. --- ### Components/Axes 1. **X-Axis (K)**: - Labeled "K" with discrete values: 1, 3, 5, 7, 9, 11, 13, 15. - Represents the number of top candidates considered for evaluation. 2. **Y-Axis (pass@K)**: - Labeled "pass@K" with values ranging from 40 to 65. - Measures the percentage of successful outcomes (e.g., correct predictions) within the top K candidates. 3. **Legends**: - Positioned in the **bottom-left** of each graph. - "Vanilla N" is represented by **orange triangles** (▲). - "TopM" is represented by **blue circles** (●). 4. **Graph Titles**: - Top-left corner of each graph specifies the dataset: - "DS-R1-Qwen-32B" - "Qwen3-8B" - "GPT-OSS-20B" --- ### Detailed Analysis #### DS-R1-Qwen-32B - **Vanilla N**: Starts at ~48 (K=1), rises sharply to ~63 (K=15). - **TopM**: Starts at ~49 (K=1), rises to ~63 (K=15). - **Trend**: Both lines converge at K=15, with TopM maintaining a ~1-2 point advantage throughout. #### Qwen3-8B - **Vanilla N**: Begins at ~42 (K=1), increases to ~54 (K=15). - **TopM**: Starts at ~43 (K=1), rises to ~55 (K=15). - **Trend**: TopM outperforms Vanilla N by ~1-2 points across all K values. #### GPT-OSS-20B - **Vanilla N**: Starts at ~44 (K=1), climbs to ~60 (K=15). - **TopM**: Begins at ~45 (K=1), reaches ~61 (K=15). - **Trend**: TopM maintains a ~1-2 point lead, with both models showing steep initial growth. --- ### Key Observations 1. **Consistent Performance Gap**: "TopM" outperforms "Vanilla N" by 1-2 points across all datasets and K values. 2. **Diminishing Returns**: Performance improvements slow as K increases, especially after K=9. 3. **Dataset Variability**: "GPT-OSS-20B" shows the largest absolute performance gap (~16 points at K=15), while "Qwen3-8B" has the smallest (~3 points at K=15). --- ### Interpretation The data suggests that "TopM" is a more effective model variant, likely due to architectural optimizations or training strategies. The marginal performance gap implies that while "TopM" is superior, the difference may not justify additional computational costs in all scenarios. The convergence of lines at higher K values indicates that both models saturate at similar performance ceilings, but "TopM" achieves this faster. This could reflect better candidate prioritization or filtering mechanisms in "TopM".