## Line Charts: Gemma Model Accuracy vs. N ### Overview The image contains three line charts comparing the accuracy of different Gemma models (2B, 9B, and 27B) across varying values of 'N'. Each chart plots the accuracy of three methods: ORM, PAV (ours), and Pass @N, against N, which is displayed on a logarithmic scale. The charts also highlight specific performance differences between the methods at certain N values. ### Components/Axes * **Titles:** * Left Chart: Gemma-2B * Middle Chart: Gemma-9B * Right Chart: Gemma-27B * **X-axis:** * Label: N * Scale: Logarithmic, with markers at 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, and 2⁷. * **Y-axis:** * Label: Accuracy * Scale: Linear, ranging from 0.1 to 0.4 (Gemma-2B), 0.4 to 0.6 (Gemma-9B), and 0.4 to 0.6 (Gemma-27B). * **Legend:** (Positioned at the top-left of each chart) * Blue dashed line with circles: ORM * Orange solid line with stars: PAV (ours) * Gray dashed line with squares: Pass @N * **Annotations:** * Arrows and text indicating performance differences (e.g., "5x", "10%", "2x", "1.5x", "8%") between methods at specific N values. * **Chart Labels:** * (a) - Left Chart * (b) - Middle Chart * (c) - Right Chart ### Detailed Analysis **Gemma-2B (Left Chart)** * **ORM (Blue):** The accuracy starts around 0.12 at N=2¹, increases to approximately 0.20 at N=2³, and then plateaus around 0.20 for higher N values. There is a shaded region around the line, indicating a confidence interval. * **PAV (Orange):** The accuracy starts around 0.15 at N=2¹, increases to approximately 0.25 at N=2⁵, and continues to increase slowly to about 0.28 at N=2⁷. There is a shaded region around the line, indicating a confidence interval. * **Pass @N (Gray):** The accuracy starts around 0.15 at N=2¹ and increases steadily to approximately 0.42 at N=2⁷. * **Annotation:** At N=2³, PAV is approximately 5 times more accurate than ORM. The accuracy of Pass @N is 10% higher than PAV at N=2⁷. **Gemma-9B (Middle Chart)** * **ORM (Blue):** The accuracy starts around 0.38 at N=2¹, increases to approximately 0.45 at N=2³, and then plateaus around 0.45 for higher N values. There is a shaded region around the line, indicating a confidence interval. * **PAV (Orange):** The accuracy starts around 0.40 at N=2¹, increases to approximately 0.55 at N=2⁵, and plateaus around 0.58 at N=2⁷. There is a shaded region around the line, indicating a confidence interval. * **Pass @N (Gray):** The accuracy starts around 0.40 at N=2¹ and increases steadily to approximately 0.68 at N=2⁷. * **Annotation:** At N=2³, PAV is approximately 2 times more accurate than ORM. The accuracy of Pass @N is 10% higher than PAV at N=2⁷. **Gemma-27B (Right Chart)** * **ORM (Blue):** The accuracy starts around 0.45 at N=2¹, increases to approximately 0.52 at N=2⁴, and then decreases slightly to about 0.50 at N=2⁷. There is a shaded region around the line, indicating a confidence interval. * **PAV (Orange):** The accuracy starts around 0.45 at N=2¹, increases to approximately 0.55 at N=2⁵, and plateaus around 0.58 at N=2⁷. There is a shaded region around the line, indicating a confidence interval. * **Pass @N (Gray):** The accuracy starts around 0.45 at N=2¹ and increases steadily to approximately 0.70 at N=2⁷. * **Annotation:** At N=2³, PAV is approximately 1.5 times more accurate than ORM. The accuracy of Pass @N is 8% higher than PAV at N=2⁷. ### Key Observations * For all three models, Pass @N consistently achieves the highest accuracy across all N values. * The performance gap between PAV and ORM decreases as the model size increases. * ORM's accuracy plateaus at lower N values compared to PAV and Pass @N. * The annotations highlight specific performance gains of PAV over ORM and Pass @N over PAV at certain N values. ### Interpretation The charts demonstrate the impact of model size (2B, 9B, 27B) and the value of N on the accuracy of different methods (ORM, PAV, Pass @N). The "Pass @N" method consistently outperforms the other two, suggesting it is the most effective approach for these models. The annotations emphasize the relative improvements of PAV over ORM and Pass @N over PAV, providing insights into the specific benefits of each method at different N values. The shaded regions around the lines indicate the uncertainty or variability in the accuracy measurements. The logarithmic scale on the x-axis suggests that the initial increases in N have a more significant impact on accuracy than later increases.

\n ## Chart: Accuracy vs. N for Different Models ### Overview This image presents three line charts, each comparing the accuracy of three methods – ORM, PAV (ours), and Pass@N – across varying values of N. Each chart corresponds to a different model: Gemma-2B, Gemma-9B, and Gemma-27B. The charts visualize how accuracy changes as N (likely representing the number of samples or attempts) increases. Shaded areas around each line represent confidence intervals. ### Components/Axes * **X-axis:** Labeled "N", with tick marks at 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, and 2⁷. * **Y-axis:** Labeled "Accuracy", with a scale ranging from approximately 0.1 to 0.65. * **Legend:** Located at the top-left of each chart, containing the labels: * ORM (Blue line with circle markers) * PAV (ours) (Orange line with circle markers) * Pass@N (Gray dashed line with diamond markers) * **Chart Titles:** * (a) Gemma-2B * (b) Gemma-9B * (c) Gemma-27B * **Annotations:** Each chart includes arrows indicating percentage improvements of PAV (ours) over ORM. ### Detailed Analysis or Content Details **Gemma-2B (Chart a):** * **ORM (Blue):** Starts at approximately 0.17 accuracy at N=2¹, rises to around 0.24 at N=2³, plateaus around 0.25-0.27 for N=2⁴ through N=2⁷. * **PAV (Orange):** Starts at approximately 0.21 accuracy at N=2¹, steadily increases to around 0.33 at N=2⁷. * **Pass@N (Gray):** Starts at approximately 0.23 accuracy at N=2¹, and increases steadily to approximately 0.41 at N=2⁷. * **Annotation:** An arrow indicates a 5x improvement of PAV over ORM at N=2³, and a 10% improvement at N=2⁷. **Gemma-9B (Chart b):** * **ORM (Blue):** Starts at approximately 0.32 accuracy at N=2¹, rises to around 0.38 at N=2³, plateaus around 0.38-0.42 for N=2⁴ through N=2⁷. * **PAV (Orange):** Starts at approximately 0.40 accuracy at N=2¹, steadily increases to around 0.56 at N=2⁷. * **Pass@N (Gray):** Starts at approximately 0.42 accuracy at N=2¹, and increases steadily to approximately 0.62 at N=2⁷. * **Annotation:** An arrow indicates a 2x improvement of PAV over ORM at N=2³, and a 10% improvement at N=2⁷. **Gemma-27B (Chart c):** * **ORM (Blue):** Starts at approximately 0.38 accuracy at N=2¹, rises to around 0.45 at N=2³, then decreases slightly to around 0.43 at N=2⁷. * **PAV (Orange):** Starts at approximately 0.48 accuracy at N=2¹, steadily increases to around 0.61 at N=2⁷. * **Pass@N (Gray):** Starts at approximately 0.50 accuracy at N=2¹, and increases steadily to approximately 0.64 at N=2⁷. * **Annotation:** An arrow indicates a 1.5x improvement of PAV over ORM at N=2³, and an 8% improvement at N=2⁷. ### Key Observations * **PAV consistently outperforms ORM** across all models and values of N. * **Pass@N generally achieves the highest accuracy** across all models and values of N. * The improvement of PAV over ORM appears to diminish as N increases, particularly for Gemma-27B. * The accuracy of ORM for Gemma-27B decreases slightly after N=2³, while PAV and Pass@N continue to improve. ### Interpretation The charts demonstrate the effectiveness of the "PAV (ours)" method in improving accuracy compared to "ORM" across different model sizes (Gemma-2B, Gemma-9B, and Gemma-27B). The "Pass@N" method consistently achieves the highest accuracy, suggesting it is the most robust approach. The annotations highlight the percentage improvements of PAV over ORM, providing a quantifiable measure of its benefit. The diminishing improvement of PAV over ORM as N increases suggests that the benefits of PAV are more pronounced at lower values of N. The slight decrease in ORM accuracy for Gemma-27B at higher N values could indicate overfitting or a limitation of the ORM method with larger models. The consistent upward trend of Pass@N suggests that increasing the number of samples (N) generally leads to higher accuracy, regardless of the model size. This is expected, as more samples provide more information for the model to learn from. The differences in accuracy between the models suggest that model size plays a significant role in performance. Gemma-27B generally exhibits higher accuracy than Gemma-9B and Gemma-2B, indicating that larger models have a greater capacity to learn and generalize.

## Line Charts: Gemma Model Accuracy Comparison (ORM vs. PAV vs. Pass@N) ### Overview The image contains three side-by-side line charts, labeled (a), (b), and (c), comparing the performance of three methods—ORM, PAV (ours), and Pass @N—across three different model sizes: Gemma-2B, Gemma-9B, and Gemma-27B. The charts plot "Accuracy" on the y-axis against a variable "N" on the x-axis, where N increases in powers of 2 from 2¹ to 2⁷. Each chart includes annotations highlighting relative performance improvements. ### Components/Axes * **Chart Layout:** Three subplots arranged horizontally. * **Titles:** Each subplot is titled with the model name: "Gemma-2B" (left), "Gemma-9B" (center), "Gemma-27B" (right). * **X-Axis:** Labeled "N". The axis markers are categorical, representing powers of two: 2¹, 2², 2³, 2⁴, 2⁵, 2⁶, 2⁷. * **Y-Axis:** Labeled "Accuracy". The scale varies per chart: * (a) Gemma-2B: 0.1 to 0.4 * (b) Gemma-9B: 0.4 to 0.6 * (c) Gemma-27B: 0.4 to 0.6 * **Legend:** Positioned in the top-left corner of each subplot. It defines three data series: * `ORM`: Blue dashed line with circular markers. * `PAV (ours)`: Orange solid line with star markers. * `Pass @N`: Gray dotted line with square markers. * **Annotations:** Each chart contains black dashed arrows and text annotations comparing the performance of PAV (ours) to ORM. ### Detailed Analysis #### **Chart (a): Gemma-2B** * **Trend Verification:** * **ORM (Blue):** Slopes gently upward from left to right. * **PAV (Orange):** Slopes upward more steeply than ORM. * **Pass @N (Gray):** Slopes upward most steeply of all three lines. * **Data Points (Approximate):** * **ORM:** Starts at ~0.12 (N=2¹), rises to ~0.20 (N=2⁷). * **PAV (ours):** Starts at ~0.15 (N=2¹), rises to ~0.28 (N=2⁷). * **Pass @N:** Starts at ~0.15 (N=2¹), rises to ~0.45 (N=2⁷). * **Annotations:** * A horizontal double-headed arrow between the ORM and PAV lines at N=2⁴ is labeled "5 ×". * A vertical double-headed arrow between the ORM and PAV lines at N=2⁷ is labeled "10%". #### **Chart (b): Gemma-9B** * **Trend Verification:** * **ORM (Blue):** Rises to a peak around N=2⁴ or 2⁵, then slightly declines. * **PAV (Orange):** Slopes steadily upward. * **Pass @N (Gray):** Slopes steadily upward, maintaining the highest accuracy. * **Data Points (Approximate):** * **ORM:** Starts at ~0.38 (N=2¹), peaks at ~0.46 (N=2⁴/2⁵), ends at ~0.45 (N=2⁷). * **PAV (ours):** Starts at ~0.40 (N=2¹), rises to ~0.54 (N=2⁷). * **Pass @N:** Starts at ~0.40 (N=2¹), rises to ~0.65 (N=2⁷). * **Annotations:** * A horizontal double-headed arrow between the ORM and PAV lines at N=2³ is labeled "2 ×". * A vertical double-headed arrow between the ORM and PAV lines at N=2⁷ is labeled "10%". #### **Chart (c): Gemma-27B** * **Trend Verification:** * **ORM (Blue):** Rises to a peak around N=2⁴, then declines more noticeably. * **PAV (Orange):** Slopes upward, peaking around N=2⁶ before a slight dip. * **Pass @N (Gray):** Slopes steadily upward. * **Data Points (Approximate):** * **ORM:** Starts at ~0.42 (N=2¹), peaks at ~0.52 (N=2⁴), drops to ~0.50 (N=2⁷). * **PAV (ours):** Starts at ~0.45 (N=2¹), peaks at ~0.58 (N=2⁶), ends at ~0.57 (N=2⁷). * **Pass @N:** Starts at ~0.45 (N=2¹), rises to ~0.68 (N=2⁷). * **Annotations:** * A horizontal double-headed arrow between the ORM and PAV lines at N=2³ is labeled "1.5 ×". * A vertical double-headed arrow between the ORM and PAV lines at N=2⁶ is labeled "8%". ### Key Observations 1. **Consistent Hierarchy:** In all three charts, the `Pass @N` method achieves the highest accuracy, followed by `PAV (ours)`, with `ORM` performing the lowest. 2. **Model Size Impact:** As the model size increases (2B → 9B → 27B), the absolute accuracy values for all methods increase significantly. The y-axis scale shifts upward. 3. **Diminishing Relative Gain:** The annotated relative improvement of PAV over ORM decreases as model size increases: "5 ×" for Gemma-2B, "2 ×" for Gemma-9B, and "1.5 ×" for Gemma-27B. This suggests the performance gap between PAV and ORM narrows with larger models. 4. **ORM Performance Plateau/Decline:** For the larger models (Gemma-9B and Gemma-27B), the ORM method's accuracy plateaus and then declines after a certain N value (around 2⁴-2⁵), while PAV and Pass@N continue to improve or maintain performance. 5. **Annotation Placement:** The "multiplier" annotations (5×, 2×, 1.5×) are placed at lower N values (2³-2⁴), while the "percentage" annotations (10%, 10%, 8%) are placed at the highest N value (2⁶-2⁷), highlighting different aspects of the comparison. ### Interpretation The data demonstrates the comparative effectiveness of the proposed `PAV (ours)` method against the `ORM` baseline across different scales of the Gemma model. The key finding is that **PAV provides a consistent accuracy improvement over ORM, but the magnitude of this relative advantage diminishes as the base model becomes larger and more capable.** The `Pass @N` method serves as a strong upper-bound benchmark, consistently outperforming both. The fact that PAV's curve is always between ORM and Pass@N suggests it successfully bridges part of the performance gap. The plateau and decline of ORM at higher N for larger models indicates a potential limitation or instability in that method under those conditions, which PAV appears to mitigate, as its curve remains more stable and continues to rise. The annotations tell a story of **scaling efficiency**: for the smallest model (2B), PAV offers a dramatic 5x improvement at a certain point, but for the largest model (27B), the improvement is a more modest 1.5x. This implies that advanced methods like PAV may be most crucial for boosting the performance of smaller, more constrained models, while larger models can achieve high performance through scale alone, though PAV still provides a meaningful absolute gain (8-10% at high N). The charts collectively argue for the value of the PAV method, especially in resource-constrained scenarios involving smaller models.

## Line Graphs: Accuracy Comparison Across Model Sizes (Gemma-2B, Gemma-9B, Gemma-27B) ### Overview The image contains three line graphs comparing the accuracy of three methods (ORM, PAV, Pass @N) across increasing values of N (2¹ to 2⁷) for three model sizes: Gemma-2B (a), Gemma-9B (b), and Gemma-27B (c). Accuracy is measured on the y-axis (0.1–0.6), while N is logarithmic on the x-axis. Annotations highlight relative performance improvements between methods. --- ### Components/Axes - **X-axis**: Labeled "N" with values 2¹ to 2⁷ (logarithmic scale). - **Y-axis**: Labeled "Accuracy" with values 0.1 to 0.6. - **Legend**: - **ORM**: Dashed blue line. - **PAV (ours)**: Solid red line. - **Pass @N**: Dashed gray line. - **Annotations**: Arrows with text indicating relative improvements (e.g., "5x", "10%", "1.5x", "8%"). --- ### Detailed Analysis #### Graph (a): Gemma-2B - **ORM**: Starts at ~0.12 (N=2¹), rises to ~0.2 (N=2⁷). Trend: Gradual upward slope. - **PAV**: Starts at ~0.15 (N=2¹), rises to ~0.3 (N=2⁷). Trend: Steeper upward slope than ORM. - **Pass @N**: Starts at ~0.1 (N=2¹), rises to ~0.4 (N=2⁷). Trend: Steepest upward slope. - **Annotations**: - Between N=2³ and 2⁴: "5x" (PAV vs. ORM) and "10%" (PAV vs. Pass @N). #### Graph (b): Gemma-9B - **ORM**: Starts at ~0.35 (N=2¹), rises to ~0.45 (N=2⁷). Trend: Moderate upward slope. - **PAV**: Starts at ~0.38 (N=2¹), rises to ~0.55 (N=2⁷). Trend: Steeper than ORM. - **Pass @N**: Starts at ~0.3 (N=2¹), rises to ~0.6 (N=2⁷). Trend: Steepest upward slope. - **Annotations**: - Between N=2³ and 2⁴: "2x" (PAV vs. ORM) and "10%" (PAV vs. Pass @N). #### Graph (c): Gemma-27B - **ORM**: Starts at ~0.4 (N=2¹), rises to ~0.5 (N=2⁷). Trend: Gradual upward slope. - **PAV**: Starts at ~0.42 (N=2¹), rises to ~0.58 (N=2⁷). Trend: Steeper than ORM. - **Pass @N**: Starts at ~0.4 (N=2¹), rises to ~0.65 (N=2⁷). Trend: Steepest upward slope. - **Annotations**: - Between N=2³ and 2⁴: "1.5x" (PAV vs. ORM) and "8%" (PAV vs. Pass @N). --- ### Key Observations 1. **PAV Consistently Outperforms**: Across all model sizes, PAV achieves higher accuracy than ORM and Pass @N, with the gap widening as N increases. 2. **Model Size Impact**: Larger models (Gemma-27B) show higher baseline accuracy and more pronounced performance improvements for PAV. 3. **Pass @N Plateaus**: Pass @N accuracy increases sharply with N but plateaus at higher N values, while PAV and ORM continue improving. 4. **Relative Gains**: Annotations indicate PAV’s accuracy improvements over ORM (e.g., "5x" in Gemma-2B) and Pass @N (e.g., "10%" in Gemma-2B). --- ### Interpretation - **PAV’s Advantage**: The red line (PAV) demonstrates superior scalability, particularly in larger models (Gemma-27B), suggesting it is more efficient at leveraging model capacity. - **Pass @N Limitations**: While Pass @N starts strong, its plateau implies diminishing returns at higher N, making it less suitable for large-scale applications. - **ORM’s Steady Growth**: ORM shows consistent but slower improvement, indicating it may require larger N to match PAV’s performance. - **Logarithmic N Scale**: The x-axis’s logarithmic scale emphasizes performance gains at lower N values, where PAV’s improvements are most dramatic. The data suggests PAV is the optimal method for accuracy across model sizes, with larger models amplifying its advantages. The annotations highlight critical inflection points where PAV’s performance diverges significantly from baselines.