## Line Chart: Accuracy vs. Number of Generated Solutions ### Overview The image is a line chart comparing the accuracy of two methods, "Maj@8" and "Last@8", against the number of generated solutions. It also includes horizontal lines representing the performance of "Llama-3.1-8B" and "Llama-3.2-3B". The x-axis represents the number of generated solutions, while the y-axis represents the accuracy in percentage. ### Components/Axes * **X-axis:** "Number of Generated Solutions" with values 1, 2, 4, 8, 16, 32, and 64. * **Y-axis:** "Accuracy (%)" with values ranging from 40% to 80% in increments of 10%. * **Legend:** Located at the top-right of the chart. * "Maj@8" (teal line with circle markers) * "Last@8" (coral line with square markers) * "Llama-3.1-8B" (black dashed line) * "Llama-3.2-3B" (gray dashed line) * **Horizontal Lines:** * Black dashed line at approximately 56% representing "Llama-3.1-8B". * Gray dashed line at approximately 49% representing "Llama-3.2-3B". ### Detailed Analysis * **Maj@8 (Teal Line):** * Trend: Generally increasing with the number of generated solutions. * Data Points: * 1 Solution: ~45% * 2 Solutions: ~50% * 4 Solutions: ~55% * 8 Solutions: ~62% * 16 Solutions: ~73% * 32 Solutions: ~72% * 64 Solutions: ~73% * **Last@8 (Coral Line):** * Trend: Increasing initially, then plateaus. * Data Points: * 1 Solution: ~47% * 2 Solutions: ~43% * 4 Solutions: ~51% * 8 Solutions: ~54% * 16 Solutions: ~72% * 32 Solutions: ~72% * 64 Solutions: ~74% ### Key Observations * Both "Maj@8" and "Last@8" show significant improvement in accuracy as the number of generated solutions increases from 1 to 16. * Beyond 16 solutions, the accuracy for both methods plateaus. * "Maj@8" generally outperforms "Last@8" except at 64 solutions where "Last@8" has a slightly higher accuracy. * "Llama-3.1-8B" and "Llama-3.2-3B" serve as baseline performance levels. ### Interpretation The chart suggests that generating more solutions initially improves the accuracy of both "Maj@8" and "Last@8" methods. However, there's a point of diminishing returns around 16 generated solutions. The performance of "Llama-3.1-8B" and "Llama-3.2-3B" provides a benchmark, indicating the relative effectiveness of the two methods being tested. The fact that both "Maj@8" and "Last@8" surpass the Llama baselines at higher numbers of generated solutions suggests that these methods are beneficial for improving accuracy in this context.

## Line Chart: Accuracy vs. Number of Generated Solutions ### Overview This image displays a line chart illustrating the accuracy of two different methods, "Maj@8" and "Last@8", as a function of the "Number of Generated Solutions". The chart also includes two horizontal dashed lines representing baseline accuracies for "Llama-3.1-8B" and "Llama-3.2-3B". The primary trend shows that accuracy generally increases with the number of generated solutions for both "Maj@8" and "Last@8". ### Components/Axes * **Chart Type**: Line Chart * **Title**: Implicitly, "Accuracy of Methods with Varying Number of Generated Solutions" * **Y-axis Label**: "Accuracy (%)" * **Scale**: Linear, ranging from approximately 40% to 80%. * **Markers**: 40%, 50%, 60%, 70%, 80%. * **X-axis Label**: "Number of Generated Solutions" * **Scale**: Logarithmic (implied by the spacing of markers), with markers at 1, 2, 4, 8, 16, 32, 64. * **Legend**: Located in the top-center of the chart. * **"Maj@8"**: Represented by a teal line with circular markers. * **"Last@8"**: Represented by an orange line with square markers. * **"Llama-3.1-8B"**: Represented by a dashed black line. * **"Llama-3.2-3B"**: Represented by a dashed grey line. ### Detailed Analysis **Data Series: "Maj@8" (Teal Line with Circles)** * **Trend**: The "Maj@8" line shows a consistent upward trend, indicating that accuracy improves as more solutions are generated. The slope appears to steepen between 4 and 16 generated solutions, then flattens out. * **Data Points (approximate values with uncertainty +/- 1%)**: * At 1 solution: ~45% * At 2 solutions: ~50% * At 4 solutions: ~57% * At 8 solutions: ~61% * At 16 solutions: ~72% * At 32 solutions: ~72% * At 64 solutions: ~74% **Data Series: "Last@8" (Orange Line with Squares)** * **Trend**: The "Last@8" line also shows an upward trend, but generally at a lower accuracy than "Maj@8" for most of the plotted range. The trend is less steep than "Maj@8" initially, but catches up and surpasses "Maj@8" between 8 and 16 solutions before falling slightly behind again. * **Data Points (approximate values with uncertainty +/- 1%)**: * At 1 solution: ~43% * At 2 solutions: ~42% * At 4 solutions: ~50% * At 8 solutions: ~51% * At 16 solutions: ~71% * At 32 solutions: ~71% * At 64 solutions: ~73% **Baseline Data Series:** * **"Llama-3.1-8B" (Dashed Black Line)**: This line is positioned at approximately 57% accuracy. * **"Llama-3.2-3B" (Dashed Grey Line)**: This line is positioned at approximately 49% accuracy. ### Key Observations * Both "Maj@8" and "Last@8" methods demonstrate a positive correlation between the number of generated solutions and accuracy. * The "Maj@8" method generally achieves higher accuracy than the "Last@8" method, especially at lower numbers of generated solutions (1-4). * The "Last@8" method shows a significant jump in accuracy between 4 and 16 generated solutions, briefly matching the performance of "Maj@8" at 16 and 32 solutions. * Both methods surpass the "Llama-3.2-3B" baseline at 4 generated solutions and the "Llama-3.1-8B" baseline at 16 generated solutions. * The accuracy gains for both methods appear to plateau or diminish at higher numbers of generated solutions (beyond 16). ### Interpretation The data suggests that for the tasks represented by "Maj@8" and "Last@8", increasing the number of generated solutions is a viable strategy to improve accuracy. The "Maj@8" method appears to be more robust and consistently performs better, particularly when fewer solutions are available. The "Last@8" method, while initially lagging, shows a strong improvement curve, indicating it might be more sensitive to a larger pool of generated solutions. The horizontal lines for "Llama-3.1-8B" and "Llama-3.2-3B" serve as benchmarks. The fact that both "Maj@8" and "Last@8" eventually exceed these benchmarks implies that these methods are effective for the problem domain. The plateauing of accuracy at higher numbers of generated solutions suggests diminishing returns, meaning that generating significantly more solutions beyond a certain point might not yield substantial improvements in accuracy and could incur higher computational costs. This could indicate that the models are converging or that the diversity of generated solutions is no longer significantly contributing to finding more correct answers.

## Line Chart: Accuracy vs. Number of Generated Solutions ### Overview This line chart compares the accuracy of two language models, Llama-3.1-8B and Llama-3.2-3B, across two metrics (Maj@8 and Last@8) as the number of generated solutions increases. Accuracy is measured in percentage, and the number of generated solutions ranges from 1 to 64. ### Components/Axes * **X-axis:** Number of Generated Solutions. Scale: 1, 2, 4, 8, 16, 32, 64. * **Y-axis:** Accuracy (%). Scale: 40% to 80%. * **Legend:** Located at the top-right of the chart. * Maj@8 (Solid Green Line with Circle Markers) * Last@8 (Solid Orange Line with Square Markers) * Llama-3.1-8B (Dashed Black Line) * Llama-3.2-3B (Dashed Gray Line) * **Horizontal Lines:** Two dashed horizontal lines are present at 50% and 60% accuracy. ### Detailed Analysis **Maj@8 (Green Line):** The green line representing Maj@8 starts at approximately 43% accuracy at 1 generated solution. It shows a generally upward trend, with a steeper increase between 4 and 8 generated solutions, reaching approximately 68% at 8 solutions. The line continues to rise, reaching around 72% at 16 solutions, and plateaus around 72-73% for 32 and 64 solutions. * 1 Solution: ~43% * 2 Solutions: ~47% * 4 Solutions: ~53% * 8 Solutions: ~68% * 16 Solutions: ~72% * 32 Solutions: ~72% * 64 Solutions: ~73% **Last@8 (Orange Line):** The orange line representing Last@8 begins at approximately 45% accuracy at 1 generated solution. It fluctuates slightly, reaching around 47% at 2 solutions, then dips to approximately 44% at 4 solutions. It then increases to around 50% at 8 solutions, and continues to rise, reaching approximately 68% at 32 and 64 solutions. * 1 Solution: ~45% * 2 Solutions: ~47% * 4 Solutions: ~44% * 8 Solutions: ~50% * 16 Solutions: ~58% * 32 Solutions: ~68% * 64 Solutions: ~68% **Llama-3.1-8B (Black Dashed Line):** The black dashed line representing Llama-3.1-8B starts at approximately 58% accuracy at 1 generated solution. It remains relatively stable, fluctuating around 58-60% across all values of generated solutions (1 to 64). * 1 Solution: ~58% * 2 Solutions: ~59% * 4 Solutions: ~59% * 8 Solutions: ~60% * 16 Solutions: ~60% * 32 Solutions: ~60% * 64 Solutions: ~60% **Llama-3.2-3B (Gray Dashed Line):** The gray dashed line representing Llama-3.2-3B starts at approximately 48% accuracy at 1 generated solution. It shows a slight upward trend, reaching around 52% at 2 solutions, and then plateaus around 52-54% for all other values of generated solutions (4 to 64). * 1 Solution: ~48% * 2 Solutions: ~52% * 4 Solutions: ~53% * 8 Solutions: ~53% * 16 Solutions: ~54% * 32 Solutions: ~54% * 64 Solutions: ~54% ### Key Observations * Maj@8 shows the most significant improvement in accuracy as the number of generated solutions increases, particularly between 4 and 16 solutions. * Last@8 shows a more gradual increase in accuracy. * Llama-3.1-8B consistently outperforms Llama-3.2-3B across all numbers of generated solutions. * Both Llama models show relatively stable accuracy beyond 16 generated solutions. * The 60% accuracy threshold is surpassed by Llama-3.1-8B and approached by Last@8 at higher solution counts. ### Interpretation The data suggests that increasing the number of generated solutions improves the accuracy of both metrics (Maj@8 and Last@8), but the effect is more pronounced for Maj@8. This indicates that generating more solutions allows the model to explore a wider range of possibilities and potentially identify better solutions. The consistent outperformance of Llama-3.1-8B suggests that it is a more capable model than Llama-3.2-3B, at least for this specific task and evaluation metrics. The plateauing of accuracy at higher solution counts suggests that there is a diminishing return to generating more solutions beyond a certain point. The horizontal lines at 50% and 60% serve as benchmarks, highlighting the performance improvements achieved by the models as the number of generated solutions increases. The difference between Maj@8 and Last@8 could indicate that the model is better at identifying the *best* solution among many (Maj@8) than at consistently placing the best solution *last* in the generated list (Last@8).

\n ## Line Chart: Accuracy vs. Number of Generated Solutions for Llama Models ### Overview This image is a line chart comparing the performance of two evaluation methods ("Maj@8" and "Last@8") across two different Llama model sizes (3.1-8B and 3.2-3B) as the number of generated solutions increases. The chart demonstrates how accuracy scales with increased solution generation. ### Components/Axes * **X-Axis:** Labeled "Number of Generated Solutions". It uses a logarithmic scale with discrete markers at 1, 2, 4, 8, 16, 32, and 64. * **Y-Axis:** Labeled "Accuracy (%)". It is a linear scale ranging from 40% to 80%, with major gridlines at 10% intervals (40%, 50%, 60%, 70%, 80%). * **Legend (Top Center):** * **Left Column (Methods):** * A teal circle icon labeled "Maj@8". * An orange square icon labeled "Last@8". * **Right Column (Model Baselines):** * A black dashed line labeled "Llama-3.1-8B". * A gray dashed line labeled "Llama-3.2-3B". * **Data Series:** 1. **Maj@8 (Teal line with circle markers):** Represents the accuracy of the "Majority@8" method. 2. **Last@8 (Orange line with square markers):** Represents the accuracy of the "Last@8" method. * **Baseline Reference Lines:** * **Llama-3.1-8B (Black dashed line):** A horizontal line at approximately 56% accuracy. * **Llama-3.2-3B (Gray dashed line):** A horizontal line at approximately 49% accuracy. ### Detailed Analysis **Data Points (Approximate Values):** | Number of Solutions | Maj@8 Accuracy (%) | Last@8 Accuracy (%) | | :----------------- | :----------------- | :------------------ | | 1 | ~45% | ~45% | | 2 | ~49% | ~44% | | 4 | ~56% | ~51% | | 8 | ~61% | ~53% | | 16 | ~74% | ~72% | | 32 | ~72% | ~72% | | 64 | ~73% | ~75% | **Trend Verification:** * **Maj@8 (Teal Line):** Shows a consistent upward trend from 1 to 16 solutions, with a steep increase between 8 and 16. It plateaus between 16 and 64 solutions, hovering in the low-to-mid 70% range. * **Last@8 (Orange Line):** Shows a more gradual upward trend from 1 to 8 solutions. It experiences a very sharp increase between 8 and 16 solutions, converging with the Maj@8 line. From 16 to 64 solutions, it remains stable in the low-to-mid 70% range, ending slightly higher than Maj@8 at 64 solutions. ### Key Observations 1. **Performance Scaling:** Both methods show significant accuracy gains as the number of generated solutions increases from 1 to 16. 2. **Critical Threshold:** The most dramatic improvement for both methods occurs when increasing from 8 to 16 generated solutions. 3. **Method Comparison:** "Maj@8" outperforms "Last@8" at lower solution counts (2, 4, 8). Their performance converges at 16 solutions and remains very similar thereafter. 4. **Surpassing Baselines:** Both methods surpass the Llama-3.1-8B baseline (~56%) at 8 solutions (Maj@8) or 16 solutions (Last@8). They significantly exceed the Llama-3.2-3B baseline (~49%) from 4 solutions onward. 5. **Plateau Effect:** After 16 solutions, generating more solutions (32, 64) yields minimal additional accuracy improvement for either method, suggesting diminishing returns. ### Interpretation The data suggests that generating multiple solutions and applying a selection strategy (either majority vote or taking the last one) is highly effective for improving the accuracy of these language models on the evaluated task. The "Maj@8" method appears more sample-efficient, achieving higher accuracy with fewer generated solutions. However, with a sufficient number of solutions (16 or more), the simpler "Last@8" method performs comparably. The sharp performance jump at 16 solutions indicates a potential phase change or threshold where the model's output distribution becomes rich enough for the selection mechanisms to work very effectively. The plateau after 16 solutions implies that the core capability of the model is being fully leveraged by these methods, and further gains would likely require improvements to the base model itself rather than just more sampling. The chart effectively argues for the utility of inference-time computation (generating more solutions) as a powerful tool to boost performance, with the choice between "Maj@8" and "Last@8" potentially depending on the computational budget (fewer solutions favor Maj@8) or implementation simplicity (Last@8).

## Line Chart: Accuracy vs. Number of Generated Solutions ### Overview The chart compares the accuracy of two metrics, **Maj@8** (teal circles) and **Last@8** (orange squares), across different numbers of generated solutions (1, 2, 4, 8, 16, 32, 64). Two baseline models, **Llama-3.1-8B** (dashed black line) and **Llama-3.2-3B** (dashed gray line), are shown for reference. Accuracy is measured on the y-axis (40%–80%), while the x-axis represents the number of generated solutions. --- ### Components/Axes - **X-axis**: "Number of Generated Solutions" (logarithmic scale: 1, 2, 4, 8, 16, 32, 64). - **Y-axis**: "Accuracy (%)" (linear scale: 40%–80%). - **Legend**: - **Maj@8**: Teal circles (solid line). - **Last@8**: Orange squares (solid line). - **Llama-3.1-8B**: Dashed black line. - **Llama-3.2-3B**: Dashed gray line. --- ### Detailed Analysis 1. **Maj@8 (Teal Circles)**: - Starts at ~45% accuracy for 1 solution. - Increases steadily to ~75% at 16 solutions. - Plateaus at ~73% for 32 and 64 solutions. 2. **Last@8 (Orange Squares)**: - Begins at ~43% for 1 solution. - Rises to ~75% at 16 solutions. - Slightly increases to ~76% at 64 solutions. 3. **Baselines**: - **Llama-3.1-8B** (dashed black): Horizontal at ~50%. - **Llama-3.2-3B** (dashed gray): Horizontal at ~45%. --- ### Key Observations - Both **Maj@8** and **Last@8** show significant accuracy improvements as the number of generated solutions increases, with the steepest gains between 1 and 16 solutions. - **Maj@8** consistently outperforms **Last@8** at lower solution counts (e.g., 1–8 solutions), but the gap narrows at higher counts (e.g., 16–64 solutions). - The baselines (**Llama-3.1-8B** and **Llama-3.2-3B**) are below the data points, indicating that the evaluated models outperform these baselines. - Accuracy plateaus for both metrics after 16 solutions, suggesting diminishing returns beyond this point. --- ### Interpretation The chart demonstrates that generating more solutions improves accuracy up to a critical threshold (16 solutions), after which further increases yield minimal gains. This suggests that the models are optimized for a specific number of solutions, and excessive generation may not enhance performance. The convergence of **Maj@8** and **Last@8** at higher solution counts implies that both metrics align in their evaluation of solution quality at scale. The baselines highlight that the models under evaluation are more effective than the referenced Llama versions, particularly for larger solution sets.