## Line Charts: Model Performance on Math Problems ### Overview The image contains eight line charts comparing the performance of different language models on various math problem datasets. The charts are arranged in a 2x4 grid, with the top row showing results for models using "Qwen" and the bottom row using "Gemma." Each chart plots the accuracy of the models against the number of solutions used. The models compared include Pass@K, GenPRM-7B with different configurations (Pass@1, Maj@4, Maj@8, Direct), Qwen2.5-Math-7B, Qwen2.5-Math-7B-PRM800K, Skywork-7B, and Skywork-1.5B. ### Components/Axes * **X-axis (Horizontal):** Number of Solutions (N). The x-axis is logarithmic, with values ranging from 2⁰ to 2⁵ in most charts, and up to 2⁸ in the AMC23 charts. * **Y-axis (Vertical):** Accuracy (%). The y-axis scale varies depending on the chart, ranging from approximately 85% to 95% for MATH, 65% to 95% for AMC23, 5% to 35% for AIME24, and 32.5% to 52.5% for Minerva Math. * **Chart Titles:** Each chart has a title indicating the dataset and the model family used: * (a) MATH (Qwen) * (b) AMC23 (Qwen) * (c) AIME24 (Qwen) * (d) Minerva Math (Qwen) * (e) MATH (Gemma) * (f) AMC23 (Gemma) * (g) AIME24 (Gemma) * (h) Minerva Math (Gemma) * **Legend (Top):** The legend is located at the top of the image and identifies each model by color and name: * **Gray dotted line:** Pass@K * **Blue line:** GenPRM-7B (Pass@1) * **Gray line:** Maj. * **Orange line:** GenPRM-7B (Maj@4) * **Green line:** GenPRM-7B (Maj@8) * **Red line:** Direct GenPRM-7B * **Purple line:** Qwen2.5-Math-7B * **Brown line:** Qwen2.5-Math-7B-PRM800K * **Pink line:** Skywork-7B * **Yellow-Green line:** Skywork-1.5B ### Detailed Analysis #### (a) MATH (Qwen) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 82% at 2⁰ to 94% at 2⁵. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 84% at 2⁰ to 89% at 2⁵. * **Maj. (Gray line):** Accuracy remains relatively flat around 86%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy increases slightly from approximately 85% to 88%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy increases slightly from approximately 85% to 90%. * **Direct GenPRM-7B (Red line):** Accuracy remains relatively flat around 85%. * **Qwen2.5-Math-7B (Purple line):** Accuracy remains relatively flat around 87%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy remains relatively flat around 86%. * **Skywork-7B (Pink line):** Accuracy remains relatively flat around 87%. * **Skywork-1.5B (Yellow-Green line):** Accuracy remains relatively flat around 86%. #### (b) AMC23 (Qwen) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 65% at 2⁰ to 95% at 2⁸. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 70% at 2⁰ to 85% at 2⁸. * **Maj. (Gray line):** Accuracy fluctuates between 75% and 80%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy fluctuates between 75% and 85%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy fluctuates between 75% and 85%. * **Direct GenPRM-7B (Red line):** Accuracy fluctuates between 70% and 75%. * **Qwen2.5-Math-7B (Purple line):** Accuracy fluctuates between 75% and 80%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy fluctuates between 70% and 75%. * **Skywork-7B (Pink line):** Accuracy fluctuates between 75% and 80%. * **Skywork-1.5B (Yellow-Green line):** Accuracy fluctuates between 70% and 75%. #### (c) AIME24 (Qwen) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 5% at 2⁰ to 35% at 2⁸. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 5% at 2⁰ to 20% at 2⁸. * **Maj. (Gray line):** Accuracy fluctuates between 5% and 10%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy fluctuates between 10% and 25%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy fluctuates between 10% and 25%. * **Direct GenPRM-7B (Red line):** Accuracy fluctuates between 5% and 10%. * **Qwen2.5-Math-7B (Purple line):** Accuracy fluctuates between 10% and 20%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy fluctuates between 5% and 10%. * **Skywork-7B (Pink line):** Accuracy fluctuates between 10% and 20%. * **Skywork-1.5B (Yellow-Green line):** Accuracy fluctuates between 5% and 10%. #### (d) Minerva Math (Qwen) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 32.5% at 2⁰ to 52.5% at 2⁵. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 32.5% at 2⁰ to 40% at 2⁵. * **Maj. (Gray line):** Accuracy remains relatively flat around 37.5%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy remains relatively flat around 42.5%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy remains relatively flat around 40%. * **Direct GenPRM-7B (Red line):** Accuracy remains relatively flat around 37.5%. * **Qwen2.5-Math-7B (Purple line):** Accuracy remains relatively flat around 37.5%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy remains relatively flat around 37.5%. * **Skywork-7B (Pink line):** Accuracy remains relatively flat around 37.5%. * **Skywork-1.5B (Yellow-Green line):** Accuracy remains relatively flat around 37.5%. #### (e) MATH (Gemma) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 82% at 2⁰ to 94% at 2⁵. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 82% at 2⁰ to 90% at 2⁵. * **Maj. (Gray line):** Accuracy remains relatively flat around 84%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy increases slightly from approximately 83% to 86%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy increases slightly from approximately 83% to 88%. * **Direct GenPRM-7B (Red line):** Accuracy remains relatively flat around 84%. * **Qwen2.5-Math-7B (Purple line):** Accuracy remains relatively flat around 85%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy remains relatively flat around 84%. * **Skywork-7B (Pink line):** Accuracy remains relatively flat around 85%. * **Skywork-1.5B (Yellow-Green line):** Accuracy remains relatively flat around 84%. #### (f) AMC23 (Gemma) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 65% at 2⁰ to 95% at 2⁸. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 65% at 2⁰ to 90% at 2⁸. * **Maj. (Gray line):** Accuracy fluctuates between 70% and 75%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy fluctuates between 75% and 85%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy fluctuates between 75% and 85%. * **Direct GenPRM-7B (Red line):** Accuracy fluctuates between 70% and 75%. * **Qwen2.5-Math-7B (Purple line):** Accuracy fluctuates between 75% and 80%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy fluctuates between 70% and 75%. * **Skywork-7B (Pink line):** Accuracy fluctuates between 75% and 80%. * **Skywork-1.5B (Yellow-Green line):** Accuracy fluctuates between 70% and 75%. #### (g) AIME24 (Gemma) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 15% at 2⁰ to 40% at 2⁸. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 15% at 2⁰ to 30% at 2⁸. * **Maj. (Gray line):** Accuracy fluctuates between 15% and 20%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy fluctuates between 20% and 25%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy fluctuates between 20% and 25%. * **Direct GenPRM-7B (Red line):** Accuracy fluctuates between 15% and 20%. * **Qwen2.5-Math-7B (Purple line):** Accuracy fluctuates between 20% and 25%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy fluctuates between 15% and 20%. * **Skywork-7B (Pink line):** Accuracy fluctuates between 20% and 25%. * **Skywork-1.5B (Yellow-Green line):** Accuracy fluctuates between 15% and 20%. #### (h) Minerva Math (Gemma) * **Pass@K (Gray dotted line):** Accuracy increases sharply from approximately 30% at 2⁰ to 52% at 2⁵. * **GenPRM-7B (Pass@1) (Blue line):** Accuracy increases from approximately 30% at 2⁰ to 40% at 2⁵. * **Maj. (Gray line):** Accuracy remains relatively flat around 35%. * **GenPRM-7B (Maj@4) (Orange line):** Accuracy remains relatively flat around 37%. * **GenPRM-7B (Maj@8) (Green line):** Accuracy remains relatively flat around 37%. * **Direct GenPRM-7B (Red line):** Accuracy remains relatively flat around 35%. * **Qwen2.5-Math-7B (Purple line):** Accuracy remains relatively flat around 35%. * **Qwen2.5-Math-7B-PRM800K (Brown line):** Accuracy remains relatively flat around 35%. * **Skywork-7B (Pink line):** Accuracy remains relatively flat around 35%. * **Skywork-1.5B (Yellow-Green line):** Accuracy remains relatively flat around 35%. ### Key Observations * **Pass@K consistently outperforms other models** when the number of solutions increases, showing a steep upward trend in all charts. * The performance of other models (GenPRM-7B variants, Qwen2.5-Math-7B, Skywork models) tends to plateau or fluctuate, with less significant improvements as the number of solutions increases. * The AMC23 and AIME24 datasets show a wider range of accuracy values and more fluctuation compared to the MATH and Minerva Math datasets. * The "Qwen" models generally show slightly better performance than the "Gemma" models, especially in the MATH and Minerva Math datasets. ### Interpretation The charts demonstrate the impact of increasing the number of solutions (N) on the accuracy of different language models when solving math problems. The Pass@K model benefits significantly from a higher number of solutions, indicating its ability to leverage multiple attempts to find the correct answer. Other models show limited improvement with increasing N, suggesting they may be less effective at utilizing multiple solutions or have reached a performance ceiling. The differences in performance across datasets (MATH, AMC23, AIME24, Minerva Math) highlight the varying difficulty and characteristics of these problem sets. The AMC23 and AIME24 datasets, with their wider accuracy ranges and fluctuations, may present more complex challenges for the models. The comparison between "Qwen" and "Gemma" models suggests that the underlying architecture or training data of the "Qwen" models may provide a slight advantage in solving these math problems, particularly in the MATH and Minerva Math datasets.

## Chart Type: Grid of Line Charts showing Model Accuracy vs. Number of Solutions ### Overview This image displays an 8-panel grid of line charts, each illustrating the "Accuracy (%)" on the Y-axis against the "Number of Solutions (N)" on the X-axis. The charts compare the performance of various language models and solution aggregation strategies across four different mathematical reasoning benchmarks: MATH, AMC23, AIME24, and Minerva Math. The top row of four charts evaluates models based on the "Qwen" family, while the bottom row evaluates models based on the "Gemma" family. A common legend at the top defines ten distinct data series, each representing a different model or strategy. ### Components/Axes **Common Legend (positioned at the top, horizontally across the entire image):** * **Pass@K**: Gray dotted line with star markers. * **GenPRM-7B (Pass@1)**: Dark blue solid line with circle markers. * **GenPRM-7B (Maj@8)**: Green solid line with triangle markers. * **Qwen2.5-Math-7B**: Purple solid line with square markers. * **Skywork-7B**: Light pink/magenta solid line with diamond markers. * **Maj.** (Majority Voting): Light gray solid line with square markers. * **GenPRM-7B (Maj@4)**: Orange solid line with square markers. * **Direct GenPRM-7B**: Red solid line with circle markers. * **Qwen2.5-Math-7B-PRM800K**: Dark brown solid line with circle markers. * **Skywork-1.5B**: Yellow-green solid line with square markers. **Common Axes:** * **X-axis Title (bottom of each chart):** "Number of Solutions (N)" * X-axis labels are powers of 2: 2^0 (1), 2^1 (2), 2^2 (4), 2^3 (8), 2^4 (16), 2^5 (32), 2^6 (64), 2^7 (128), 2^8 (256). The specific range varies per chart. * **Y-axis Title (left of each chart):** "Accuracy (%)" **Sub-Chart Titles and Specific Axis Ranges:** **Top Row (Qwen Models):** * **(a) MATH (Qwen)** * X-axis range: 2^0 to 2^5. * Y-axis range: 82% to 94%, with major ticks at 82, 84, 86, 88, 90, 92, 94. * **(b) AMC23 (Qwen)** * X-axis range: 2^0 to 2^8. * Y-axis range: 70% to 95%, with major ticks at 70, 75, 80, 85, 90, 95. * **(c) AIME24 (Qwen)** * X-axis range: 2^0 to 2^8. * Y-axis range: 0% to 35%, with major ticks at 5, 10, 15, 20, 25, 30, 35. * **(d) Minerva Math (Qwen)** * X-axis range: 2^0 to 2^5. * Y-axis range: 32.5% to 52.5%, with major ticks at 32.5, 35.0, 37.5, 40.0, 42.5, 45.0, 47.5, 50.0, 52.5. **Bottom Row (Gemma Models):** * **(e) MATH (Gemma)** * X-axis range: 2^0 to 2^5. * Y-axis range: 82% to 94%, with major ticks at 82, 84, 86, 88, 90, 92, 94. * **(f) AMC23 (Gemma)** * X-axis range: 2^0 to 2^8. * Y-axis range: 60% to 95%, with major ticks at 65, 70, 75, 80, 85, 90, 95. * **(g) AIME24 (Gemma)** * X-axis range: 2^0 to 2^8. * Y-axis range: 15% to 40%, with major ticks at 15, 20, 25, 30, 35, 40. * **(h) Minerva Math (Gemma)** * X-axis range: 2^0 to 2^5. * Y-axis range: 30% to 55%, with major ticks at 30, 35, 40, 45, 50, 55. ### Detailed Analysis **Chart (a) MATH (Qwen)** * **Pass@K (Gray dotted star)**: Steadily increases from ~82.5% at 2^0 to ~95% at 2^5. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~82% at 2^0 to ~89% at 2^5. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~82% at 2^0 to ~89.5% at 2^5. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~82% at 2^0 to ~87% at 2^5. * **Skywork-7B (Light pink solid diamond)**: Increases from ~82% at 2^0 to ~87.5% at 2^5. * **Maj. (Light gray solid square)**: Increases from ~82% at 2^0 to ~86.5% at 2^5. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~82% at 2^0 to ~88.5% at 2^5. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~82% at 2^0 to ~86% at 2^5. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~82% at 2^0 to ~87% at 2^5. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~82% at 2^0 to ~86% at 2^5. **Chart (b) AMC23 (Qwen)** * **Pass@K (Gray dotted star)**: Steadily increases from ~70% at 2^0 to ~96% at 2^8. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~70% at 2^0, peaks at ~80% at 2^3, then fluctuates, ending at ~79% at 2^8. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~70% at 2^0, peaks at ~83% at 2^3, then fluctuates, ending at ~82% at 2^8. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~70% at 2^0, peaks at ~79% at 2^3, then fluctuates, ending at ~77% at 2^8. * **Skywork-7B (Light pink solid diamond)**: Increases from ~70% at 2^0, peaks at ~77% at 2^3, then fluctuates, ending at ~74% at 2^8. * **Maj. (Light gray solid square)**: Increases from ~70% at 2^0, peaks at ~77% at 2^3, then fluctuates, ending at ~73% at 2^8. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~70% at 2^0, peaks at ~83% at 2^3, then fluctuates, ending at ~81% at 2^8. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~70% at 2^0, peaks at ~75% at 2^3, then fluctuates, ending at ~71% at 2^8. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~70% at 2^0, peaks at ~78% at 2^3, then fluctuates, ending at ~75% at 2^8. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~70% at 2^0, peaks at ~76% at 2^3, then fluctuates, ending at ~72% at 2^8. **Chart (c) AIME24 (Qwen)** * **Pass@K (Gray dotted star)**: Steadily increases from ~2.5% at 2^0 to ~36% at 2^8. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~2.5% at 2^0, peaks at ~12% at 2^3, then fluctuates, ending at ~15% at 2^8. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~2.5% at 2^0, peaks at ~13% at 2^3, then fluctuates, ending at ~17% at 2^8. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~2.5% at 2^0, peaks at ~11% at 2^3, then fluctuates, ending at ~14% at 2^8. * **Skywork-7B (Light pink solid diamond)**: Increases from ~2.5% at 2^0, peaks at ~10% at 2^3, then fluctuates, ending at ~12% at 2^8. * **Maj. (Light gray solid square)**: Increases from ~2.5% at 2^0, peaks at ~10% at 2^3, then fluctuates, ending at ~11% at 2^8. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~2.5% at 2^0, peaks at ~13% at 2^3, then fluctuates, ending at ~16% at 2^8. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~2.5% at 2^0, peaks at ~8% at 2^3, then fluctuates, ending at ~9% at 2^8. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~2.5% at 2^0, peaks at ~11% at 2^3, then fluctuates, ending at ~13% at 2^8. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~2.5% at 2^0, peaks at ~9% at 2^3, then fluctuates, ending at ~10% at 2^8. **Chart (d) Minerva Math (Qwen)** * **Pass@K (Gray dotted star)**: Steadily increases from ~33% at 2^0 to ~53% at 2^5. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~33% at 2^0 to ~38.5% at 2^5. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~33% at 2^0 to ~39.5% at 2^5. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~33% at 2^0 to ~37% at 2^5. * **Skywork-7B (Light pink solid diamond)**: Increases from ~33% at 2^0 to ~36.5% at 2^5. * **Maj. (Light gray solid square)**: Increases from ~33% at 2^0 to ~35.5% at 2^5. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~33% at 2^0 to ~38% at 2^5. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~33% at 2^0 to ~35% at 2^5. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~33% at 2^0 to ~36.5% at 2^5. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~33% at 2^0 to ~35.5% at 2^5. **Chart (e) MATH (Gemma)** * **Pass@K (Gray dotted star)**: Steadily increases from ~82.5% at 2^0 to ~95% at 2^5. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~82% at 2^0 to ~88% at 2^5. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~82% at 2^0 to ~89% at 2^5. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~82% at 2^0 to ~86% at 2^5. * **Skywork-7B (Light pink solid diamond)**: Increases from ~82% at 2^0 to ~86.5% at 2^5. * **Maj. (Light gray solid square)**: Increases from ~82% at 2^0 to ~85% at 2^5. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~82% at 2^0 to ~87.5% at 2^5. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~82% at 2^0 to ~85% at 2^5. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~82% at 2^0 to ~86% at 2^5. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~82% at 2^0 to ~85% at 2^5. **Chart (f) AMC23 (Gemma)** * **Pass@K (Gray dotted star)**: Steadily increases from ~64% at 2^0 to ~96% at 2^8. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~64% at 2^0, peaks at ~80% at 2^3, then fluctuates, ending at ~78% at 2^8. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~64% at 2^0, peaks at ~85% at 2^3, then fluctuates, ending at ~83% at 2^8. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~64% at 2^0, peaks at ~78% at 2^3, then fluctuates, ending at ~76% at 2^8. * **Skywork-7B (Light pink solid diamond)**: Increases from ~64% at 2^0, peaks at ~75% at 2^3, then fluctuates, ending at ~72% at 2^8. * **Maj. (Light gray solid square)**: Increases from ~64% at 2^0, peaks at ~75% at 2^3, then fluctuates, ending at ~71% at 2^8. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~64% at 2^0, peaks at ~84% at 2^3, then fluctuates, ending at ~82% at 2^8. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~64% at 2^0, peaks at ~70% at 2^3, then fluctuates, ending at ~68% at 2^8. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~64% at 2^0, peaks at ~76% at 2^3, then fluctuates, ending at ~73% at 2^8. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~64% at 2^0, peaks at ~72% at 2^3, then fluctuates, ending at ~69% at 2^8. **Chart (g) AIME24 (Gemma)** * **Pass@K (Gray dotted star)**: Steadily increases from ~16% at 2^0 to ~40% at 2^8. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~16% at 2^0, peaks at ~27% at 2^3, then fluctuates, ending at ~26% at 2^8. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~16% at 2^0, peaks at ~28% at 2^3, then fluctuates, ending at ~27% at 2^8. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~16% at 2^0, peaks at ~25% at 2^3, then fluctuates, ending at ~24% at 2^8. * **Skywork-7B (Light pink solid diamond)**: Increases from ~16% at 2^0, peaks at ~23% at 2^3, then fluctuates, ending at ~21% at 2^8. * **Maj. (Light gray solid square)**: Increases from ~16% at 2^0, peaks at ~22% at 2^3, then fluctuates, ending at ~20% at 2^8. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~16% at 2^0, peaks at ~27% at 2^3, then fluctuates, ending at ~26% at 2^8. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~16% at 2^0, peaks at ~20% at 2^3, then fluctuates, ending at ~18% at 2^8. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~16% at 2^0, peaks at ~24% at 2^3, then fluctuates, ending at ~23% at 2^8. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~16% at 2^0, peaks at ~21% at 2^3, then fluctuates, ending at ~19% at 2^8. **Chart (h) Minerva Math (Gemma)** * **Pass@K (Gray dotted star)**: Steadily increases from ~30% at 2^0 to ~53% at 2^5. * **GenPRM-7B (Pass@1) (Dark blue solid circle)**: Increases from ~30% at 2^0 to ~35% at 2^5. * **GenPRM-7B (Maj@8) (Green solid triangle)**: Increases from ~30% at 2^0 to ~36% at 2^5. * **Qwen2.5-Math-7B (Purple solid square)**: Increases from ~30% at 2^0 to ~34% at 2^5. * **Skywork-7B (Light pink solid diamond)**: Increases from ~30% at 2^0 to ~33.5% at 2^5. * **Maj. (Light gray solid square)**: Increases from ~30% at 2^0 to ~32.5% at 2^5. * **GenPRM-7B (Maj@4) (Orange solid square)**: Increases from ~30% at 2^0 to ~35.5% at 2^5. * **Direct GenPRM-7B (Red solid circle)**: Increases from ~30% at 2^0 to ~32% at 2^5. * **Qwen2.5-Math-7B-PRM800K (Dark brown solid circle)**: Increases from ~30% at 2^0 to ~33.5% at 2^5. * **Skywork-1.5B (Yellow-green solid square)**: Increases from ~30% at 2^0 to ~32.5% at 2^5. ### Key Observations 1. **Dominance of Pass@K**: Across all 8 benchmarks and both Qwen and Gemma models, "Pass@K" consistently achieves the highest accuracy and shows the most significant, near-linear improvement with an increasing "Number of Solutions (N)". It serves as a strong upper bound or ideal performance metric. 2. **General Trend of Increasing Accuracy**: For almost all methods and benchmarks, accuracy generally increases as the "Number of Solutions (N)" increases, indicating that providing more solutions for evaluation or aggregation improves performance. 3. **Performance Plateaus/Fluctuations**: While accuracy generally increases, many methods, especially on AMC23 and AIME24 benchmarks, show a plateau or even slight fluctuations/decreases in accuracy after an initial rapid rise (e.g., around 2^3 or 2^4). This suggests diminishing returns or potential instability with very high numbers of solutions for certain aggregation strategies. 4. **GenPRM Variants Outperform Baselines**: The "GenPRM-7B (Maj@8)" (green triangle) and "GenPRM-7B (Maj@4)" (orange square) generally perform better than other non-Pass@K methods, including "GenPRM-7B (Pass@1)" (dark blue circle), "Maj." (light gray square), "Direct GenPRM-7B" (red circle), and the Qwen/Skywork base models. "GenPRM-7B (Maj@8)" often slightly edges out "GenPRM-7B (Maj@4)". 5. **"Direct GenPRM-7B" is a Lower Performer**: The "Direct GenPRM-7B" (red circle) consistently ranks among the lowest-performing methods across all benchmarks and model families, often performing similarly to or worse than simple majority voting ("Maj."). 6. **Benchmark Difficulty**: * MATH shows the highest overall accuracies (80-95%). * AMC23 shows moderate accuracies (60-95%). * Minerva Math shows lower moderate accuracies (30-55%). * AIME24 is the most challenging benchmark, with accuracies ranging from very low single digits to around 40%. 7. **Model Family Comparison (Qwen vs. Gemma)**: * For MATH and Minerva Math, the overall accuracy ranges and relative performance of methods are quite similar between Qwen and Gemma models. * For AMC23, Gemma models start at a slightly lower baseline (~64% vs ~70% for Qwen) but show similar trends and peak performance. * For AIME24, Gemma models also start at a slightly higher baseline (~16% vs ~2.5% for Qwen) and achieve slightly higher peak accuracies for most methods compared to Qwen, suggesting Gemma might be marginally better on this specific hard task. 8. **Impact of PRM800K**: "Qwen2.5-Math-7B-PRM800K" (dark brown circle) generally performs slightly better than "Qwen2.5-Math-7B" (purple square) but not as well as the top GenPRM-7B (Maj@X) variants. ### Interpretation The data strongly suggests that increasing the number of solutions (N) generally improves the accuracy of mathematical reasoning models, but the extent and nature of this improvement depend heavily on the aggregation strategy and the difficulty of the benchmark. The "Pass@K" metric, which likely represents the accuracy if at least one correct solution is found among K attempts, sets an upper bound on performance. Its consistent, strong upward trend indicates that the underlying models are generating correct solutions, even if the aggregation methods struggle to identify them. The gap between "Pass@K" and other methods highlights the challenge of effectively leveraging multiple generated solutions. Among the practical aggregation strategies, the "GenPRM-7B (Maj@8)" and "GenPRM-7B (Maj@4)" methods consistently demonstrate superior performance. This implies that using a Program-of-Thought (PoT) based approach with majority voting over a moderate number of solutions (4 or 8) is more effective than simpler methods like "Pass@1" (likely taking the first correct solution) or plain "Maj." (majority voting without PRM). The "Direct GenPRM-7B" performs poorly, suggesting that the specific implementation or direct application of GenPRM without further aggregation or refinement is not effective. The performance differences across benchmarks underscore their varying difficulty levels. Tasks like AIME24, with low absolute accuracies, indicate that even with advanced models and aggregation, these problems remain highly challenging. The diminishing returns observed for some methods at higher N values, particularly on AMC23 and AIME24, could be due to several factors: 1. **Error Accumulation**: With more solutions, there might be more incorrect solutions that confuse the aggregation mechanism. 2. **Redundancy**: Beyond a certain point, additional solutions may not provide novel correct information, leading to a plateau.

\n ## Line Charts: Accuracy vs. Number of Solutions for Different Models ### Overview The image presents eight line charts, arranged in a 2x4 grid, displaying the accuracy of various language models (Owen and Gemma) on four different math datasets: MATH, AMC23, AIME24, and Minerva Math. Accuracy is plotted against the number of solutions (N), ranging from 2² to 2⁵ (4 to 32). Each chart focuses on a specific dataset and compares the performance of different model variants. ### Components/Axes * **X-axis:** Number of Solutions (N), labeled as "Number of Solutions (N)". Scale: 4, 8, 16, 32. * **Y-axis:** Accuracy (%), labeled as "Accuracy (%)". Scales vary per chart, but generally range from approximately 32% to 95%. * **Legend:** Located at the top of the image, spanning all charts. The legend identifies the following models/metrics: * Pass@K (Dark Blue) * Maj. (Light Blue) * GenPRM-7B (Pass@1) (Green) * GenPRM-7B (Maj@8) (Orange) * Qwen2.5-Math-7B (Red) * Qwen2.5-Math-7B-PRM800K (Purple) * Skywork-7B (Teal) * Skywork-1.5B (Yellow) * **Chart Titles:** Each chart is labeled with the dataset name and the model family (Owen or Gemma) in parentheses, e.g., "(a) MATH (Qwen)", "(f) AMC23 (Gemma)". ### Detailed Analysis or Content Details **Chart (a) MATH (Qwen):** * Pass@K: Starts at ~86%, increases to ~93% at N=32. * Maj.: Starts at ~85%, increases to ~91% at N=32. * GenPRM-7B (Pass@1): Starts at ~85%, increases to ~92% at N=32. * GenPRM-7B (Maj@8): Starts at ~84%, increases to ~90% at N=32. * Qwen2.5-Math-7B: Starts at ~86%, increases to ~93% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~85%, increases to ~92% at N=32. * Skywork-7B: Starts at ~85%, increases to ~91% at N=32. * Skywork-1.5B: Starts at ~84%, increases to ~90% at N=32. **Chart (b) AMC23 (Qwen):** * Pass@K: Starts at ~72%, increases to ~95% at N=32. * Maj.: Starts at ~72%, increases to ~94% at N=32. * GenPRM-7B (Pass@1): Starts at ~73%, increases to ~95% at N=32. * GenPRM-7B (Maj@8): Starts at ~72%, increases to ~94% at N=32. * Qwen2.5-Math-7B: Starts at ~73%, increases to ~95% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~72%, increases to ~94% at N=32. * Skywork-7B: Starts at ~72%, increases to ~94% at N=32. * Skywork-1.5B: Starts at ~71%, increases to ~93% at N=32. **Chart (c) AIME24 (Qwen):** * Pass@K: Starts at ~10%, increases to ~35% at N=32. * Maj.: Starts at ~10%, increases to ~33% at N=32. * GenPRM-7B (Pass@1): Starts at ~10%, increases to ~34% at N=32. * GenPRM-7B (Maj@8): Starts at ~10%, increases to ~32% at N=32. * Qwen2.5-Math-7B: Starts at ~12%, increases to ~35% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~11%, increases to ~33% at N=32. * Skywork-7B: Starts at ~11%, increases to ~33% at N=32. * Skywork-1.5B: Starts at ~10%, increases to ~32% at N=32. **Chart (d) Minerva Math (Qwen):** * Pass@K: Starts at ~33%, increases to ~50% at N=32. * Maj.: Starts at ~33%, increases to ~48% at N=32. * GenPRM-7B (Pass@1): Starts at ~33%, increases to ~49% at N=32. * GenPRM-7B (Maj@8): Starts at ~33%, increases to ~47% at N=32. * Qwen2.5-Math-7B: Starts at ~34%, increases to ~50% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~33%, increases to ~48% at N=32. * Skywork-7B: Starts at ~33%, increases to ~47% at N=32. * Skywork-1.5B: Starts at ~32%, increases to ~46% at N=32. **Chart (e) MATH (Gemma):** * Pass@K: Starts at ~84%, increases to ~94% at N=32. * Maj.: Starts at ~83%, increases to ~92% at N=32. * GenPRM-7B (Pass@1): Starts at ~84%, increases to ~93% at N=32. * GenPRM-7B (Maj@8): Starts at ~83%, increases to ~91% at N=32. * Qwen2.5-Math-7B: Starts at ~85%, increases to ~94% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~84%, increases to ~93% at N=32. * Skywork-7B: Starts at ~84%, increases to ~92% at N=32. * Skywork-1.5B: Starts at ~83%, increases to ~91% at N=32. **Chart (f) AMC23 (Gemma):** * Pass@K: Starts at ~70%, increases to ~95% at N=32. * Maj.: Starts at ~69%, increases to ~94% at N=32. * GenPRM-7B (Pass@1): Starts at ~71%, increases to ~95% at N=32. * GenPRM-7B (Maj@8): Starts at ~70%, increases to ~94% at N=32. * Qwen2.5-Math-7B: Starts at ~72%, increases to ~95% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~71%, increases to ~94% at N=32. * Skywork-7B: Starts at ~70%, increases to ~94% at N=32. * Skywork-1.5B: Starts at ~69%, increases to ~93% at N=32. **Chart (g) AIME24 (Gemma):** * Pass@K: Starts at ~15%, increases to ~40% at N=32. * Maj.: Starts at ~15%, increases to ~38% at N=32. * GenPRM-7B (Pass@1): Starts at ~16%, increases to ~39% at N=32. * GenPRM-7B (Maj@8): Starts at ~15%, increases to ~37% at N=32. * Qwen2.5-Math-7B: Starts at ~17%, increases to ~40% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~16%, increases to ~38% at N=32. * Skywork-7B: Starts at ~16%, increases to ~38% at N=32. * Skywork-1.5B: Starts at ~15%, increases to ~37% at N=32. **Chart (h) Minerva Math (Gemma):** * Pass@K: Starts at ~30%, increases to ~47% at N=32. * Maj.: Starts at ~30%, increases to ~45% at N=32. * GenPRM-7B (Pass@1): Starts at ~31%, increases to ~46% at N=32. * GenPRM-7B (Maj@8): Starts at ~30%, increases to ~44% at N=32. * Qwen2.5-Math-7B: Starts at ~32%, increases to ~47% at N=32. * Qwen2.5-Math-7B-PRM800K: Starts at ~31%, increases to ~45% at N=32. * Skywork-7B: Starts at ~31%, increases to ~44% at N=32. * Skywork-1.5B: Starts at ~30%, increases to ~43% at N=32. ### Key Observations * Accuracy generally increases with the number of solutions (N) across all datasets and models. * The Qwen2.5-Math-7B model consistently performs at or near the top across all datasets. * The Skywork-1.5B model generally exhibits the lowest accuracy. * The AIME24 dataset consistently shows the lowest overall accuracy compared to the other datasets. * The performance difference between the "Pass@K" and "Maj." metrics is minimal. * The performance of the models is relatively similar between the Owen and Gemma families. ### Interpretation The charts demonstrate the impact of increasing the number of solutions (N) on the accuracy of various language models in solving math problems. The consistent upward trend across all datasets suggests that providing more solutions improves the models' ability to arrive at the correct answer. The superior performance of the Qwen2.5-Math-7B model indicates its effectiveness in tackling these types of mathematical challenges. The lower accuracy on the AIME24 dataset suggests that this dataset presents a higher level of difficulty or requires different reasoning skills compared to the others. The relatively small differences between the models within each dataset suggest that the core architecture and training data play a significant role in performance, while the number of solutions acts as a supplementary factor. The consistent performance between the Owen and Gemma families suggests that the underlying model architecture is comparable, and the differences in performance are likely due to variations in training data or fine-tuning strategies.

## Line Graphs: Model Accuracy vs. Number of Solutions Across Datasets ### Overview The image contains eight line graphs comparing the accuracy of various AI models across four datasets (MATH, AMC23, AIME24, Minerva Math) as the number of solutions (N) increases. Each graph tests two problem sets (Qwen and Gemma) and evaluates models like Pass@K, GenPRM-7B variants, Direct GenPRM-7B, Qwen2.5-Math-7B, and Skywork models. Accuracy (%) is plotted against N (logarithmic scale: 2⁰ to 2⁸). --- ### Components/Axes - **X-axis**: Number of Solutions (N) – Logarithmic scale (2⁰, 2¹, ..., 2⁸). - **Y-axis**: Accuracy (%) – Ranges from ~30% to 95% depending on the dataset. - **Legends**: - **Colors**: - Gray (dotted): Pass@K (baseline). - Blue: GenPRM-7B (Pass@1). - Green: GenPRM-7B (Maj@8). - Orange: GenPRM-7B (Maj@4). - Red: Direct GenPRM-7B. - Purple: Qwen2.5-Math-7B. - Pink: Skywork-7B. - Brown: Qwen2.5-Math-7B-PRM800K. - Yellow: Skywork-1.5B. - **Placement**: Top-right corner of each graph. --- ### Detailed Analysis #### (a) MATH (Qwen) - **Trend**: Pass@K (gray dotted) slopes steeply upward (~85% at 2⁰ → ~94% at 2⁸). - **Models**: - GenPRM-7B (Pass@1, blue) starts at ~85% (2⁰) and reaches ~90% (2⁸). - GenPRM-7B (Maj@8, green) peaks at ~91% (2⁸). - Direct GenPRM-7B (red dashed) lags behind, peaking at ~88% (2⁸). - **Key**: Pass@K dominates; GenPRM-7B variants improve with N. #### (b) AMC23 (Qwen) - **Trend**: Pass@K rises from ~70% (2⁰) to ~95% (2⁸). - **Models**: - GenPRM-7B (Maj@4, orange) peaks at ~88% (2⁶) then declines slightly. - Qwen2.5-Math-7B (purple) stabilizes at ~85% (2⁸). - **Key**: Maj@4 configuration underperforms at higher N. #### (c) AIME24 (Qwen) - **Trend**: Pass@K increases from ~35% (2⁰) to ~52% (2⁸). - **Models**: - GenPRM-7B (Pass@1, blue) reaches ~42% (2⁸). - Skywork-7B (pink) peaks at ~38% (2⁸). - **Key**: All models lag Pass@K; Skywork-7B performs better than GenPRM-7B. #### (d) Minerva Math (Qwen) - **Trend**: Pass@K rises from ~32% (2⁰) to ~53% (2⁸). - **Models**: - GenPRM-7B (Maj@8, green) peaks at ~42% (2⁸). - Qwen2.5-Math-7B-PRM800K (brown) stabilizes at ~38% (2⁸). - **Key**: Maj@8 configuration outperforms others. #### (e) MATH (Gemma) - **Trend**: Pass@K rises from ~82% (2⁰) to ~94% (2⁸). - **Models**: - GenPRM-7B (Pass@1, blue) reaches ~89% (2⁸). - Skywork-1.5B (yellow) peaks at ~87% (2⁸). - **Key**: Skywork-1.5B closely matches GenPRM-7B. #### (f) AMC23 (Gemma) - **Trend**: Pass@K increases from ~65% (2⁰) to ~95% (2⁸). - **Models**: - GenPRM-7B (Maj@4, orange) peaks at ~85% (2⁶) then drops. - Qwen2.5-Math-7B (purple) stabilizes at ~80% (2⁸). - **Key**: Maj@4 configuration shows overfitting. #### (g) AIME24 (Gemma) - **Trend**: Pass@K rises from ~15% (2⁰) to ~40% (2⁸). - **Models**: - GenPRM-7B (Pass@1, blue) reaches ~35% (2⁸). - Skywork-7B (pink) peaks at ~32% (2⁸). - **Key**: Skywork-7B slightly outperforms GenPRM-7B. #### (h) Minerva Math (Gemma) - **Trend**: Pass@K increases from ~30% (2⁰) to ~50% (2⁸). - **Models**: - GenPRM-7B (Maj@8, green) peaks at ~40% (2⁸). - Skywork-1.5B (yellow) stabilizes at ~37% (2⁸). - **Key**: Maj@8 configuration leads. --- ### Key Observations 1. **Pass@K Baseline**: Consistently outperforms all models across datasets. 2. **Model Variants**: - GenPRM-7B (Maj@8) generally performs better than Pass@1 or Maj@4. - Direct GenPRM-7B (red dashed) underperforms variants. 3. **Diminishing Returns**: Some models (e.g., GenPRM-7B Maj@4 in AMC23) plateau or decline at higher N. 4. **Skywork Models**: Competitive with GenPRM-7B in MATH and AIME24 but lag in AMC23. --- ### Interpretation - **Scalability**: Increasing N improves accuracy, but gains diminish as N grows (e.g., AMC23 Qwen). - **Model Efficiency**: GenPRM-7B variants with majority voting (Maj@8) often outperform simpler configurations. - **Dataset-Specific Behavior**: - AIME24 (high-difficulty math) shows the largest gap between models and Pass@K. - Minerva Math (reasoning-focused) benefits more from majority voting. - **Anomalies**: - Direct GenPRM-7B (red dashed) consistently underperforms, suggesting architectural limitations. - Skywork-1.5B matches GenPRM-7B in MATH but struggles in AMC23, indicating dataset-specific weaknesses. This analysis highlights the trade-offs between model complexity, voting strategies, and dataset characteristics in solving math problems.