## Line Charts: Model Performance on Math Problems ### Overview The image contains four line charts comparing the performance of different language models on various math problem sets. The charts display accuracy (%) as a function of the number of sampled solutions. The models compared are "o1-preview", "o1-mini", "rStar-Math (7B SLM+7B PPM)", "Qwen2.5 Best-of-N (7B SLM+72B ORM)", and "Qwen2.5 Best-of-N (72B LLM+72B ORM)". The problem sets are "MATH", "AIME 2024", "Olympiad Bench", and "College Math". ### Components/Axes * **X-axis (Horizontal):** "#Sampled Solutions" with values 1, 2, 4, 8, 16, 32, 64. * **Y-axis (Vertical):** "Accuracy (%)" with values ranging from 80 to 90 for the MATH chart, 20 to 40 for the AIME 2024 chart, 50 to 60 for the Olympiad Bench chart, and 50 to 60 for the College Math chart. * **Legends (Top):** * `o1-preview`: Light Blue, dashed-dotted line. * `o1-mini`: Red, dashed line. * `rStar-Math (7B SLM+7B PPM)`: Green, solid line with pentagon markers. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)`: Purple, dotted line with square markers. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)`: Yellow, dashed line with hexagon markers. * **Chart Titles (Top):** MATH, AIME 2024, Olympiad Bench, College Math. ### Detailed Analysis **1. MATH Chart:** * `o1-preview` (Light Blue, dashed-dotted line): Constant accuracy at approximately 86%. * `o1-mini` (Red, dashed line): Constant accuracy at approximately 90%. * `rStar-Math (7B SLM+7B PPM)` (Green, solid line with pentagon markers): Starts at approximately 78% accuracy with 1 sampled solution, increases sharply to approximately 87% at 2 sampled solutions, then gradually increases to approximately 90% at 64 sampled solutions. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)` (Purple, dotted line with square markers): Starts at approximately 82% accuracy with 1 sampled solution, increases to approximately 87% at 8 sampled solutions, and then plateaus. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)` (Yellow, dashed line with hexagon markers): Starts at approximately 84% accuracy with 1 sampled solution, increases to approximately 87% at 8 sampled solutions, and then plateaus. **2. AIME 2024 Chart:** * `o1-preview` (Light Blue, dashed-dotted line): Constant accuracy at approximately 42%. * `o1-mini` (Red, dashed line): Constant accuracy at approximately 45%. * `rStar-Math (7B SLM+7B PPM)` (Green, solid line with pentagon markers): Starts at approximately 25% accuracy with 1 sampled solution, increases to approximately 45% at 8 sampled solutions, and then plateaus. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)` (Purple, dotted line with square markers): Starts at approximately 5% accuracy with 1 sampled solution, increases to approximately 25% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)` (Yellow, dashed line with hexagon markers): Starts at approximately 20% accuracy with 1 sampled solution, increases to approximately 38% at 64 sampled solutions. **3. Olympiad Bench Chart:** * `o1-preview` (Light Blue, dashed-dotted line): Constant accuracy at approximately 42%. * `o1-mini` (Red, dashed line): Constant accuracy at approximately 62%. * `rStar-Math (7B SLM+7B PPM)` (Green, solid line with pentagon markers): Starts at approximately 50% accuracy with 1 sampled solution, increases to approximately 62% at 8 sampled solutions, and then plateaus. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)` (Purple, dotted line with square markers): Starts at approximately 48% accuracy with 1 sampled solution, increases to approximately 53% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)` (Yellow, dashed line with hexagon markers): Starts at approximately 50% accuracy with 1 sampled solution, increases to approximately 55% at 64 sampled solutions. **4. College Math Chart:** * `o1-preview` (Light Blue, dashed-dotted line): Constant accuracy at approximately 42%. * `o1-mini` (Red, dashed line): Constant accuracy at approximately 58%. * `rStar-Math (7B SLM+7B PPM)` (Green, solid line with pentagon markers): Starts at approximately 52% accuracy with 1 sampled solution, increases to approximately 62% at 64 sampled solutions. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)` (Purple, dotted line with square markers): Starts at approximately 47% accuracy with 1 sampled solution, increases to approximately 51% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)` (Yellow, dashed line with hexagon markers): Starts at approximately 48% accuracy with 1 sampled solution, increases to approximately 52% at 64 sampled solutions. ### Key Observations * `o1-mini` consistently achieves the highest accuracy among all models on the MATH and Olympiad Bench datasets. * `rStar-Math` shows a significant improvement in accuracy as the number of sampled solutions increases, particularly in the AIME 2024 dataset. * `o1-preview` shows a constant accuracy across all numbers of sampled solutions. * The performance of `Qwen2.5 Best-of-N` models is generally lower than `rStar-Math` and `o1-mini` across all datasets. ### Interpretation The charts illustrate the performance of different language models on various math problem sets, highlighting the impact of the number of sampled solutions on accuracy. The `o1-mini` model appears to be the most effective on the MATH and Olympiad Bench datasets, while `rStar-Math` demonstrates a notable improvement with increased sampling, especially on the AIME 2024 dataset. The consistent performance of `o1-preview` suggests a stable but potentially less adaptable model. The `Qwen2.5 Best-of-N` models show lower overall accuracy compared to the other models, indicating potential areas for improvement in their architecture or training. The data suggests that increasing the number of sampled solutions can significantly enhance the performance of certain models, particularly `rStar-Math`, while others, like `o1-preview`, may benefit more from architectural or training enhancements.

\n ## Chart: Model Performance on Math Benchmarks vs. Sampled Solutions ### Overview The image presents a comparative analysis of four language models – o1-preview, o1-mini, rStar-Math, and Qwen2.5 (with two variants: Best-of-N and LLM+ORM) – across four distinct math benchmarks: MATH, AIME 2024, Olympiad Bench, and College Math. The performance metric is accuracy (in percentage), plotted against the number of sampled solutions. Each benchmark is represented by a separate subplot. ### Components/Axes * **X-axis (all subplots):** "#Sampled Solutions" with markers at 1, 2, 4, 8, 16, 32, and 64. * **Y-axis (all subplots):** "Accuracy (%)" ranging from approximately 20% to 90%, with increments of 10%. * **Legend (top-right):** * `o1-preview` (light blue, solid line) * `o1-mini` (red, dashed line) * `rStar-Math` (green, solid line with triangle markers) * `Qwen2.5 Best-of-N (7B SLM+72B ORM)` (purple, dotted line) * `Qwen2.5 Best-of-N (72B LLM+72B ORM)` (orange, dotted line with circle markers) * **Subplot Titles (top-center of each subplot):** * MATH * AIME 2024 * Olympiad Bench * College Math ### Detailed Analysis or Content Details **MATH Subplot:** * `rStar-Math`: Starts at approximately 82% accuracy at 1 sampled solution, rises sharply to around 92% at 8 sampled solutions, and plateaus around 92-93% for the remaining sampled solutions. * `o1-preview`: Starts at approximately 80% accuracy at 1 sampled solution, increases steadily to around 88% at 64 sampled solutions. * `o1-mini`: Starts at approximately 78% accuracy at 1 sampled solution, increases to around 85% at 64 sampled solutions. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)`: Starts at approximately 75% accuracy at 1 sampled solution, increases to around 82% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)`: Remains relatively flat around 80-82% accuracy across all sampled solutions. **AIME 2024 Subplot:** * `rStar-Math`: Shows a steep increase from approximately 20% at 1 sampled solution to around 45% at 32 sampled solutions, then plateaus around 45-47%. * `o1-preview`: Increases steadily from approximately 20% at 1 sampled solution to around 35% at 64 sampled solutions. * `o1-mini`: Starts at approximately 15% accuracy at 1 sampled solution, increases to around 25% at 64 sampled solutions. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)`: Increases from approximately 15% at 1 sampled solution to around 28% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)`: Starts at approximately 18% accuracy at 1 sampled solution, increases to around 30% at 64 sampled solutions. **Olympiad Bench Subplot:** * `rStar-Math`: Increases rapidly from approximately 45% at 1 sampled solution to around 68% at 32 sampled solutions, then plateaus around 68-70%. * `o1-preview`: Increases steadily from approximately 45% at 1 sampled solution to around 58% at 64 sampled solutions. * `o1-mini`: Starts at approximately 40% accuracy at 1 sampled solution, increases to around 50% at 64 sampled solutions. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)`: Increases from approximately 40% at 1 sampled solution to around 52% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)`: Remains relatively flat around 50-52% accuracy across all sampled solutions. **College Math Subplot:** * `rStar-Math`: Increases from approximately 52% at 1 sampled solution to around 65% at 64 sampled solutions. * `o1-preview`: Increases steadily from approximately 52% at 1 sampled solution to around 62% at 64 sampled solutions. * `o1-mini`: Starts at approximately 50% accuracy at 1 sampled solution, increases to around 58% at 64 sampled solutions. * `Qwen2.5 Best-of-N (7B SLM+72B ORM)`: Increases from approximately 50% at 1 sampled solution to around 56% at 64 sampled solutions. * `Qwen2.5 Best-of-N (72B LLM+72B ORM)`: Remains relatively flat around 50-52% accuracy across all sampled solutions. ### Key Observations * `rStar-Math` consistently outperforms other models on all benchmarks, especially at higher numbers of sampled solutions. * The performance of `Qwen2.5 Best-of-N (72B LLM+72B ORM)` plateaus quickly and does not improve significantly with more sampled solutions. * `o1-preview` generally outperforms `o1-mini` across all benchmarks. * The AIME 2024 benchmark shows the lowest overall accuracy scores compared to the other benchmarks. ### Interpretation The data suggests that increasing the number of sampled solutions generally improves the accuracy of the language models, but the rate of improvement diminishes as the number of samples increases. `rStar-Math` demonstrates a superior ability to leverage sampled solutions for improved performance, particularly on the more challenging benchmarks like MATH and Olympiad Bench. The plateauing performance of `Qwen2.5 Best-of-N (72B LLM+72B ORM)` indicates a potential limitation in its architecture or training data, preventing it from effectively utilizing additional sampled solutions. The lower accuracy scores on the AIME 2024 benchmark suggest that this benchmark presents a unique challenge for these models, potentially requiring different strategies or training data. The consistent performance differences between the models highlight the importance of model architecture and training methodology in achieving high accuracy on math benchmarks.

## Multi-Panel Line Chart: Model Accuracy vs. Number of Sampled Solutions ### Overview The image contains four line charts (panels) comparing the accuracy (in percentage) of five models across varying numbers of sampled solutions (x - axis: 1, 2, 4, 8, 16, 32, 64, logarithmic base - 2 scale). The panels are labeled “MATH”, “AIME 2024”, “Olympiad Bench”, and “College Math”. The y - axis for each panel represents “Accuracy (%)” with different ranges per panel. ### Components/Axes - **X - axis (all panels)**: “#Sampled Solutions” with values 1, 2, 4, 8, 16, 32, 64 (logarithmic, base - 2 scale). - **Y - axis (per panel)**: - MATH: 78–90% - AIME 2024: 20–45% - Olympiad Bench: 45–65% - College Math: 45–60% - **Legend (top of image)**: - o1 - preview: Blue dashed line (flat across sampled solutions). - o1 - mini: Red dashed line (flat across sampled solutions). - rStar - Math (7B SLM + 7B PPM): Teal solid line with circular markers (increasing trend). - Qwen2.5 Best - of - N (7B SLM + 72B ORM): Purple dotted line with square markers (increasing trend). - Qwen2.5 Best - of - N (72B LLM + 72B ORM): Yellow dotted line with circular markers (increasing trend). ### Detailed Analysis (Per Panel) #### 1. MATH Panel - **o1 - preview (blue dashed)**: Flat at ~85% accuracy (no change with sampled solutions). - **o1 - mini (red dashed)**: Flat at ~90% accuracy (no change with sampled solutions). - **rStar - Math (teal)**: Starts at ~78% (x = 1), rises to ~90% (x = 64). Trend: Strongly increasing with sampled solutions. - **Qwen2.5 (7B SLM + 72B ORM, purple squares)**: Starts at ~82% (x = 1), rises to ~88% (x = 64). Trend: Increasing. - **Qwen2.5 (72B LLM + 72B ORM, yellow circles)**: Starts at ~83% (x = 1), rises to ~87% (x = 64). Trend: Increasing. #### 2. AIME 2024 Panel - **o1 - preview (blue dashed)**: Flat at ~45% accuracy (no change with sampled solutions). - **o1 - mini (red dashed)**: Flat at ~45% accuracy (no change with sampled solutions). - **rStar - Math (teal)**: Starts at ~25% (x = 1), rises to ~45% (x = 64). Trend: Increasing. - **Qwen2.5 (7B SLM + 72B ORM, purple squares)**: Starts at ~15% (x = 1), rises to ~30% (x = 64). Trend: Increasing. - **Qwen2.5 (72B LLM + 72B ORM, yellow circles)**: Starts at ~20% (x = 1), rises to ~35% (x = 64). Trend: Increasing. #### 3. Olympiad Bench Panel - **o1 - preview (blue dashed)**: Flat at ~65% accuracy (no change with sampled solutions). - **o1 - mini (red dashed)**: Flat at ~65% accuracy (no change with sampled solutions). - **rStar - Math (teal)**: Starts at ~50% (x = 1), rises to ~65% (x = 64). Trend: Increasing. - **Qwen2.5 (7B SLM + 72B ORM, purple squares)**: Starts at ~45% (x = 1), rises to ~55% (x = 64). Trend: Increasing. - **Qwen2.5 (72B LLM + 72B ORM, yellow circles)**: Starts at ~48% (x = 1), rises to ~58% (x = 64). Trend: Increasing. #### 4. College Math Panel - **o1 - preview (blue dashed)**: Flat at ~58% accuracy (no change with sampled solutions). - **o1 - mini (red dashed)**: Flat at ~58% accuracy (no change with sampled solutions). - **rStar - Math (teal)**: Starts at ~52% (x = 1), rises to ~60% (x = 64). Trend: Increasing. - **Qwen2.5 (7B SLM + 72B ORM, purple squares)**: Starts at ~45% (x = 1), rises to ~50% (x = 64). Trend: Increasing. - **Qwen2.5 (72B LLM + 72B ORM, yellow circles)**: Starts at ~47% (x = 1), rises to ~52% (x = 64). Trend: Increasing. ### Key Observations - **Flat Trends (o1 - preview, o1 - mini)**: These models show no accuracy improvement with more sampled solutions (flat lines), indicating their performance is independent of the number of solutions sampled. - **Increasing Trends (rStar - Math, Qwen2.5 variants)**: All three models with “Best - of - N” or “rStar - Math” show accuracy increasing with more sampled solutions, meaning sampling more solutions improves their performance. - **Model Comparison**: - In MATH, o1 - mini (red) outperforms o1 - preview (blue) and Qwen2.5 variants, while rStar - Math approaches o1 - mini’s accuracy at high sampled solutions. - In AIME 2024, Olympiad Bench, and College Math, o1 - preview and o1 - mini (both ~45%, ~65%, ~58% respectively) outperform Qwen2.5 variants, with rStar - Math approaching their accuracy at x = 64. ### Interpretation The data implies that o1 - preview and o1 - mini have stable accuracy regardless of the number of sampled solutions, suggesting their performance is robust or not dependent on sampling more solutions. In contrast, rStar - Math and Qwen2.5 Best - of - N models benefit from more sampled solutions, with accuracy increasing as the number of solutions sampled grows. This could mean these models rely on sampling multiple solutions to find the best one (e.g., via a “best - of - N” strategy), while o1 - preview and o1 - mini might have a more deterministic or single - solution approach. The consistent outperformance of o1 - preview and o1 - mini across most panels (except MATH, where rStar - Math catches up) suggests they are more effective for these math - related tasks, especially when sampling solutions is not a factor. The increasing trend for rStar - Math and Qwen2.5 variants highlights the value of sampling more solutions for these models, potentially due to their reliance on generating multiple candidates and selecting the best one.

## Line Graphs: Model Performance Across Benchmarks ### Overview The image contains four line graphs comparing the accuracy of different AI models across four benchmarks: MATH, AIME 2024, Olympiad Bench, and College Math. Each graph plots accuracy (%) against the number of sampled solutions (1, 2, 4, 8, 16, 32, 64). Models include o1-preview, o1-mini, rStar-Math, and two variants of Gwen2.5 Best-of-N. ### Components/Axes - **X-axis**: "#Sampled Solutions" (logarithmic scale: 1, 2, 4, 8, 16, 32, 64). - **Y-axis**: "Accuracy (%)" (ranging from ~75% to 90% in MATH, ~20% to 90% in AIME 2024, ~50% to 60% in Olympiad Bench, and ~50% to 60% in College Math). - **Legends**: Located in the top-left corner of each graph, with distinct line styles and colors: - **o1-preview**: Blue dashed line. - **o1-mini**: Red dashed line. - **rStar-Math (7B SLM+7B PPM)**: Green solid line. - **Gwen2.5 Best-of-N (7B SLM+72B ORM)**: Purple dotted line. - **Gwen2.5 Best-of-N (72B LLM+72B ORM)**: Yellow dotted line. ### Detailed Analysis #### MATH - **rStar-Math**: Starts at ~75% accuracy with 1 sample, rising sharply to ~90% by 64 samples. - **o1-mini**: Flat at ~90% accuracy across all sample counts. - **o1-preview**: Begins at ~85%, plateaus near 85% after 8 samples. - **Gwen2.5 Models**: Both variants start below 85%, with the 72B LLM+ORM variant reaching ~88% at 64 samples. #### AIME 2024 - **rStar-Math**: Starts at ~20%, rising to ~85% by 64 samples. - **o1-mini**: Flat at ~40% accuracy. - **o1-preview**: Flat at ~40% accuracy. - **Gwen2.5 Models**: Both variants plateau near ~30% accuracy. #### Olympiad Bench - **rStar-Math**: Starts at ~50%, rising to ~60% by 64 samples. - **o1-mini**: Flat at ~60% accuracy. - **o1-preview**: Starts at ~50%, plateaus near 55%. - **Gwen2.5 Models**: Both variants reach ~55% accuracy at 64 samples. #### College Math - **rStar-Math**: Starts at ~50%, rising to ~60% by 64 samples. - **o1-mini**: Flat at ~55% accuracy. - **o1-preview**: Starts at ~50%, plateaus near 55%. - **Gwen2.5 Models**: Both variants reach ~55% accuracy at 64 samples. ### Key Observations 1. **rStar-Math** consistently improves with more samples across all benchmarks, outperforming other models in MATH and AIME 2024. 2. **o1-mini** maintains flat performance, suggesting limited sensitivity to sampling. 3. **Gwen2.5 Models** underperform compared to rStar-Math and o1-mini, with the 72B LLM+ORM variant slightly outperforming the 7B SLM+ORM variant. 4. **o1-preview** shows minimal improvement, indicating potential inefficiencies in scaling. ### Interpretation The data suggests that **rStar-Math** is the most effective model for these benchmarks, particularly in MATH and AIME 2024, where it achieves near-human-level accuracy with sufficient sampling. The **o1-mini** model’s flat performance implies it may be optimized for specific tasks or constrained by architectural limitations. The **Gwen2.5 Models**’ lower accuracy could reflect suboptimal configurations (e.g., smaller parameter sizes) or training data gaps. The **o1-preview**’s stagnant performance raises questions about its scalability or training methodology. Overall, the results highlight the importance of model architecture and sampling efficiency in achieving high accuracy.