## Bar Chart: Model Accuracy vs. Problem Difficulty ### Overview The image presents two bar charts comparing the accuracy of two models, ThinkPRM-14B and DiscPRM-14B, across problems binned by difficulty. The left chart focuses on "Math-500" problems, while the right chart focuses on "GPQA-Physics" problems. Both charts display accuracy (in percentage) on the y-axis and problem difficulty (binned) on the x-axis. ### Components/Axes * **Titles:** * Left Chart: "Best-of-32: Math-500" * Right Chart: "Best-of-32: GPQA-Physics" * **Y-axis:** "Accuracy (%)", ranging from 0 to 100 in increments of 20. * **X-axis:** "Problems binned by difficulty". * Left Chart: Categories 1, 2, 3, 4, 5 * Right Chart: Categories 1, 2, 3, 4 * **Legend:** Located at the bottom of the image. * Orange: "ThinkPRM-14B" * Teal: "DiscPRM-14B" ### Detailed Analysis **Left Chart: Math-500** * **ThinkPRM-14B (Orange):** * Difficulty 1: Accuracy ~97% * Difficulty 2: Accuracy ~80% * Difficulty 3: Accuracy ~92% * Difficulty 4: Accuracy ~72% * Difficulty 5: Accuracy ~48% * Trend: Generally decreasing accuracy as difficulty increases. * **DiscPRM-14B (Teal):** * Difficulty 1: Accuracy ~97% * Difficulty 2: Accuracy ~80% * Difficulty 3: Accuracy ~83% * Difficulty 4: Accuracy ~58% * Difficulty 5: Accuracy ~36% * Trend: Decreasing accuracy as difficulty increases. **Right Chart: GPQA-Physics** * **ThinkPRM-14B (Orange):** * Difficulty 1: Accuracy ~99% * Difficulty 2: Accuracy ~97% * Difficulty 3: Accuracy ~57% * Difficulty 4: Accuracy ~14% * Trend: Decreasing accuracy as difficulty increases. * **DiscPRM-14B (Teal):** * Difficulty 1: Accuracy ~99% * Difficulty 2: Accuracy ~79% * Difficulty 3: Accuracy ~40% * Difficulty 4: Accuracy ~10% * Trend: Decreasing accuracy as difficulty increases. ### Key Observations * Both models perform well on easier problems (difficulty 1 and 2) for both Math-500 and GPQA-Physics. * Accuracy decreases as problem difficulty increases for both models and both problem sets. * For Math-500, ThinkPRM-14B generally outperforms DiscPRM-14B, especially at difficulty levels 3, 4, and 5. * For GPQA-Physics, ThinkPRM-14B outperforms DiscPRM-14B at difficulty levels 3 and 4. * The drop in accuracy is more pronounced for GPQA-Physics than for Math-500 as difficulty increases. ### Interpretation The data suggests that both models are more proficient at solving easier problems. As the difficulty level increases, the accuracy of both models decreases, indicating a limitation in their ability to handle more complex problems. The ThinkPRM-14B model generally performs better than the DiscPRM-14B model, especially for Math-500 problems. The more significant drop in accuracy for GPQA-Physics problems suggests that these problems are inherently more challenging for both models compared to Math-500 problems. The "Best-of-32" prefix in the titles likely refers to the number of attempts or samples used to determine the accuracy, implying a statistical approach to evaluating model performance.

\n ## Bar Charts: Model Accuracy on Math and Physics Problems ### Overview The image presents two side-by-side bar charts comparing the accuracy of two models, ThinkPRM-14B (orange) and DiscPRM-14B (teal), on math and physics problems. The x-axis represents problems binned by difficulty (1 to 5), and the y-axis represents accuracy in percentage (%). The left chart focuses on Math-500 problems, while the right chart focuses on GPQA-Physics problems. ### Components/Axes * **X-axis Label (Both Charts):** "Problems binned by difficulty" * **Y-axis Label (Both Charts):** "Accuracy (%)" * **Left Chart Title:** "Best-of-32: Math-500" * **Right Chart Title:** "Best-of-32: GPQA-Physics" * **Legend (Bottom Center):** * Orange: "ThinkPRM-14B" * Teal: "DiscPRM-14B" * **X-axis Markers (Both Charts):** 1, 2, 3, 4, 5 (representing difficulty levels) * **Y-axis Markers (Both Charts):** 0, 20, 40, 60, 80, 100 ### Detailed Analysis or Content Details **Left Chart (Math-500):** * **DiscPRM-14B (Teal):** * Difficulty 1: Approximately 95% accuracy. * Difficulty 2: Approximately 80% accuracy. * Difficulty 3: Approximately 85% accuracy. * Difficulty 4: Approximately 70% accuracy. * Difficulty 5: Approximately 45% accuracy. * Trend: The teal bars generally decrease in height from difficulty 1 to 5, indicating decreasing accuracy with increasing difficulty. * **ThinkPRM-14B (Orange):** * Difficulty 1: Approximately 98% accuracy. * Difficulty 2: Approximately 90% accuracy. * Difficulty 3: Approximately 95% accuracy. * Difficulty 4: Approximately 70% accuracy. * Difficulty 5: Approximately 40% accuracy. * Trend: The orange bars also generally decrease in height from difficulty 1 to 5, mirroring the teal bars. **Right Chart (GPQA-Physics):** * **DiscPRM-14B (Teal):** * Difficulty 1: Approximately 100% accuracy. * Difficulty 2: Approximately 80% accuracy. * Difficulty 3: Approximately 60% accuracy. * Difficulty 4: Approximately 10% accuracy. * Trend: The teal bars show a significant decrease in height from difficulty 1 to 4. * **ThinkPRM-14B (Orange):** * Difficulty 1: Approximately 100% accuracy. * Difficulty 2: Approximately 95% accuracy. * Difficulty 3: Approximately 70% accuracy. * Difficulty 4: Approximately 15% accuracy. * Trend: The orange bars also show a decrease in height from difficulty 1 to 4, but the decrease is less pronounced than for the teal bars. ### Key Observations * In both charts, both models perform best on the easiest problems (difficulty 1) and their performance degrades as the difficulty increases. * For Math-500, ThinkPRM-14B consistently outperforms DiscPRM-14B across all difficulty levels, though the difference is not substantial. * For GPQA-Physics, ThinkPRM-14B also generally outperforms DiscPRM-14B, especially at higher difficulty levels. The performance drop for DiscPRM-14B is more dramatic on the physics problems. * The accuracy of both models on the most difficult problems (difficulty 5 for Math-500 and difficulty 4 for GPQA-Physics) is significantly lower than on easier problems. ### Interpretation The data suggests that both ThinkPRM-14B and DiscPRM-14B are capable of solving math and physics problems, but their performance is highly sensitive to the difficulty of the problems. ThinkPRM-14B appears to be slightly more robust to increasing difficulty, particularly in the GPQA-Physics domain. The substantial drop in accuracy for both models on the most difficult problems indicates a limitation in their ability to handle complex reasoning or knowledge requirements. The difference in performance between the two models on the physics problems could be due to differences in their training data or architectures, potentially making ThinkPRM-14B better suited for the specific challenges posed by physics questions. The charts provide a comparative performance assessment of the two models across different problem difficulties, highlighting their strengths and weaknesses. The consistent trend of decreasing accuracy with increasing difficulty is expected, as harder problems inherently require more sophisticated problem-solving skills.

\n ## Grouped Bar Charts: Model Accuracy by Problem Difficulty ### Overview The image displays two side-by-side grouped bar charts comparing the performance of two AI models, "ThinkPRM-14B" and "DiscPRM-14B," on two different problem sets. The charts show the percentage accuracy of each model as problem difficulty increases. The overall trend indicates that accuracy decreases for both models as problems become more difficult, with ThinkPRM-14B consistently outperforming DiscPRM-14B, especially on harder problems. ### Components/Axes * **Chart Titles:** * Left Chart: "Best-of-32: Math-500" * Right Chart: "Best-of-32: GPQA-Physics" * **Y-Axis (Both Charts):** Labeled "Accuracy (%)". The scale runs from 0 to 100 in increments of 20. * **X-Axis (Both Charts):** Labeled "Problems binned by difficulty". The bins are numbered sequentially. * Left Chart (Math-500): Bins 1, 2, 3, 4, 5. * Right Chart (GPQA-Physics): Bins 1, 2, 3, 4. * **Legend:** Located at the bottom center of the image, spanning both charts. * Orange square: "ThinkPRM-14B" * Teal square: "DiscPRM-14B" * **Data Series:** Each difficulty bin contains two bars, one for each model, placed side-by-side. The left bar in each pair is teal (DiscPRM-14B), and the right bar is orange (ThinkPRM-14B). ### Detailed Analysis **Left Chart: Best-of-32: Math-500** * **Trend Verification:** Both models show a clear downward trend in accuracy as difficulty increases from bin 1 to bin 5. ThinkPRM-14B's decline is less steep. * **Data Points (Approximate):** * **Bin 1:** DiscPRM-14B ≈ 98%, ThinkPRM-14B ≈ 98%. (Nearly identical, very high accuracy). * **Bin 2:** DiscPRM-14B ≈ 80%, ThinkPRM-14B ≈ 80%. (Identical, significant drop from bin 1). * **Bin 3:** DiscPRM-14B ≈ 84%, ThinkPRM-14B ≈ 92%. (ThinkPRM shows a notable advantage). * **Bin 4:** DiscPRM-14B ≈ 58%, ThinkPRM-14B ≈ 72%. (ThinkPRM maintains a ~14 percentage point lead). * **Bin 5:** DiscPRM-14B ≈ 36%, ThinkPRM-14B ≈ 48%. (Both models struggle, ThinkPRM retains a ~12 point lead). **Right Chart: Best-of-32: GPQA-Physics** * **Trend Verification:** A very steep downward trend for both models. The performance gap between the models widens significantly in the middle bins before converging at the highest difficulty. * **Data Points (Approximate):** * **Bin 1:** DiscPRM-14B ≈ 100%, ThinkPRM-14B ≈ 100%. (Perfect or near-perfect accuracy). * **Bin 2:** DiscPRM-14B ≈ 79%, ThinkPRM-14B ≈ 100%. (ThinkPRM maintains perfect accuracy, while DiscPRM drops significantly). * **Bin 3:** DiscPRM-14B ≈ 40%, ThinkPRM-14B ≈ 60%. (Both drop sharply, ThinkPRM leads by ~20 points). * **Bin 4:** DiscPRM-14B ≈ 10%, ThinkPRM-14B ≈ 15%. (Both models perform very poorly, with a minimal difference). ### Key Observations 1. **Consistent Superiority:** ThinkPRM-14B (orange bars) achieves equal or higher accuracy than DiscPRM-14B (teal bars) in every single difficulty bin across both subjects. 2. **Difficulty Impact:** The "GPQA-Physics" problems appear to be more challenging overall, especially at higher difficulties. While both models start at 100% in bin 1, their accuracy plummets to 10-15% by bin 4. In contrast, the "Math-500" decline is more gradual, with both models still scoring between 36-48% at bin 5. 3. **Performance Gap:** The advantage of ThinkPRM-14B is most pronounced in the middle difficulty ranges (Bin 3 for Math, Bins 2 & 3 for Physics). This suggests its reasoning capabilities are particularly beneficial for moderately complex problems. 4. **Convergence at Extremes:** At the easiest problems (Bin 1 for both, Bin 2 for Physics/ThinkPRM) and the hardest problems (Bin 5 for Math, Bin 4 for Physics), the performance gap between the two models narrows or disappears. ### Interpretation The data demonstrates a clear hierarchy in problem-solving capability between the two models under a "Best-of-32" sampling strategy. ThinkPRM-14B exhibits more robust reasoning, as evidenced by its slower decay in accuracy with increasing problem difficulty. This is especially evident in the physics domain, where it maintains perfect accuracy one bin longer than DiscPRM-14B. The stark difference in the slope of decline between Math-500 and GPQA-Physics suggests the nature of the challenges differs. The physics problems likely involve more complex, multi-step reasoning or specialized knowledge that causes a catastrophic drop in performance for both models once a certain complexity threshold is crossed. The math problems, while difficult, allow for a more linear degradation of performance. From a Peircean perspective, the charts are an indexical sign of the models' underlying reasoning architecture. The consistent performance gap points to a fundamental difference in how ThinkPRM and DiscPRM process and solve problems, with ThinkPRM's approach being more resilient to complexity. The charts do not reveal *why* this is the case, but they provide strong evidence *that* it is the case, prompting investigation into the architectural or training differences between the "Think" and "Disc" paradigms.

## Bar Charts: Best-of-32 Performance Comparison ### Overview The image contains two side-by-side bar charts comparing the accuracy of two models, **ThinkPRM-14B** (orange) and **DiscPRM-14B** (teal), across problem sets binned by difficulty. The left chart evaluates performance on **Math-500**, while the right chart evaluates **GPQA-Physics**. Each chart uses a "Best-of-32" evaluation protocol, with accuracy reported in percentage. ### Components/Axes - **Left Chart (Math-500)**: - **X-axis**: Problems binned by difficulty (1–5). - **Y-axis**: Accuracy (%) from 0 to 100. - **Legend**: Orange = ThinkPRM-14B, Teal = DiscPRM-14B. - **Right Chart (GPQA-Physics)**: - **X-axis**: Problems binned by difficulty (1–4). - **Y-axis**: Accuracy (%) from 0 to 100. - **Legend**: Orange = ThinkPRM-14B, Teal = DiscPRM-14B. ### Detailed Analysis #### Math-500 (Left Chart) - **Problem 1**: - ThinkPRM-14B: ~98% - DiscPRM-14B: ~98% - **Problem 2**: - ThinkPRM-14B: ~80% - DiscPRM-14B: ~80% - **Problem 3**: - ThinkPRM-14B: ~88% - DiscPRM-14B: ~82% - **Problem 4**: - ThinkPRM-14B: ~70% - DiscPRM-14B: ~58% - **Problem 5**: - ThinkPRM-14B: ~48% - DiscPRM-14B: ~36% #### GPQA-Physics (Right Chart) - **Problem 1**: - ThinkPRM-14B: ~100% - DiscPRM-14B: ~100% - **Problem 2**: - ThinkPRM-14B: ~100% - DiscPRM-14B: ~78% - **Problem 3**: - ThinkPRM-14B: ~60% - DiscPRM-14B: ~40% - **Problem 4**: - ThinkPRM-14B: ~14% - DiscPRM-14B: ~10% ### Key Observations 1. **Math-500**: - Both models show declining accuracy with increasing difficulty. - DiscPRM-14B consistently underperforms ThinkPRM-14B in Problems 3–5. - Problem 5 has the largest gap (~12% difference). 2. **GPQA-Physics**: - ThinkPRM-14B dominates in Problems 1–2 but collapses in Problem 4. - DiscPRM-14B maintains higher accuracy in Problem 3 but also drops sharply in Problem 4. - Problem 4 has a drastic performance gap (~50% difference). 3. **General Trends**: - Both models struggle with higher-difficulty problems. - DiscPRM-14B exhibits more consistent performance in Math-500 but falters in Physics. ### Interpretation The data suggests that **ThinkPRM-14B** excels in lower-difficulty problems across both domains but experiences significant performance degradation in higher-difficulty tasks. **DiscPRM-14B** performs more consistently in Math-500 but struggles disproportionately in GPQA-Physics, particularly in Problem 4. The stark drop in Problem 4 for both models in GPQA-Physics may indicate a fundamental limitation in handling complex physics problems, even when binned as "high difficulty." The absence of Problem 5 in GPQA-Physics (compared to Math-500) could reflect either data scarcity or a different difficulty distribution between the two datasets. The "Best-of-32" protocol implies that these results represent the best performance across 32 trials, suggesting that even the models' optimal outputs degrade under increased problem complexity.