## Chart: Scaling Performance and Efficiency Analysis ### Overview The image presents four scatter plots analyzing the scaling performance and efficiency of different models across various tasks. The first three plots (a, b, c) depict "Task Accuracy vs. Model Size" for Complex Reasoning, Math Reasoning, and Question-Answering tasks, respectively. The fourth plot (d) shows "Task Runtime vs. Complexity" for Neuro-symbolic and RL-based models. ### Components/Axes **General:** * **Title:** Scaling Performance Analysis: Task Accuracy vs. Model Size (for plots a, b, c) and Scaling Efficiency Analysis: Task Runtime vs. Complexity (for plot d) * Each plot is labeled with a letter: (a), (b), (c), (d) in the bottom left corner. **Plot (a): Complex Reasoning Tasks** * **X-axis:** Model Size (7B, 8B, 13B, 70B, GPT) * **Y-axis:** Task Accuracy (%) (Scale from 0 to 100, increments of 20) * **Legend (top-left):** * Blue Square: Textedit (C) * Orange Triangle: ACLUTRR (C) * Green Circle: ProofWriter (C) * Gray Square: Textedit (M) * Gray Triangle: ACLUTRR (M) * Gray Circle: ProofWriter (M) **Plot (b): Math Reasoning Tasks** * **X-axis:** Model Size (7B, 8B, 13B, 70B, GPT) * **Y-axis:** Task Accuracy (%) (Scale from 0 to 100, increments of 20) * **Legend (top-left):** * Blue Square: GSM8K (C) * Orange Triangle: SVAMP (C) * Green Circle: TabMWP (C) * Red Diamond: In-Domain GSM8K (C) * Purple Diamond: In-Domain MATH (C) **Plot (c): Question-Answering Tasks** * **X-axis:** Model Size (7B, 8B, 13B, 70B, GPT) * **Y-axis:** Task Accuracy (%) (Scale from 30 to 80, increments of 10) * **Legend (top-left):** * Blue Square: AmbigNQ (C) * Orange Triangle: TriviaQA (C) * Green Circle: HotpotQA (C) * Gray Square: AmbigNQ (M) * Gray Triangle: TriviaQA (M) * Gray Circle: HotpotQA (M) **Plot (d): Task Runtime vs. Complexity** * **X-axis:** Inter. Math Olympics reasoning (Year_Problem) (00_P1, 08_P6, 04_P1, 12_P5, 20_P6, 19_P6) * **Y-axis:** Task runtime (min) (Scale from 5 to 30, increments of 5) * **Legend (top-right):** * Blue Circle: Neuro-symbolic models (AlphaGeometry) * Gray Circle: RL-based CoT reasoning models * There is a right-pointing arrow at the bottom of the chart, indicating increasing complexity. ### Detailed Analysis **Plot (a): Complex Reasoning Tasks** * **Textedit (C):** Accuracy increases from approximately 85% at 7B to 98% at 70B. * **ACLUTRR (C):** Accuracy increases from approximately 58% at 7B to 70% at 70B. * **ProofWriter (C):** Accuracy increases from approximately 20% at 7B to 65% at 13B, then to 70% at 70B. * **Textedit (M):** Accuracy increases from approximately 45% at 7B to 48% at 8B, then to 55% at 13B, and then decreases to 45% at 70B. * **ACLUTRR (M):** Accuracy increases from approximately 25% at 7B to 35% at 13B, then to 45% at 70B. * **ProofWriter (M):** Accuracy increases from approximately 20% at 7B to 25% at 8B, then to 25% at 13B, and then to 30% at 70B. **Plot (b): Math Reasoning Tasks** * The data points are scattered, making it difficult to discern clear trends for each model. * **GSM8K (C):** Accuracy ranges from approximately 20% to 55%. * **SVAMP (C):** Accuracy ranges from approximately 25% to 75%. * **TabMWP (C):** Accuracy ranges from approximately 30% to 95%. * **In-Domain GSM8K (C):** Accuracy ranges from approximately 40% to 75%. * **In-Domain MATH (C):** Accuracy ranges from approximately 10% to 35%. **Plot (c): Question-Answering Tasks** * **AmbigNQ (C):** Accuracy increases from approximately 65% at 7B to 70% at 8B, then to 75% at 13B, and then to 80% at 70B. * **TriviaQA (C):** Accuracy increases from approximately 55% at 7B to 68% at 8B, then to 60% at 13B, and then to 80% at 70B. * **HotpotQA (C):** Accuracy increases from approximately 38% at 7B to 40% at 8B, then to 65% at 13B, and then to 75% at 70B. * **AmbigNQ (M):** Accuracy increases from approximately 45% at 7B to 55% at 8B, then to 58% at 13B, and then to 75% at 70B. * **TriviaQA (M):** Accuracy increases from approximately 50% at 7B to 58% at 8B, then to 58% at 13B, and then to 75% at 70B. * **HotpotQA (M):** Accuracy increases from approximately 30% at 7B to 30% at 8B, then to 40% at 13B, and then to 50% at 70B. **Plot (d): Task Runtime vs. Complexity** * **Neuro-symbolic models (AlphaGeometry):** Task runtime increases linearly from approximately 8 minutes to 15 minutes as complexity increases. * **RL-based CoT reasoning models:** Task runtime increases linearly from approximately 12 minutes to 28 minutes as complexity increases. ### Key Observations * For Complex Reasoning and Question-Answering tasks, the "C" versions of the models generally outperform the "M" versions. * In Math Reasoning tasks, the performance varies significantly across different models and model sizes. * In the Task Runtime vs. Complexity plot, RL-based CoT reasoning models consistently have higher task runtime compared to Neuro-symbolic models (AlphaGeometry). * The GPT model size is only present in the Task Accuracy plots, and not in the Task Runtime plot. ### Interpretation The data suggests that increasing model size generally improves task accuracy for Complex Reasoning and Question-Answering tasks, but the effect is less consistent for Math Reasoning tasks. The difference in performance between "C" and "M" versions of the models indicates that certain model architectures or training methods are more effective for specific tasks. The Task Runtime vs. Complexity plot highlights a trade-off between model type and computational cost, with Neuro-symbolic models demonstrating lower runtime compared to RL-based models for the same level of complexity. The arrow on the x-axis of plot (d) indicates that the problems are ordered by increasing difficulty.

## Charts: Scaling Performance Analysis & Scaling Efficiency Analysis ### Overview The image presents four charts comparing the performance of different models on various tasks as a function of model size and complexity. The first three charts focus on task accuracy versus model size for Complex Reasoning, Math Reasoning, and Question-Answering tasks. The fourth chart shows task runtime versus complexity for Neuro-symbolic and KL-based reasoning models. ### Components/Axes **Chart 1: Complex Reasoning Tasks** * **X-axis:** Model Size (TB) - Values: 7B, 8B, 13B, 70B, GPT * **Y-axis:** Task Accuracy (%) - Scale: 20 to 100, increments of 10. * **Legend:** * TextEdit (C) - Orange Squares * CLUTRR (C) - Red Circles * ProofWriter (C) - Blue Triangles * TextEdit (M) - Light Orange Squares * CLUTRR (M) - Light Red Circles * ProofWriter (M) - Light Blue Triangles **Chart 2: Math Reasoning Tasks** * **X-axis:** Model Size (TB) - Values: 7B, 8B, 13B, 70B, GPT * **Y-axis:** Task Accuracy (%) - Scale: 20 to 100, increments of 10. * **Legend:** * GSM8K (C) - Orange Squares * SVAMP (C) - Red Circles * TabMWP (C) - Blue Triangles * In-Domain GSM8K (C) - Light Orange Squares * In-Domain SVAMP (C) - Light Red Circles * In-Domain MATH (C) - Light Blue Triangles **Chart 3: Question-Answering Tasks** * **X-axis:** Model Size (TB) - Values: 7B, 8B, 13B, 70B, GPT * **Y-axis:** Task Accuracy (%) - Scale: 30 to 100, increments of 10. * **Legend:** * AmbiguityQA (C) - Orange Squares * TriviaQA (C) - Red Circles * HotpotQA (C) - Blue Triangles * AmbiguityQA (M) - Light Orange Squares * TriviaQA (M) - Light Red Circles * HotpotQA (M) - Light Blue Triangles **Chart 4: Scaling Efficiency Analysis** * **X-axis:** Complexity (Inter Math Olympics reasoning (Year Problem)) - Values: P1, 08, P6, 04, P12, P5, 20, P9, P6 * **Y-axis:** Task runtime (min) - Scale: 0 to 30, increments of 5. * **Legend:** * Neuro-symbolic models (AlphaGeometry) - Gray Circles * KL-based (T reasoning models - Blue Triangles ### Detailed Analysis or Content Details **Chart 1: Complex Reasoning Tasks** * TextEdit (C): Starts at approximately 40% at 7B, increases to around 60% at 8B, plateaus around 70% at 13B, and reaches approximately 90% at 70B. * CLUTRR (C): Starts at approximately 30% at 7B, increases to around 50% at 8B, plateaus around 60% at 13B, and reaches approximately 80% at 70B. * ProofWriter (C): Starts at approximately 20% at 7B, increases to around 40% at 8B, plateaus around 50% at 13B, and reaches approximately 70% at 70B. * TextEdit (M): Starts at approximately 30% at 7B, increases to around 50% at 8B, plateaus around 60% at 13B, and reaches approximately 80% at 70B. * CLUTRR (M): Starts at approximately 20% at 7B, increases to around 40% at 8B, plateaus around 50% at 13B, and reaches approximately 70% at 70B. * ProofWriter (M): Starts at approximately 10% at 7B, increases to around 30% at 8B, plateaus around 40% at 13B, and reaches approximately 60% at 70B. **Chart 2: Math Reasoning Tasks** * GSM8K (C): Starts at approximately 30% at 7B, increases to around 50% at 8B, plateaus around 60% at 13B, and reaches approximately 80% at 70B. * SVAMP (C): Starts at approximately 20% at 7B, increases to around 40% at 8B, plateaus around 50% at 13B, and reaches approximately 70% at 70B. * TabMWP (C): Starts at approximately 20% at 7B, increases to around 40% at 8B, plateaus around 50% at 13B, and reaches approximately 70% at 70B. * In-Domain GSM8K (C): Starts at approximately 60% at 7B, increases to around 70% at 8B, plateaus around 80% at 13B, and reaches approximately 90% at 70B. * In-Domain SVAMP (C): Starts at approximately 50% at 7B, increases to around 60% at 8B, plateaus around 70% at 13B, and reaches approximately 80% at 70B. * In-Domain MATH (C): Starts at approximately 40% at 7B, increases to around 60% at 8B, plateaus around 70% at 13B, and reaches approximately 80% at 70B. **Chart 3: Question-Answering Tasks** * AmbiguityQA (C): Starts at approximately 40% at 7B, increases to around 60% at 8B, plateaus around 70% at 13B, and reaches approximately 90% at 70B. * TriviaQA (C): Starts at approximately 30% at 7B, increases to around 50% at 8B, plateaus around 60% at 13B, and reaches approximately 80% at 70B. * HotpotQA (C): Starts at approximately 20% at 7B, increases to around 40% at 8B, plateaus around 50% at 13B, and reaches approximately 70% at 70B. * AmbiguityQA (M): Starts at approximately 30% at 7B, increases to around 50% at 8B, plateaus around 60% at 13B, and reaches approximately 80% at 70B. * TriviaQA (M): Starts at approximately 20% at 7B, increases to around 40% at 8B, plateaus around 50% at 13B, and reaches approximately 70% at 70B. * HotpotQA (M): Starts at approximately 10% at 7B, increases to around 30% at 8B, plateaus around 40% at 13B, and reaches approximately 60% at 70B. **Chart 4: Scaling Efficiency Analysis** * Neuro-symbolic models: Starts at approximately 5 minutes at P1, increases linearly to approximately 25 minutes at P9, and reaches approximately 30 minutes at P6. * KL-based models: Starts at approximately 10 minutes at P1, increases linearly to approximately 20 minutes at P9, and plateaus around 20 minutes at P6. ### Key Observations * Across all three accuracy charts, increasing model size generally leads to improved task accuracy, with diminishing returns beyond 13B. * The "In-Domain" tasks consistently outperform their corresponding "C" (presumably "Cross-Domain") counterparts. * The Neuro-symbolic models exhibit a steeper runtime increase with complexity compared to the KL-based models. * The runtime of KL-based models appears to plateau after a certain level of complexity. ### Interpretation The data suggests that scaling model size is an effective strategy for improving performance on complex reasoning, math reasoning, and question-answering tasks. However, the gains diminish as the model size increases, indicating a potential limit to the benefits of simply increasing parameters. The performance difference between in-domain and cross-domain tasks highlights the importance of training data distribution. The runtime analysis reveals a trade-off between model complexity and computational efficiency, with neuro-symbolic models being more computationally expensive than KL-based models. The plateau in runtime for KL-based models suggests that they may be more scalable for highly complex problems. The charts collectively demonstrate the ongoing research into balancing accuracy and efficiency in large language models. The use of "(C)" and "(M)" likely denotes different training methodologies or data splits, with "(M)" potentially representing a more refined or targeted training approach. The x-axis labels in the final chart are somewhat cryptic ("P1", "08", etc.), suggesting a specific benchmark or competition context (Inter Math Olympics).

## Scaling Performance Analysis: Task Accuracy vs. Model Size & Scaling Efficiency Analysis: Task Runtime vs. Complexity ### Overview The image is a composite figure containing four distinct charts, labeled (a), (b), (c), and (d). The first three charts (a, b, c) are scatter plots analyzing the relationship between **Model Size** and **Task Accuracy (%)** across different task categories. The fourth chart (d) is a line graph comparing the **Task runtime (min)** of two different model types against increasing problem complexity. The overall theme is the scaling behavior of AI models in terms of accuracy and computational efficiency. ### Components/Axes **Common Elements (Charts a, b, c):** * **X-Axis:** "Model Size" with categorical markers: `7B`, `8B`, `13B`, `70B`, `GPT`. * **Y-Axis:** "Task Accuracy (%)" with a linear scale from 0 to 100, marked at intervals of 20. * **Legends:** Each chart has a legend positioned in the top-left corner, listing specific tasks with unique color and marker shape combinations. The legend distinguishes between two variants for each task, denoted by `(C)` and `(M)`. * **Data Points:** Each data point is a marker (circle, square, triangle, etc.) representing the accuracy of a specific task variant on a specific model size. **Chart (d) Specifics:** * **Title:** "Scaling Efficiency Analysis: Task Runtime vs. Complexity" * **X-Axis:** "Inter. Math Olympiad reasoning (Year_Problem)" with categorical labels: `P01_P08`, `P06_P12`, `P04_P12`, `P12_P15`, `P20_P16`, `P19_P6`. These likely represent problem sets from different years. * **Y-Axis:** "Task runtime (min)" with a linear scale from 0 to 30, marked at intervals of 10. * **Legend:** Positioned in the top-left corner, identifying two data series: * `Neuro-symbolic models (AlphaGeometry)`: Represented by a blue line with circular markers. * `RL-based OST reasoning models`: Represented by a gray line with circular markers. ### Detailed Analysis **Chart (a): Complex Reasoning Tasks** * **Tasks & Legend:** * `TextEdit (C)`: Blue circle * `ACLUTR (C)`: Orange triangle (pointing up) * `ProofWriter (C)`: Green triangle (pointing down) * `TextEdit (M)`: Gray square * `ACLUTR (M)`: Brown diamond * `ProofWriter (M)`: Light blue circle * **Trend & Data Points (Approximate):** * **General Trend:** Accuracy for all tasks generally increases with model size. The `(C)` variants consistently outperform their `(M)` counterparts. * `TextEdit (C)`: Starts high (~90% at 7B), approaches ~100% at GPT. * `ACLUTR (C)`: Starts low (~20% at 7B), shows strong improvement to ~70% at GPT. * `ProofWriter (C)`: Starts very low (~10% at 7B), improves to ~80% at GPT. * `TextEdit (M)`: Starts around 50% at 7B, improves to ~90% at GPT. * `ACLUTR (M)`: Starts around 20% at 7B, improves to ~60% at GPT. * `ProofWriter (M)`: Starts near 0% at 7B, improves to ~40% at GPT. **Chart (b): Math Reasoning Tasks** * **Tasks & Legend:** * `GSM8K (C)`: Blue circle * `SVAMP (C)`: Orange triangle (pointing up) * `TabMWP (C)`: Green triangle (pointing down) * `In-Domain GSM8K (C)`: Gray square * `In-Domain MATH (C)`: Brown diamond * **Trend & Data Points (Approximate):** * **General Trend:** Accuracy improves with model size, but performance is more varied and generally lower than in complex reasoning tasks. `In-Domain MATH (C)` is the most challenging. * `GSM8K (C)`: Starts ~55% at 7B, improves to ~95% at GPT. * `SVAMP (C)`: Starts ~60% at 7B, improves to ~90% at GPT. * `TabMWP (C)`: Starts ~40% at 7B, improves to ~85% at GPT. * `In-Domain GSM8K (C)`: Starts ~25% at 7B, improves to ~70% at GPT. * `In-Domain MATH (C)`: Starts ~10% at 7B, improves to ~50% at GPT. **Chart (c): Question-Answering Tasks** * **Tasks & Legend:** * `AmbigNQ (C)`: Blue circle * `TriviaQA (C)`: Orange triangle (pointing up) * `HotpotQA (C)`: Green triangle (pointing down) * `AmbigNQ (M)`: Gray square * `TriviaQA (M)`: Brown diamond * `HotpotQA (M)`: Light blue circle * **Trend & Data Points (Approximate):** * **General Trend:** Strong positive correlation between model size and accuracy. `(C)` variants again outperform `(M)` variants. * `AmbigNQ (C)`: Starts ~65% at 7B, improves to ~95% at GPT. * `TriviaQA (C)`: Starts ~60% at 7B, improves to ~90% at GPT. * `HotpotQA (C)`: Starts ~55% at 7B, improves to ~85% at GPT. * `AmbigNQ (M)`: Starts ~55% at 7B, improves to ~80% at GPT. * `TriviaQA (M)`: Starts ~40% at 7B, improves to ~75% at GPT. * `HotpotQA (M)`: Starts ~30% at 7B, improves to ~60% at GPT. **Chart (d): Scaling Efficiency Analysis** * **Trend & Data Points (Approximate):** * **Neuro-symbolic models (AlphaGeometry) - Blue Line:** Shows a steep, near-linear increase in runtime with problem complexity. Starts at ~8 min for `P01_P08`, rises to ~13 min for `P19_P6`. * **RL-based OST reasoning models - Gray Line:** Shows a more gradual, slightly super-linear increase. Starts at ~12 min for `P01_P08`, rises to ~28 min for `P19_P6`. * **Key Observation:** The RL-based models have a higher initial runtime but scale worse (steeper slope) than the neuro-symbolic models as problem complexity increases. The lines cross between `P04_P12` and `P12_P15`, after which the neuro-symbolic models become more efficient. ### Key Observations 1. **Consistent Scaling:** Across all task types (Complex Reasoning, Math, QA), increasing model size from 7B to GPT leads to significant accuracy gains. 2. **Task Difficulty Hierarchy:** Within each chart, certain tasks are consistently harder. For example, `ProofWriter (M)` in (a), `In-Domain MATH (C)` in (b), and `HotpotQA (M)` in (c) show the lowest accuracies. 3. **Variant Performance Gap:** The `(C)` variant of each task consistently achieves higher accuracy than the `(M)` variant across all model sizes, suggesting a fundamental difference in difficulty or evaluation setup. 4. **Efficiency Trade-off (Chart d):** There is a clear trade-off between model architecture and scaling efficiency. While RL-based models may have higher base runtime, neuro-symbolic models (AlphaGeometry) demonstrate superior scaling characteristics for this specific reasoning domain, becoming more efficient on more complex problems. ### Interpretation The data presents a multi-faceted view of AI scaling. Charts (a-c) demonstrate the **"scaling law"** phenomenon: larger models are more capable, as measured by task accuracy. However, the gains are not uniform; they depend heavily on the specific task and its variant. The persistent gap between `(C)` and `(M)` variants suggests that model improvements alone may not close performance gaps on inherently more difficult problem formulations. Chart (d) shifts the focus from capability (accuracy) to **efficiency (runtime)**. It reveals that scaling behavior is not monolithic. Different architectural paradigms (neuro-symbolic vs. reinforcement learning-based) exhibit fundamentally different computational cost profiles as problem complexity grows. The crossover point indicates that the "best" model depends on the operational context—specifically, the expected complexity of the problems to be solved. For simpler problems, one architecture may be preferable, while for highly complex Olympiad-level reasoning, the other becomes more efficient. In summary, the image argues that evaluating AI systems requires looking beyond a single metric. True understanding comes from analyzing **capability scaling** (accuracy vs. size) alongside **efficiency scaling** (runtime vs. complexity), and doing so across a diverse set of tasks that probe different facets of intelligence.

## Chart/Diagram Type: Scaling Performance Analysis and Efficiency Analysis ### Overview The image contains four graphs and a line chart analyzing the scaling performance of AI models across reasoning tasks and computational efficiency. The left three graphs (a, b, c) compare task accuracy vs. model size for complex reasoning, math reasoning, and question-answering tasks. The right graph (d) compares task runtime vs. complexity for neuro-symbolic and RL-based CoT models. ### Components/Axes #### Graphs (a), (b), (c): Task Accuracy vs. Model Size - **X-axis**: Model Size (7B, 8B, 13B, 70B, GPT) - **Y-axis**: Task Accuracy (%) - **Legends**: - **Models**: Textedit, CLUTR, ProofWriter, GSM8K, SVAMP, TabMWP, In-Domain GSM8K, In-Domain MATH, AmbigNQ, TriviaQA, HotpotQA - **Markers/Colors**: - Squares (□): Textedit, GSM8K, AmbigNQ - Triangles (△): CLUTR, SVAMP, TriviaQA - Circles (●): ProofWriter, TabMWP, HotpotQA - Colors: Blue, Orange, Green, Red, Purple, Gray - **Subcategories**: - (C): Chain-of-Thought (CoT) reasoning - (M): MCT (Multi-step CoT) reasoning #### Graph (d): Task Runtime vs. Complexity - **X-axis**: Problem Complexity (P1, P8, P6, P4, P12, P5, P20, P6, P19, P6) - **Y-axis**: Task Runtime (minutes) - **Lines**: - Blue: Neuro-symbolic models (AlphaGeometry) - Gray: RL-based CoT reasoning models ### Detailed Analysis #### Graphs (a), (b), (c): Task Accuracy vs. Model Size - **Trends**: - **Complex Reasoning (a)**: - Textedit(C) (blue square): 50% (7B) → 90% (GPT) - CLUTR(C) (orange triangle): 60% (7B) → 85% (GPT) - ProofWriter(C) (green circle): 70% (7B) → 95% (GPT) - **Math Reasoning (b)**: - GSM8K(C) (blue square): 40% (7B) → 80% (GPT) - In-Domain GSM8K(C) (red triangle): 50% (7B) → 85% (GPT) - **Question-Answering (c)**: - AmbigNQ(C) (blue square): 30% (7B) → 70% (GPT) - TriviaQA(C) (orange triangle): 40% (7B) → 80% (GPT) #### Graph (d): Task Runtime vs. Complexity - **Trends**: - Neuro-symbolic models (blue line): Runtime increases linearly from ~5 min (P1) to ~30 min (P19). - RL-based CoT models (gray line): Runtime increases more steeply, from ~10 min (P1) to ~40 min (P19). ### Key Observations 1. **Model Size vs. Accuracy**: - Larger models (e.g., GPT) consistently outperform smaller models (7B, 8B) across all tasks. - CoT models (C) generally achieve higher accuracy than MCT models (M) for the same task. 2. **Runtime vs. Complexity**: - RL-based CoT models require significantly more time than neuro-symbolic models for complex problems. - Runtime scales non-linearly with problem complexity for RL-based models. ### Interpretation - **Scaling Performance**: - Increasing model size improves task accuracy, particularly for complex reasoning (e.g., GPT achieves 95% accuracy in ProofWriter(C) vs. 70% for 7B models). - CoT approaches (C) are more effective than MCT (M) but may require larger models to reach peak performance. - **Efficiency Trade-offs**: - Neuro-symbolic models (AlphaGeometry) are computationally efficient but less accurate on highly complex tasks. - RL-based CoT models achieve higher accuracy at the cost of increased runtime, suggesting a trade-off between precision and efficiency. - **Outliers**: - In graph (a), ProofWriter(C) (green circle) shows the steepest accuracy improvement with model size, suggesting it is optimized for scaling. - In graph (d), the divergence between neuro-symbolic and RL-based models widens for problems P12 and P19, indicating RL models struggle more with extreme complexity. This analysis highlights the importance of model architecture and scaling strategies in balancing accuracy and computational efficiency for AI reasoning systems.