## Multi-Line Chart: Accuracy of Four AI Models Across Mathematical Topics ### Overview This image is a multi-line chart comparing the performance (accuracy) of four different large language models (LLMs) across a wide range of mathematical problem categories. The chart visualizes how each model's accuracy fluctuates significantly depending on the specific mathematical topic. ### Components/Axes * **Chart Type:** Multi-line chart with markers. * **Y-Axis:** * **Label:** "Accuracy" * **Scale:** Linear, from 0 to 100. * **Major Ticks:** 0, 20, 40, 60, 80, 100. * **X-Axis:** * **Label:** Not explicitly labeled, but contains a dense list of mathematical topics. * **Content:** A series of 52 distinct mathematical categories, listed from left to right. The text is rotated approximately 45 degrees for readability. * **Language:** All x-axis labels are in English. * **Legend:** * **Position:** Top-center, above the plot area. * **Content:** Four entries, each associating a color and marker style with a model name. 1. **Blue line with circle markers:** Yi-6B 2. **Orange line with circle markers:** ChatGLM3-6B 3. **Green line with circle markers:** LLaMA2-7B 4. **Red line with circle markers:** DeepSeekMath-7B ### Detailed Analysis The chart shows high variability in model performance. Below is an analysis grouped by general mathematical domain, listing approximate accuracy values (to the nearest 5%) for key points. Values are approximate due to visual estimation from the chart. **1. Arithmetic & Basic Operations (Leftmost section):** * **Trend:** Models show moderate to high accuracy, with significant divergence. * **Data Points (Approx.):** * *Add & subtract:* Yi-6B ~70%, ChatGLM3-6B ~65%, LLaMA2-7B ~45%, DeepSeekMath-7B ~70%. * *Arithmetic sequences:* All models dip, with LLaMA2-7B lowest (~20%). * *Consumer math:* Yi-6B peaks (~75%), others are lower (30-50%). **2. Algebra & Equations (Left-center section):** * **Trend:** Extreme volatility. Some models achieve near-perfect scores on specific topics while failing on others. * **Data Points (Approx.):** * *Linear equations:* Yi-6B ~70%, ChatGLM3-6B ~55%, LLaMA2-7B ~45%, DeepSeekMath-7B ~70%. * *Nonlinear functions:* A notable peak for DeepSeekMath-7B (~90%) and Yi-6B (~85%). * *Probability of compound events:* A major low point for all models. Yi-6B ~35%, ChatGLM3-6B ~10%, LLaMA2-7B ~15%, DeepSeekMath-7B ~15%. **3. Geometry & Measurement (Center section):** * **Trend:** Mixed performance. LLaMA2-7B consistently underperforms in this domain. * **Data Points (Approx.):** * *Perimeter & area:* DeepSeekMath-7B ~90%, Yi-6B ~60%, ChatGLM3-6B ~65%, LLaMA2-7B ~40%. * *Pythagorean theorem:* DeepSeekMath-7B ~85%, others between 40-65%. * *Surface area & volume:* All models show a dip, with LLaMA2-7B near 0%. **4. Advanced & Applied Topics (Right-center section):** * **Trend:** Continued high variance. Some advanced topics see strong performance from specialized models. * **Data Points (Approx.):** * *Two-variable statistics:* Yi-6B ~80%, DeepSeekMath-7B ~65%, others lower. * *Systems of equations:* DeepSeekMath-7B ~80%, Yi-6B ~70%, ChatGLM3-6B ~65%, LLaMA2-7B ~45%. * *Independent & dependent events:* A catastrophic drop for ChatGLM3-6B to ~0%. Others range from 20-75%. **5. Calculus & Functions (Rightmost section):** * **Trend:** DeepSeekMath-7B and Yi-6B show strong, leading performance. * **Data Points (Approx.):** * *Transformations:* DeepSeekMath-7B ~90%, Yi-6B ~80%, ChatGLM3-6B ~60%, LLaMA2-7B ~25%. * *Variable exprs (final point):* DeepSeekMath-7B ~55%, Yi-6B ~60%, ChatGLM3-6B ~55%, LLaMA2-7B ~35%. ### Key Observations 1. **Model Specialization:** DeepSeekMath-7B (red) frequently achieves the highest peaks, especially in geometry, algebra, and calculus topics, suggesting strong mathematical specialization. Yi-6B (blue) is a consistent high performer across many domains. 2. **General Weakness:** LLaMA2-7B (green) is consistently the lowest or among the lowest performers across nearly all categories, rarely exceeding 50% accuracy. 3. **Topic-Specific Failures:** All models exhibit severe performance drops on specific topics. Notable universal low points include "Probability of compound events" and "Surface area & volume." ChatGLM3-6B has an extreme outlier near 0% on "Independent & dependent events." 4. **High Volatility:** No model demonstrates smooth, consistent performance. Accuracy is highly dependent on the specific mathematical concept being tested. ### Interpretation This chart demonstrates that mathematical reasoning in LLMs is not a monolithic capability but a highly fragmented one. Performance is exquisitely sensitive to the specific sub-domain of mathematics. * **What the data suggests:** The models likely have uneven coverage in their training data or differing architectural biases for handling symbolic vs. numerical reasoning. The near-perfect scores on some topics (e.g., DeepSeekMath-7B on "Perimeter & area") contrasted with near-zero scores on others indicate brittle knowledge rather than robust, generalized mathematical understanding. * **How elements relate:** The x-axis represents a spectrum of mathematical complexity and abstraction. The jagged, non-parallel lines show that model capabilities are not simply "better" or "worse" in a linear fashion; they are incomparable across different topics. A model strong in algebra may be weak in geometry. * **Notable anomalies:** The catastrophic failure of ChatGLM3-6B on "Independent & dependent events" is a critical outlier. It suggests a potential fundamental gap in its training or reasoning approach for probabilistic concepts involving dependency, which is a core statistical idea. The universal struggle with "Probability of compound events" highlights a common challenge area for current LLMs in handling layered probabilistic logic. **Conclusion for a Technical Document:** The chart provides compelling evidence that evaluating LLMs on "math" as a single category is insufficient. A granular, topic-by-topic analysis is required to understand a model's true capabilities and limitations. For applications requiring mathematical reasoning, model selection must be tightly coupled to the specific mathematical domain of the task.