## Scatter Plot: Model Performance vs. Average Tokens (AIME-25) ### Overview The chart compares the accuracy of AI models against their average token usage during inference. A green shaded region labeled "Higher Efficiency" (with an arrow pointing left) indicates the ideal trade-off zone where models achieve higher accuracy with fewer tokens. Data points represent specific models, annotated with their accuracy percentages and token counts. --- ### Components/Axes - **X-axis**: "Average Tokens" (logarithmic scale, 1000 to 6000 tokens) - **Y-axis**: "Accuracy (%)" (linear scale, 30% to 80%) - **Legend**: Green shaded area labeled "Higher Efficiency" (top-left quadrant) - **Data Points**: Labeled with model names, accuracy, and contextual notes (e.g., "no extended thinking") --- ### Detailed Analysis 1. **Claude-4.1-Opus-2508** - Accuracy: 37.55% - Tokens: 1000 - Note: "w/o extended thinking" - Position: Bottom-left corner (lowest accuracy, fewest tokens) 2. **DeepSeek-V3.1-Terminus** - Accuracy: 55.21% - Tokens: 3000 - Note: "NonThinking" - Position: Mid-left quadrant 3. **GPT-5 (no tools/without thinking)** - Accuracy: 59.43% - Tokens: 3000 - Position: Mid-left quadrant, slightly higher accuracy than DeepSeek 4. **Ling-1T** - Accuracy: 70.42% - Tokens: 4000 - Position: Mid-right quadrant (high accuracy, moderate tokens) 5. **Gemini 2.5 Pro (thinking_budget=128)** - Accuracy: 70.10% - Tokens: 5500 - Position: Far-right quadrant (highest token usage, near-peak accuracy) 6. **Kimi-K2-Instruct-0905** - Accuracy: 50.16% - Tokens: 5000 - Position: Mid-right quadrant (lower accuracy despite high token usage) --- ### Key Observations - **Efficiency Trade-off**: Models with fewer tokens (e.g., Claude-4.1) have lower accuracy, while those with more tokens (e.g., Gemini 2.5 Pro) achieve higher accuracy. - **Optimal Efficiency**: Ling-1T and Gemini 2.5 Pro occupy the "Higher Efficiency" zone, balancing accuracy and token usage. - **Anomaly**: Kimi-K2-Instruct-0905 underperforms relative to its token count (5000 tokens but only 50.16% accuracy). - **Accuracy Ceiling**: No model exceeds 70.42% accuracy, suggesting a practical limit for this task. --- ### Interpretation The chart demonstrates a clear inverse relationship between token usage and efficiency: models requiring fewer tokens (e.g., Claude-4.1) sacrifice accuracy, while those prioritizing accuracy (e.g., Gemini 2.5 Pro) consume more tokens. The "Higher Efficiency" zone highlights Ling-1T and Gemini 2.5 Pro as leaders, achieving >70% accuracy with ≤5500 tokens. Kimi-K2-Instruct-0905’s poor performance relative to its token count suggests suboptimal resource allocation or model design. The data implies that efficiency gains are possible by optimizing token usage without sacrificing accuracy, as seen in Ling-1T’s performance.