## Scatter Plot: MathVision Pass@1 vs. Activated Parameters ### Overview The image is a scatter plot comparing the performance of various language models on the MathVision Pass@1 benchmark against the number of activated parameters (in billions). The plot displays data points for different models, each labeled with its name, and uses different colors to distinguish between model families. Trend lines connect related models. ### Components/Axes * **X-axis:** Activated Parameters (B), with a logarithmic scale. Axis markers are present at approximately 3, 10, 30, and 70. * **Y-axis:** MathVision Pass@1, with a linear scale. Axis markers are present at 20, 35, 50, and 65. * **Data Points:** Each data point represents a specific language model. The models are labeled with their names (e.g., "Kimi-VL-A3B-Thinking-2506", "Gemma-3-4B-IT", "Qwen-2.5-VL-3B"). * **Trend Lines:** Dashed lines connect related models, showing the trend in performance as the number of activated parameters increases. There are two trend lines: one for the Gemma models (purple) and one for the Qwen models (gray). * **Legend:** There is no explicit legend, but the colors of the data points implicitly represent different model families. ### Detailed Analysis * **Kimi-VL-A3B-Thinking-2506 (Dark Blue Star):** Located at approximately (3, 60). * **Kimi-VL-A3B-Thinking (Light Blue Star):** Located at approximately (3, 37). * **Gemma-3-4B-IT (Purple Dot):** Located at approximately (4, 25). * **Gemma-3-12B-IT (Purple Dot):** Located at approximately (12, 33). * **Gemma-3-27B-IT (Purple Dot):** Located at approximately (28, 36). * **Trend:** The Gemma models show an upward trend, with performance increasing as the number of activated parameters increases. * **Qwen-2.5-VL-3B (Gray Dot):** Located at approximately (5, 22). * **Qwen-2.5-VC-7B (Gray Dot):** Located at approximately (11, 27). * **Qwen-2.5-VL-32B (Gray Dot):** Located at approximately (35, 38). * **Qwen-2.5-VL-72B (Gray Dot):** Located at approximately (75, 38). * **Trend:** The Qwen models show an upward trend initially, but performance plateaus after 32B parameters. * **DeepSeek-VL2-A4.5B (Blue Dot):** Located at approximately (6, 18). * **Llama-3.2-11B-Inst. (Red Dot):** Located at approximately (12, 15). * **QVQ-72B-Preview (Green X):** Located at approximately (75, 33). * **QVQ-Max-Preview (Green X):** Located at approximately (75, 48). ### Key Observations * The Kimi models, represented by stars, outperform other models with similar numbers of activated parameters. * The Gemma models show a consistent increase in performance with increasing parameters. * The Qwen models plateau in performance after a certain number of parameters. * The QVQ models show a significant jump in performance compared to the Qwen models, despite having similar numbers of parameters. * The DeepSeek and Llama models have relatively low MathVision Pass@1 scores compared to other models. ### Interpretation The scatter plot illustrates the relationship between model size (activated parameters) and performance on the MathVision Pass@1 benchmark. The data suggests that increasing model size generally leads to better performance, but there are diminishing returns for some model families (e.g., Qwen). The Kimi and QVQ models appear to be more efficient, achieving higher performance with fewer parameters compared to other models. The plot highlights the importance of model architecture and training techniques in addition to model size. The plateauing of the Qwen models suggests that simply increasing the number of parameters may not always lead to significant improvements in performance.

## Scatter Plot: Model Performance vs. Parameters ### Overview This image presents a scatter plot comparing the performance of various models on a MathVision Pass@1 task against the number of activated parameters they utilize. Each model is represented by a data point, and a trend line is fitted to a subset of the models. The plot aims to illustrate the relationship between model size (parameter count) and mathematical reasoning ability. ### Components/Axes * **X-axis:** Activated Parameters (B) - Scale ranges from approximately 3 to 75 Billion parameters. * **Y-axis:** MathVision Pass@1 - Scale ranges from approximately 25 to 65. * **Data Points:** Represent individual models. * **Trend Line:** A dashed red line attempting to show the correlation between parameters and performance for a subset of models. * **Legend:** Implicitly defined by the labels next to each data point. ### Detailed Analysis The following data points are visible, with approximate values read from the plot: * **Kimi-VL-A3B-Thinking-2506 (Purple Star):** Approximately (3, 35.5). * **Kimi-VL-A3B-Thinking (Purple Star):** Approximately (3, 33). * **DeepSeek-VL2-44.5B (Dark Blue Circle):** Approximately (7, 27). * **Llama-3.2-11B-Inst. (Dark Blue Circle):** Approximately (11, 27.5). * **Gemma-3-4B-IT (Orange Circle):** Approximately (11, 30). * **Owen-2.5-VL-3B (Orange Circle):** Approximately (11, 29). * **Gemma-3-12B-IT (Orange Circle):** Approximately (33, 33). * **Qwen-2.5-VL-32B (Red Circle):** Approximately (33, 35). * **Qwen-2.5-VL-72B (Red Circle):** Approximately (73, 36). * **QVQ-72B-Preview (Red Circle):** Approximately (73, 52). * **QVQ-Max-Preview (Red Circle):** Approximately (73, 54). * **Owen-2.5-VL-7B (Orange Circle):** Approximately (11, 31). The trend line (dashed red) connects the following points: Gemma-3-4B-IT, Gemma-3-12B-IT, Qwen-2.5-VL-32B, Qwen-2.5-VL-72B. The line shows a generally upward trend, indicating that as the number of activated parameters increases, the MathVision Pass@1 score tends to increase as well. ### Key Observations * **Outliers:** Kimi-VL-A3B-Thinking-2506 and Kimi-VL-A3B-Thinking show relatively high performance with a small number of parameters compared to other models. * **Trend:** The trend line suggests a positive correlation between model size and performance, but the correlation is not strong, as evidenced by the scatter of points around the line. * **Clustering:** Models with similar parameter counts tend to cluster together, particularly in the 10-12B range. * **QVQ Models:** The QVQ models (QVQ-72B-Preview and QVQ-Max-Preview) demonstrate the highest performance, but also require the largest number of parameters. ### Interpretation The data suggests that increasing the number of activated parameters generally improves performance on the MathVision Pass@1 task. However, the relationship is not linear, and there is significant variation among models with similar parameter counts. The Kimi models stand out as achieving high performance with relatively few parameters, suggesting a potentially more efficient architecture or training methodology. The QVQ models represent the state-of-the-art in terms of performance, but at the cost of significantly increased computational resources. The trend line provides a rough estimate of the expected performance gain for a given increase in parameters, but it should be interpreted with caution due to the scatter in the data. The plot highlights the trade-off between model size, performance, and computational cost in the context of mathematical reasoning.

## Scatter Plot: AI Model Performance vs. Model Size ### Overview This image is a scatter plot comparing the performance of various multimodal AI models on a mathematical vision benchmark against their computational scale (activated parameters). The plot reveals a general trend where larger models tend to achieve higher scores, but with significant outliers demonstrating high efficiency. ### Components/Axes * **X-Axis:** Labeled **"Activated Parameters (B)"**. The scale is logarithmic, with major tick marks at **3, 10, 30, and 70** billion parameters. * **Y-Axis:** Labeled **"MathVision Pass@1"**. The scale is linear, ranging from **20 to 65**, with major grid lines at intervals of 15 (20, 35, 50, 65). * **Data Series & Legend:** The plot contains multiple data series, each represented by a distinct color and marker shape. The legend is embedded directly as labels next to each data point. * **Dark Blue Star:** `Kimi-VL-A3B-Thinking-2506` * **Light Blue Star:** `Kimi-VL-A3B-Thinking` * **Purple Circles (connected by dashed line):** `Gemma-3-4B-IT`, `Gemma-3-12B-IT`, `Gemma-3-27B-IT` * **Gray Circles (connected by dashed line):** `Qwen-2.5-VL-3B`, `Qwen-2.5-VL-7B`, `Qwen-2.5-VL-32B`, `Qwen-2.5-VL-72B` * **Blue Circle:** `DeepSeek-VL2-A4.5B` * **Red Circle:** `Llama-3.2-11B-Inst.` * **Green Crosses:** `QVQ-72B-Preview`, `QVQ-Max-Preview` ### Detailed Analysis **Data Points (Approximate Coordinates: Activated Parameters (B), MathVision Pass@1):** 1. **Kimi-VL-A3B-Thinking-2506 (Dark Blue Star):** Positioned at the top-left. Coordinates: **(~3B, ~60)**. This is the highest-performing model on the chart. 2. **Kimi-VL-A3B-Thinking (Light Blue Star):** Positioned below the first star. Coordinates: **(~3B, ~37)**. 3. **Gemma-3 Series (Purple Circles, upward trend):** * `Gemma-3-4B-IT`: **(~4B, ~25)** * `Gemma-3-12B-IT`: **(~12B, ~32)** * `Gemma-3-27B-IT`: **(~27B, ~35)** * *Trend:* Performance increases with model size, but the rate of improvement slows. 4. **Qwen-2.5-VL Series (Gray Circles, upward then plateauing trend):** * `Qwen-2.5-VL-3B`: **(~3B, ~21)** * `Qwen-2.5-VL-7B`: **(~7B, ~25)** * `Qwen-2.5-VL-32B`: **(~32B, ~38)** * `Qwen-2.5-VL-72B`: **(~72B, ~38)** * *Trend:* Strong improvement from 3B to 32B, then a plateau between 32B and 72B. 5. **DeepSeek-VL2-A4.5B (Blue Circle):** Coordinates: **(~4.5B, ~18)**. Positioned below the Gemma-3-4B-IT point. 6. **Llama-3.2-11B-Inst. (Red Circle):** Coordinates: **(~11B, ~15)**. This is the lowest-performing model on the chart for its size. 7. **QVQ Series (Green Crosses, high-parameter region):** * `QVQ-72B-Preview`: **(~72B, ~36)**. Positioned slightly below the Qwen-2.5-VL-72B point. * `QVQ-Max-Preview`: **(~120B?, ~49)**. The rightmost point, with an estimated parameter count beyond the 70B tick. ### Key Observations 1. **Efficiency Outliers:** The `Kimi-VL-A3B-Thinking-2506` model is a dramatic outlier, achieving the highest score (~60) with one of the smallest parameter counts (~3B). This indicates exceptional parameter efficiency for this specific task. 2. **Performance Plateau:** The `Qwen-2.5-VL` series shows a clear performance plateau, where increasing parameters from 32B to 72B yields no improvement in the MathVision Pass@1 score. 3. **Size-Performance Disconnect:** Larger models do not guarantee better performance. `Llama-3.2-11B-Inst.` (~11B) underperforms both smaller models (e.g., `Qwen-2.5-VL-3B`) and similarly sized models (e.g., `Gemma-3-12B-IT`). 4. **General Trend:** Excluding the major outliers, there is a loose positive correlation between activated parameters and benchmark score, as seen in the Gemma-3 and the initial segment of the Qwen-2.5-VL series. ### Interpretation This chart visualizes the trade-off and variance in **efficiency versus scale** for multimodal AI models on a mathematical reasoning task. * **The "Kimi" models** suggest that architectural innovations or training techniques (implied by the "-Thinking" suffix) can lead to breakthroughs in efficiency, achieving state-of-the-art results with a fraction of the parameters used by competitors. * The **plateau in the Qwen series** indicates diminishing returns for simply scaling a particular model architecture on this benchmark. It suggests that beyond a certain point (~32B for this model family), other factors like data quality, training methodology, or architectural limits become the primary bottleneck. * The **underperformance of Llama-3.2-11B-Inst.** highlights that not all models of a certain size are created equal; their training data, objective alignment, and architecture critically determine their capability on specialized tasks like visual math. * The **QVQ-Max-Preview** point shows that very large scale can still push performance higher, but it requires a massive increase in parameters to achieve a score that is still below the much smaller "Kimi" model. **In summary, the data argues that for specialized reasoning tasks, intelligent model design and training can be far more impactful than brute-force scaling. The chart serves as a benchmark for evaluating not just raw performance, but the efficiency and effectiveness of different AI development approaches.**

## Scatter Plot: Model Performance vs. Parameter Count ### Overview The image is a scatter plot comparing the performance of various AI models (MathVision Pass@1) against their activated parameter counts (in billions). The plot includes labeled data points, connecting lines for model families, and a legend for color coding. Key trends show a general relationship between parameter count and performance, with notable outliers. --- ### Components/Axes - **X-axis**: "Activated Parameters (B)" with ticks at 3, 10, 30, 70, and 80 billion parameters. - **Y-axis**: "MathVision Pass@1" with ticks from 15 to 65. - **Legend**: Located on the right, mapping colors to model families: - **Blue**: Kimi - **Purple**: Gemma - **Gray**: Owen - **Green**: QVQ - **Red**: DeepSeek and Llama - **Data Points**: Labeled with model names, parameter counts, and versions (e.g., "Kimi-VL-A3B-Thinking-2506"). - **Lines**: Dashed lines connect data points for model families (e.g., Owen-2.5-VL-3B → Owen-2.5-VL-7B → Owen-2.5-VL-32B). --- ### Detailed Analysis #### Data Points and Trends 1. **Kimi Models**: - **Kimi-VL-A3B-Thinking-2506**: 3B parameters, 55 Pass@1 (top-left). - **Kimi-VL-A3B-Thinking**: 3B parameters, 35 Pass@1 (lower-left). - **Trend**: Horizontal line at 3B parameters, showing no improvement in Pass@1 with parameter count. 2. **Gemma Models**: - **Gemma-3-4B-IT**: 3B parameters, 25 Pass@1. - **Gemma-3-12B-IT**: 10B parameters, 35 Pass@1. - **Gemma-3-27B-IT**: 30B parameters, 35 Pass@1. - **Trend**: Dashed purple line slopes upward, indicating improved performance with parameter count up to 30B. 3. **Owen Models**: - **Owen-2.5-VL-3B**: 3B parameters, 20 Pass@1. - **Owen-2.5-VL-7B**: 10B parameters, 25 Pass@1. - **Owen-2.5-VL-32B**: 30B parameters, 35 Pass@1. - **Owen-2.5-VL-72B**: 70B parameters, 35 Pass@1. - **Trend**: Dashed gray line slopes upward, then plateaus at 35 Pass@1 despite increased parameters. 4. **QVQ Models**: - **QVQ-72B-Preview**: 70B parameters, 35 Pass@1. - **QVQ-Max-Preview**: 80B parameters, 48 Pass@1. - **Trend**: Dashed green line slopes upward, showing significant improvement at 80B parameters. 5. **DeepSeek and Llama**: - **DeepSeek-VL2-A4.5B**: 5B parameters, 18 Pass@1. - **Llama-3.2-11B-Inst.**: 11B parameters, 15 Pass@1. - **Trend**: No connecting lines; isolated points with low Pass@1 scores. --- ### Key Observations 1. **Parameter Efficiency**: - Kimi models achieve high Pass@1 (55) with minimal parameters (3B), suggesting superior efficiency. - QVQ models require 80B parameters to reach 48 Pass@1, indicating lower efficiency. 2. **Performance Plateaus**: - Owen and Gemma models plateau at 35 Pass@1 despite increasing parameters beyond 30B, suggesting diminishing returns. 3. **Outliers**: - **QVQ-Max-Preview**: Highest Pass@1 (48) but requires 80B parameters, making it an outlier in efficiency. - **Kimi-VL-A3B-Thinking-2506**: Highest Pass@1 (55) with only 3B parameters, another efficiency outlier. 4. **Low-Performance Models**: - DeepSeek and Llama models have the lowest Pass@1 scores (15–18) despite moderate parameter counts (5–11B). --- ### Interpretation The data suggests that **parameter count alone does not guarantee performance**. Kimi models demonstrate exceptional efficiency, achieving high Pass@1 scores with minimal parameters. In contrast, QVQ models require significantly more parameters for comparable or lower performance, highlighting potential over-parameterization. The Owen and Gemma models show a trend where increasing parameters improves performance up to a point (30B), after which gains plateau. This implies that architectural innovations (e.g., Kimi’s design) may be more critical than raw parameter count. The low performance of DeepSeek and Llama models suggests limitations in their training or optimization strategies. Overall, the plot underscores the importance of balancing parameter efficiency with model architecture for optimal results.