## Line Chart: Model Performance Scores across Model Numbers ### Overview This image is a multi-series line chart displaying the performance scores (in percentages) of various models across different benchmarks or datasets. The x-axis represents a sequential "Model Number," while the y-axis represents the "Score (%)." Instead of a traditional legend box, the data series are labeled directly on the chart area, typically near the beginning or end of their respective lines. ### Components/Axes * **Y-Axis (Vertical):** * **Label:** "Score (%)" * **Scale:** Ranges from 30 to roughly 95. * **Markers:** Major tick marks and corresponding horizontal dashed light-gray gridlines are placed at 40, 50, 60, 70, 80, and 90. * **X-Axis (Horizontal):** * **Label:** "Model Number" * **Scale:** Ranges from 1 to 22. * **Markers:** Major tick marks and corresponding vertical dashed light-gray gridlines are placed at every integer from 1 to 22. * **Legend/Labels:** There is no separate legend. Labels are color-coded to match their respective lines and are placed adjacent to the data points. ### Detailed Analysis The data series can be categorized into three distinct visual patterns: short-span early models, highly volatile mid-span models, and long-span steady scaling models. #### Group 1: Short-Span Early Models (x=3 to x=5) These lines only contain two data points and represent benchmarks evaluated only on early model numbers. All show an upward trend. * **AI2D (Purple line, square markers):** * *Position:* Top-left. * *Trend:* Slopes upward. * *Data Points:* (x=3, y~89.5), (x=5, y~94.5) * **DocVQA (Green line, upward triangle markers):** * *Position:* Top-left, just below AI2D. * *Trend:* Slopes upward. * *Data Points:* (x=3, y~87), (x=5, y~92.5) * **ChartQA (Red line, diamond markers):** * *Position:* Top-left, below DocVQA. * *Trend:* Slopes upward steeply. * *Data Points:* (x=3, y~78), (x=5, y~86) * **EgoSchema (Dark Blue line, circle markers):** * *Position:* Mid-left. * *Trend:* Slopes upward. * *Data Points:* (x=3, y~64), (x=5, y~72) * **ActivityNet (Orange line, square markers with cross inside):** * *Position:* Mid-left. * *Trend:* Slopes upward slightly. * *Data Points:* (x=3, y~59.5), (x=5, y~62) #### Group 2: Volatile Mid-Span Models (x=3/4 to x=13/21) These lines exhibit significant fluctuations, notably sharing a sharp, distinct drop at Model Number 10. * **CharXiv-D (Pink line, asterisk/star markers):** * *Position:* Upper-middle. Label is at x~13. * *Trend:* Starts high, dips slightly, rises, experiences a sharp drop at x=10, recovers immediately, and plateaus high. * *Data Points:* (x=4, y~76.5), (x=5, y~85.5), (x=8, y~89), (x=10, y~74), (x=11, y~88.5), (x=12, y~88), (x=13, y~90) * **MMMU (Brown line, pentagon markers):** * *Position:* Spans from mid-left to top-right. Label is at x~21. * *Trend:* Fluctuates early, drops sharply at x=10, then climbs steadily to merge with VideoMMMU at the end. * *Data Points:* (x=3, y~63), (x=4, y~59.5), (x=8, y~78), (x=10, y~55.5), (x=11, y~72.5), (x=12, y~74.5), (x=13, y~75), (x=15, y~81.5), (x=16, y~83), (x=21, y~84.5) * **CharXiv-R (Gray line, 'x' markers):** * *Position:* Spans from bottom-left to upper-right. Label is at x~21. * *Trend:* Starts very low, spikes up, declines steadily, drops sharply at x=10, recovers to a plateau, then climbs steeply. * *Data Points:* (x=4, y~37), (x=5, y~59), (x=8, y~55), (x=10, y~40.5), (x=11, y~56.5), (x=12, y~56.5), (x=13, y~55.5), (x=15, y~72), (x=16, y~78.5), (x=21, y~81) #### Group 3: Long-Span Steady Models (x=5 to x=21) These lines show a consistent, near-linear upward trajectory with very few data points spread across a wide range. * **VideoMMMU (Olive Green line, plus '+' markers):** * *Position:* Spans mid-left to top-right. Label is at x~21. * *Trend:* Steady, smooth upward slope. * *Data Points:* (x=5, y~61.5), (x=11, y~73), (x=15, y~81.5), (x=16, y~83.5), (x=21, y~84.5) * **MMMU Pro (Teal line, circle markers):** * *Position:* Spans mid-left to mid-right. Label is at x~21. * *Trend:* Steady, smooth upward slope. * *Data Points:* (x=5, y~60), (x=16, y~76.5), (x=21, y~78.5) * **ERQA (Light Blue line, downward triangle markers):** * *Position:* Spans bottom-left to mid-right. Label is at x~21. * *Trend:* Steady, smooth upward slope. * *Data Points:* (x=5, y~35.5), (x=16, y~64), (x=21, y~65.5) ### Key Observations 1. **The "Model 10" Anomaly:** There is a severe, synchronized drop in performance at Model Number 10 for three specific benchmarks: CharXiv-D, MMMU, and CharXiv-R. 2. **General Upward Trend:** Despite the volatility in the middle section, the overarching trend for every single benchmark is positive; performance at the highest recorded model number is always greater than at the lowest recorded model number. 3. **Data Sparsity:** The chart mixes high-frequency testing (e.g., models 10, 11, 12, 13 for CharXiv and MMMU) with very low-frequency testing (e.g., ERQA and MMMU Pro only have data points at 5, 16, and 21). 4. **Convergence:** By Model 21, VideoMMMU and MMMU converge at nearly the exact same score (~84.5%). ### Interpretation This chart likely illustrates the scaling laws or iterative improvements of a specific family of AI models (e.g., a series of Large Language Models or Multimodal Models released sequentially or scaled by parameter count, represented by "Model Number"). The general upward trend demonstrates that as the "Model Number" increases, the model becomes more capable across a wide variety of tasks (document reading, chart analysis, video understanding, etc.). The most critical investigative takeaway is the anomaly at **Model 10**. Because CharXiv-D, MMMU, and CharXiv-R all crash simultaneously at this exact point, it strongly implies that Model 10 suffered from a specific architectural flaw, a bug during training, or was a specialized checkpoint that catastrophically forgot certain reasoning capabilities while perhaps optimizing for something else. The immediate recovery at Model 11 suggests the developers identified and fixed this issue. Furthermore, the grouping of the data suggests different testing regimens. The short lines on the left (AI2D, DocVQA) might represent older benchmarks that were "solved" (reaching 90%+) early on and thus abandoned for later models. Conversely, the long, straight lines (VideoMMMU, ERQA) suggest benchmarks that are computationally expensive to run, resulting in them only being tested on major milestone models (e.g., 5, 16, 21) rather than every incremental iteration.