## Line Chart: Performance vs. Training Tokens ### Overview The image is a line chart comparing the performance of two models, "SWE-Bench Verified" and "SWE-Bench Multilingual," against the number of training tokens (in billions). The chart shows how performance (in percentage) changes as the models are trained with more data. ### Components/Axes * **X-axis:** Training Tokens (Billion). Scale ranges from approximately 5 to 70, with markers at 10, 20, 30, 40, 50, 60, and 70. * **Y-axis:** Performance (%). Scale ranges from 30 to 60, with markers at 30, 35, 40, 45, 50, 55, and 60. * **Legend:** Located in the bottom-right of the chart. * Blue line: SWE-Bench Verified * Orange line: SWE-Bench Multilingual ### Detailed Analysis * **SWE-Bench Verified (Blue Line):** * Trend: Generally slopes upward, indicating improved performance with more training tokens. * Data Points: * At 5 Billion Tokens: ~50.5% * At 10 Billion Tokens: ~52.5% * At 20 Billion Tokens: ~52.7% * At 30 Billion Tokens: ~52.3% * At 40 Billion Tokens: ~54.2% * At 50 Billion Tokens: ~56.0% * At 60 Billion Tokens: ~58.5% * At 65 Billion Tokens: ~58.5% * At 70 Billion Tokens: ~61.2% * **SWE-Bench Multilingual (Orange Line):** * Trend: Initially slopes upward, then fluctuates, but generally increases overall. * Data Points: * At 5 Billion Tokens: ~31.0% * At 10 Billion Tokens: ~38.5% * At 20 Billion Tokens: ~39.5% * At 30 Billion Tokens: ~41.2% * At 40 Billion Tokens: ~40.2% * At 50 Billion Tokens: ~44.0% * At 60 Billion Tokens: ~40.5% * At 65 Billion Tokens: ~40.0% * At 70 Billion Tokens: ~46.5% ### Key Observations * The "SWE-Bench Verified" model consistently outperforms the "SWE-Bench Multilingual" model across all training token values. * Both models show improvement in performance as the number of training tokens increases, but the "SWE-Bench Verified" model demonstrates a more stable and consistent upward trend. * The "SWE-Bench Multilingual" model shows more fluctuation in performance, particularly between 40 and 65 billion training tokens. ### Interpretation The data suggests that the "SWE-Bench Verified" model is more effective or better suited for the task being measured, as it consistently achieves higher performance levels than the "SWE-Bench Multilingual" model. The upward trends indicate that both models benefit from increased training data, but the "SWE-Bench Verified" model appears to leverage the data more efficiently. The fluctuations in the "SWE-Bench Multilingual" model's performance could indicate sensitivity to specific data subsets or potential overfitting issues. The performance gap between the two models widens as the number of training tokens increases, suggesting that the "SWE-Bench Verified" model scales better with more data.

\n ## Line Chart: Performance vs. Training Tokens ### Overview This line chart depicts the performance of two models, "SWE-Bench Verified" and "SWE-Bench Multilingual", as a function of training tokens. The x-axis represents the number of training tokens in billions, and the y-axis represents performance as a percentage. The chart shows how performance changes as the models are trained with increasing amounts of data. ### Components/Axes * **X-axis Title:** "Training Tokens (Billion)" * **Y-axis Title:** "Performance (%)" * **X-axis Markers:** 10, 20, 30, 40, 50, 60, 70 * **Y-axis Markers:** 30, 35, 40, 45, 50, 55, 60 * **Legend:** Located in the bottom-right corner. * "SWE-Bench Verified" - Blue line with circle markers. * "SWE-Bench Multilingual" - Orange line with circle markers. ### Detailed Analysis **SWE-Bench Verified (Blue Line):** The blue line representing "SWE-Bench Verified" generally slopes upward, indicating increasing performance with more training tokens. * At 10 Billion Tokens: Approximately 51% performance. * At 20 Billion Tokens: Approximately 52.5% performance. * At 30 Billion Tokens: Approximately 53.5% performance. * At 40 Billion Tokens: Approximately 55% performance. * At 50 Billion Tokens: Approximately 56% performance. * At 60 Billion Tokens: Approximately 57.5% performance. * At 70 Billion Tokens: Approximately 61% performance. **SWE-Bench Multilingual (Orange Line):** The orange line representing "SWE-Bench Multilingual" shows a more fluctuating trend. It initially increases, then decreases, and finally increases again. * At 10 Billion Tokens: Approximately 31% performance. * At 20 Billion Tokens: Approximately 40% performance. * At 30 Billion Tokens: Approximately 41.5% performance. * At 40 Billion Tokens: Approximately 43% performance. * At 50 Billion Tokens: Approximately 41% performance. * At 60 Billion Tokens: Approximately 40% performance. * At 70 Billion Tokens: Approximately 46% performance. ### Key Observations * "SWE-Bench Verified" consistently outperforms "SWE-Bench Multilingual" across all training token values. * The performance of "SWE-Bench Verified" shows a steady increase, with a more significant jump between 60 and 70 billion tokens. * "SWE-Bench Multilingual" exhibits a peak performance around 40 billion tokens, followed by a slight dip before increasing again at 70 billion tokens. * The gap between the two models widens as the number of training tokens increases. ### Interpretation The data suggests that increasing the number of training tokens generally improves the performance of both models. However, "SWE-Bench Verified" benefits more consistently from additional training data than "SWE-Bench Multilingual". The fluctuating performance of "SWE-Bench Multilingual" could indicate that it is more sensitive to the specific composition of the training data or that it may require more sophisticated training techniques to fully leverage larger datasets. The significant performance increase for "SWE-Bench Verified" at 70 billion tokens suggests a potential threshold effect, where a critical mass of data is required to unlock substantial performance gains. The consistent outperformance of "SWE-Bench Verified" implies it is a more robust or better-optimized model for the given task. The difference in performance could be due to architectural differences, training methodologies, or the specific datasets used for training.

## Line Chart: Performance vs. Training Tokens for SWE-Bench Benchmarks ### Overview This is a line chart comparing the performance of two benchmarks, "SWE-Bench Verified" and "SWE-Bench Multilingual," as a function of the number of training tokens (in billions). The chart shows a general upward trend for both benchmarks, with the "Verified" set consistently achieving higher performance scores than the "Multilingual" set across all measured training scales. ### Components/Axes * **Chart Type:** Line chart with markers. * **X-Axis:** * **Label:** "Training Tokens (Billion)" * **Scale:** Linear, ranging from approximately 5 to 70 billion. * **Major Tick Marks:** 10, 20, 30, 40, 50, 60, 70. * **Y-Axis:** * **Label:** "Performance (%)" * **Scale:** Linear, ranging from 30% to 60%. * **Major Tick Marks:** 30, 35, 40, 45, 50, 55, 60. * **Legend:** * **Position:** Bottom-right corner of the plot area. * **Entry 1:** Blue line with circular markers, labeled "SWE-Bench Verified". * **Entry 2:** Orange line with circular markers, labeled "SWE-Bench Multilingual". ### Detailed Analysis **Data Series 1: SWE-Bench Verified (Blue Line)** * **Trend:** The line shows a steady, generally upward trend with minor fluctuations. It starts just above 50% and ends above 60%. * **Approximate Data Points (Training Tokens (B), Performance (%)):** * (7, 50.5) * (13, 52.5) * (20, 52.7) * (27, 52.3) * (33, 54.7) * (40, 53.9) * (47, 56.0) * (53, 58.2) * (60, 58.2) * (67, 61.2) **Data Series 2: SWE-Bench Multilingual (Orange Line)** * **Trend:** The line shows an overall upward trend but with more pronounced volatility compared to the blue line. It starts near 31% and ends near 46%. * **Approximate Data Points (Training Tokens (B), Performance (%)):** * (7, 31.0) * (13, 38.5) * (20, 39.5) * (27, 41.0) * (33, 41.0) * (40, 40.0) * (47, 43.8) * (53, 40.7) * (60, 40.0) * (67, 46.3) ### Key Observations 1. **Performance Gap:** The "SWE-Bench Verified" benchmark consistently outperforms the "SWE-Bench Multilingual" benchmark by a significant margin (approximately 15-20 percentage points) at every measured training token scale. 2. **Scaling Law:** Both benchmarks demonstrate a positive correlation between the number of training tokens and performance, suggesting that model capability on these tasks improves with scale. 3. **Volatility Difference:** The "Multilingual" series exhibits more performance volatility (e.g., dips at 40B and 60B tokens) compared to the relatively smoother progression of the "Verified" series. 4. **Final Surge:** Both series show their steepest performance increase in the final segment, from 60B to 67B tokens. ### Interpretation The chart illustrates a fundamental scaling relationship in machine learning: increasing the volume of training data (tokens) generally leads to better performance on downstream benchmarks. The consistent performance gap between "SWE-Bench Verified" and "SWE-Bench Multilingual" suggests that the multilingual variant of the task is inherently more challenging for the model, possibly due to the need to generalize across multiple programming languages or handle more diverse codebases. The higher volatility in the multilingual series could indicate that performance on this more complex task is more sensitive to the specific composition of the training data at different scales, or that the model's development path for multilingual understanding is less monotonic. The final sharp uptick for both lines might signify a phase change in model capability or the effect of a specific training strategy employed at the largest scale. **Language Note:** All text in the image is in English.

## Line Graph: SWE-Bench Model Performance vs. Training Tokens ### Overview The image depicts a line graph comparing the performance of two SWE-Bench models ("Verified" and "Multilingual") across varying amounts of training tokens (in billions). Performance is measured as a percentage on the y-axis, while training tokens are plotted on the x-axis. The graph spans from 10B to 70B training tokens. ### Components/Axes - **X-axis**: Training Tokens (Billion) – labeled with increments of 10B (10, 20, 30, ..., 70). - **Y-axis**: Performance (%) – labeled with increments of 5% (30, 35, 40, ..., 60). - **Legend**: Located in the bottom-right corner, with two entries: - **Blue line**: "SWE-Bench Verified" - **Orange line**: "SWE-Bench Multilingual" ### Detailed Analysis #### SWE-Bench Verified (Blue Line) - **Trend**: Steadily increases from ~50% at 10B tokens to ~60% at 70B tokens. - **Key Data Points**: - 10B: ~50% - 20B: ~52% - 30B: ~52% - 40B: ~54% - 50B: ~56% - 60B: ~58% - 70B: ~60% #### SWE-Bench Multilingual (Orange Line) - **Trend**: Initial sharp rise from 30% at 10B tokens to 45% at 50B tokens, followed by a dip to 40% at 60B tokens, then a recovery to 45% at 70B tokens. - **Key Data Points**: - 10B: ~30% - 20B: ~38% - 30B: ~41% - 40B: ~40% - 50B: ~44% - 60B: ~40% - 70B: ~45% ### Key Observations 1. **Performance Gap**: The Verified model consistently outperforms the Multilingual model across all token ranges. 2. **Multilingual Model Volatility**: The Multilingual model shows a peak at 50B tokens (44%) followed by a sharp decline to 40% at 60B tokens, suggesting potential overfitting or instability at higher training scales. 3. **Scalability**: The Verified model demonstrates linear improvement with increased training tokens, while the Multilingual model’s gains plateau or regress. ### Interpretation The data suggests that the **SWE-Bench Verified** model benefits significantly from additional training data, achieving near-linear performance gains. In contrast, the **SWE-Bench Multilingual** model exhibits diminishing returns and instability at higher token counts, possibly due to challenges in handling multilingual data diversity or overfitting to specific language subsets. The Multilingual model’s dip at 60B tokens may indicate architectural limitations or data quality issues when scaled beyond a certain threshold. These findings highlight the importance of model verification and targeted training strategies for multilingual applications.