## Line Chart: Pass Rate vs. SWE-Agent SFT Tokens ### Overview The image is a line chart comparing the pass rates of different models (RL, SFT, MT, and Base) at different "Pass" levels (@1, @2, @3) as the number of SWE-Agent SFT tokens increases. The x-axis represents the number of tokens, and the y-axis represents the pass rate in percentage. ### Components/Axes * **Title:** There is no explicit title on the chart. * **X-axis:** * Label: "# SWE-Agent SFT tokens" * Scale: The x-axis is logarithmic, with values at: 0, 2²¹, 2²³, 2²⁴, 1.1 x 2²⁵, 1.1 x 2²⁶, 1.1 x 2²⁷, 1.5 x 2²⁸ * **Y-axis:** * Label: "Pass Rate (%)" * Scale: Linear, ranging from 0 to 60, with increments of 10. * **Legend:** Located on the right side of the chart. It maps colors and shapes to different models and pass levels: * Red circle: RL Pass@1 * Red square: RL Pass@2 * Red triangle: RL Pass@3 * Orange circle: SFT Pass@1 * Orange square: SFT Pass@2 * Orange triangle: SFT Pass@3 * Purple circle: MT Pass@1 * Purple square: MT Pass@2 * Purple triangle: MT Pass@3 * Blue circle: Base Pass@1 * Blue square: Base Pass@2 * Blue triangle: Base Pass@3 ### Detailed Analysis Here's a breakdown of each data series and their trends: * **RL Pass@1 (Red Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~8% * 2²¹ tokens: ~23% * 2²³ tokens: ~34% * 2²⁴ tokens: ~34% * 1.1 x 2²⁵ tokens: ~46% * 1.1 x 2²⁶ tokens: ~51% * 1.1 x 2²⁷ tokens: ~58% * 1.5 x 2²⁸ tokens: ~62% * **RL Pass@2 (Red Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~9% * 2²¹ tokens: ~23% * 2²³ tokens: ~43% * 2²⁴ tokens: ~48% * 1.1 x 2²⁵ tokens: ~46% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~58% * 1.5 x 2²⁸ tokens: ~64% * **RL Pass@3 (Red Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~11% * 2²¹ tokens: ~38% * 2²³ tokens: ~44% * 2²⁴ tokens: ~48% * 1.1 x 2²⁵ tokens: ~54% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~61% * 1.5 x 2²⁸ tokens: ~66% * **SFT Pass@1 (Orange Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~13% * 2²¹ tokens: ~20% * 2²³ tokens: ~20% * 2²⁴ tokens: ~30% * 1.1 x 2²⁵ tokens: ~48% * 1.1 x 2²⁶ tokens: ~50% * 1.1 x 2²⁷ tokens: ~48% * 1.5 x 2²⁸ tokens: ~48% * **SFT Pass@2 (Orange Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~8% * 2²¹ tokens: ~15% * 2²³ tokens: ~31% * 2²⁴ tokens: ~31% * 1.1 x 2²⁵ tokens: ~51% * 1.1 x 2²⁶ tokens: ~51% * 1.1 x 2²⁷ tokens: ~58% * 1.5 x 2²⁸ tokens: ~58% * **SFT Pass@3 (Orange Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~12% * 2²¹ tokens: ~16% * 2²³ tokens: ~40% * 2²⁴ tokens: ~36% * 1.1 x 2²⁵ tokens: ~56% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~60% * 1.5 x 2²⁸ tokens: ~60% * **MT Pass@1 (Purple Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~1% * 2²³ tokens: ~6% * 2²⁴ tokens: ~29% * 1.1 x 2²⁵ tokens: ~45% * 1.1 x 2²⁶ tokens: ~45% * 1.1 x 2²⁷ tokens: ~46% * 1.5 x 2²⁸ tokens: ~59% * **MT Pass@2 (Purple Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~2% * 2²³ tokens: ~35% * 2²⁴ tokens: ~42% * 1.1 x 2²⁵ tokens: ~46% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~57% * 1.5 x 2²⁸ tokens: ~61% * **MT Pass@3 (Purple Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~2% * 2²³ tokens: ~40% * 2²⁴ tokens: ~43% * 1.1 x 2²⁵ tokens: ~53% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~57% * 1.5 x 2²⁸ tokens: ~63% * **Base Pass@1 (Blue Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~0% * 2²³ tokens: ~13% * 2²⁴ tokens: ~12% * 1.1 x 2²⁵ tokens: ~12% * 1.1 x 2²⁶ tokens: ~45% * 1.1 x 2²⁷ tokens: ~48% * 1.5 x 2²⁸ tokens: ~53% * **Base Pass@2 (Blue Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~2% * 2²³ tokens: ~2% * 2²⁴ tokens: ~22% * 1.1 x 2²⁵ tokens: ~22% * 1.1 x 2²⁶ tokens: ~22% * 1.1 x 2²⁷ tokens: ~36% * 1.5 x 2²⁸ tokens: ~57% * **Base Pass@3 (Blue Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~3% * 2²³ tokens: ~3% * 2²⁴ tokens: ~27% * 1.1 x 2²⁵ tokens: ~27% * 1.1 x 2²⁶ tokens: ~27% * 1.1 x 2²⁷ tokens: ~45% * 1.5 x 2²⁸ tokens: ~58% ### Key Observations * The RL models generally have the highest pass rates across all token counts. * The Base models generally have the lowest pass rates across all token counts, especially at lower token counts. * The pass rates for all models tend to increase as the number of tokens increases, but the rate of increase varies. * There are plateaus in some of the lines, where increasing the number of tokens does not immediately result in a higher pass rate. * The MT models start with very low pass rates at 0 tokens, but their performance improves significantly as the token count increases. ### Interpretation The chart demonstrates the impact of the number of SWE-Agent SFT tokens on the pass rates of different models (RL, SFT, MT, and Base) at different pass levels. The RL models appear to be the most effective, achieving the highest pass rates overall. The Base models, on the other hand, struggle at lower token counts but show significant improvement as the token count increases. The MT models exhibit a similar trend, starting with very low pass rates but catching up as the token count grows. The SFT models show a more moderate improvement with increasing token counts. The plateaus in some of the lines suggest that there may be a point of diminishing returns for increasing the number of tokens. It's possible that other factors, such as model architecture or training data, become more important beyond a certain token count. The data suggests that increasing the number of SWE-Agent SFT tokens can improve the performance of these models, but the extent of the improvement varies depending on the model and pass level.