## Line Chart: Pass Rate vs. SWE-Agent SFT Tokens ### Overview The image is a line chart comparing the pass rates of different models (RL, SFT, MT, and Base) at different "Pass" levels (@1, @2, @3) as the number of SWE-Agent SFT tokens increases. The x-axis represents the number of tokens, and the y-axis represents the pass rate in percentage. ### Components/Axes * **Title:** There is no explicit title on the chart. * **X-axis:** * Label: "# SWE-Agent SFT tokens" * Scale: The x-axis is logarithmic, with values at: 0, 2²¹, 2²³, 2²⁴, 1.1 x 2²⁵, 1.1 x 2²⁶, 1.1 x 2²⁷, 1.5 x 2²⁸ * **Y-axis:** * Label: "Pass Rate (%)" * Scale: Linear, ranging from 0 to 60, with increments of 10. * **Legend:** Located on the right side of the chart. It maps colors and shapes to different models and pass levels: * Red circle: RL Pass@1 * Red square: RL Pass@2 * Red triangle: RL Pass@3 * Orange circle: SFT Pass@1 * Orange square: SFT Pass@2 * Orange triangle: SFT Pass@3 * Purple circle: MT Pass@1 * Purple square: MT Pass@2 * Purple triangle: MT Pass@3 * Blue circle: Base Pass@1 * Blue square: Base Pass@2 * Blue triangle: Base Pass@3 ### Detailed Analysis Here's a breakdown of each data series and their trends: * **RL Pass@1 (Red Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~8% * 2²¹ tokens: ~23% * 2²³ tokens: ~34% * 2²⁴ tokens: ~34% * 1.1 x 2²⁵ tokens: ~46% * 1.1 x 2²⁶ tokens: ~51% * 1.1 x 2²⁷ tokens: ~58% * 1.5 x 2²⁸ tokens: ~62% * **RL Pass@2 (Red Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~9% * 2²¹ tokens: ~23% * 2²³ tokens: ~43% * 2²⁴ tokens: ~48% * 1.1 x 2²⁵ tokens: ~46% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~58% * 1.5 x 2²⁸ tokens: ~64% * **RL Pass@3 (Red Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~11% * 2²¹ tokens: ~38% * 2²³ tokens: ~44% * 2²⁴ tokens: ~48% * 1.1 x 2²⁵ tokens: ~54% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~61% * 1.5 x 2²⁸ tokens: ~66% * **SFT Pass@1 (Orange Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~13% * 2²¹ tokens: ~20% * 2²³ tokens: ~20% * 2²⁴ tokens: ~30% * 1.1 x 2²⁵ tokens: ~48% * 1.1 x 2²⁶ tokens: ~50% * 1.1 x 2²⁷ tokens: ~48% * 1.5 x 2²⁸ tokens: ~48% * **SFT Pass@2 (Orange Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~8% * 2²¹ tokens: ~15% * 2²³ tokens: ~31% * 2²⁴ tokens: ~31% * 1.1 x 2²⁵ tokens: ~51% * 1.1 x 2²⁶ tokens: ~51% * 1.1 x 2²⁷ tokens: ~58% * 1.5 x 2²⁸ tokens: ~58% * **SFT Pass@3 (Orange Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~12% * 2²¹ tokens: ~16% * 2²³ tokens: ~40% * 2²⁴ tokens: ~36% * 1.1 x 2²⁵ tokens: ~56% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~60% * 1.5 x 2²⁸ tokens: ~60% * **MT Pass@1 (Purple Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~1% * 2²³ tokens: ~6% * 2²⁴ tokens: ~29% * 1.1 x 2²⁵ tokens: ~45% * 1.1 x 2²⁶ tokens: ~45% * 1.1 x 2²⁷ tokens: ~46% * 1.5 x 2²⁸ tokens: ~59% * **MT Pass@2 (Purple Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~2% * 2²³ tokens: ~35% * 2²⁴ tokens: ~42% * 1.1 x 2²⁵ tokens: ~46% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~57% * 1.5 x 2²⁸ tokens: ~61% * **MT Pass@3 (Purple Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~2% * 2²³ tokens: ~40% * 2²⁴ tokens: ~43% * 1.1 x 2²⁵ tokens: ~53% * 1.1 x 2²⁶ tokens: ~57% * 1.1 x 2²⁷ tokens: ~57% * 1.5 x 2²⁸ tokens: ~63% * **Base Pass@1 (Blue Circle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~0% * 2²³ tokens: ~13% * 2²⁴ tokens: ~12% * 1.1 x 2²⁵ tokens: ~12% * 1.1 x 2²⁶ tokens: ~45% * 1.1 x 2²⁷ tokens: ~48% * 1.5 x 2²⁸ tokens: ~53% * **Base Pass@2 (Blue Square):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~2% * 2²³ tokens: ~2% * 2²⁴ tokens: ~22% * 1.1 x 2²⁵ tokens: ~22% * 1.1 x 2²⁶ tokens: ~22% * 1.1 x 2²⁷ tokens: ~36% * 1.5 x 2²⁸ tokens: ~57% * **Base Pass@3 (Blue Triangle):** The pass rate generally increases with the number of tokens. * 0 tokens: ~0% * 2²¹ tokens: ~3% * 2²³ tokens: ~3% * 2²⁴ tokens: ~27% * 1.1 x 2²⁵ tokens: ~27% * 1.1 x 2²⁶ tokens: ~27% * 1.1 x 2²⁷ tokens: ~45% * 1.5 x 2²⁸ tokens: ~58% ### Key Observations * The RL models generally have the highest pass rates across all token counts. * The Base models generally have the lowest pass rates across all token counts, especially at lower token counts. * The pass rates for all models tend to increase as the number of tokens increases, but the rate of increase varies. * There are plateaus in some of the lines, where increasing the number of tokens does not immediately result in a higher pass rate. * The MT models start with very low pass rates at 0 tokens, but their performance improves significantly as the token count increases. ### Interpretation The chart demonstrates the impact of the number of SWE-Agent SFT tokens on the pass rates of different models (RL, SFT, MT, and Base) at different pass levels. The RL models appear to be the most effective, achieving the highest pass rates overall. The Base models, on the other hand, struggle at lower token counts but show significant improvement as the token count increases. The MT models exhibit a similar trend, starting with very low pass rates but catching up as the token count grows. The SFT models show a more moderate improvement with increasing token counts. The plateaus in some of the lines suggest that there may be a point of diminishing returns for increasing the number of tokens. It's possible that other factors, such as model architecture or training data, become more important beyond a certain token count. The data suggests that increasing the number of SWE-Agent SFT tokens can improve the performance of these models, but the extent of the improvement varies depending on the model and pass level.

## Line Chart: Pass Rate vs. SWE-Agent SFT Tokens ### Overview This image is a complex line chart illustrating the performance (measured as "Pass Rate (%)") of four different machine learning models or training methodologies across varying amounts of training data (measured in "# SWE-Agent SFT tokens"). The chart evaluates each method using three different metrics: Pass@1, Pass@2, and Pass@3, resulting in 12 distinct data series. ### Components/Axes **1. Y-Axis (Left):** * **Label:** `Pass Rate (%)` * **Scale:** Linear, ranging from 0 to 65. * **Markers/Ticks:** Major ticks are marked at 0, 10, 20, 30, 40, 50, and 60. Faint, solid light-gray horizontal gridlines extend from these ticks across the chart area. **2. X-Axis (Bottom):** * **Label:** `# SWE-Agent SFT tokens` * **Scale:** Categorical/Non-linear progression of token counts. * **Markers/Ticks:** 0, $2^{21}$, $2^{23}$, $2^{24}$, $1.1 \times 2^{25}$, $1.1 \times 2^{26}$, $1.1 \times 2^{27}$, $1.5 \times 2^{28}$. Vertical dashed gray gridlines extend upward from each tick mark. **3. Legend (Right):** Positioned outside the main chart area on the right side, enclosed in a bounding box. It maps colors to methodologies and shapes to metrics. * **Colors (Methodologies):** * Red: RL (Reinforcement Learning) * Orange: SFT (Supervised Fine-Tuning) * Purple: MT (Multi-Task) * Blue: Base * **Shapes (Metrics):** * Circle: Pass@1 * Square: Pass@2 * Triangle: Pass@3 * **Exact Legend Entries (Top to Bottom):** * Red Circle: `RL Pass@1` * Red Square: `RL Pass@2` * Red Triangle: `RL Pass@3` * Orange Circle: `SFT Pass@1` * Orange Square: `SFT Pass@2` * Orange Triangle: `SFT Pass@3` * Purple Circle: `MT Pass@1` * Purple Square: `MT Pass@2` * Purple Triangle: `MT Pass@3` * Blue Circle: `Base Pass@1` * Blue Square: `Base Pass@2` * Blue Triangle: `Base Pass@3` ### Detailed Analysis **Visual Encoding & Trend Verification:** The chart utilizes two types of lines to convey information: 1. **Solid Lines (Intra-token scaling):** At every single X-axis tick, for every color, a solid line connects the Circle (Pass@1) to the Square (Pass@2) to the Triangle (Pass@3). *Trend:* Without exception, these solid lines slope steeply upward, confirming that for any given model at any given training stage, Pass@3 > Pass@2 > Pass@1. 2. **Dashed Lines (Inter-token scaling):** Dashed lines connect identical shapes of the same color across different X-axis ticks (e.g., connecting all Red Circles). *Trend:* The general trend for all dashed lines is upward from left to right, indicating that increasing SFT tokens generally improves the pass rate across all methods and metrics. **Data Extraction Table:** *Note: Values are visual approximations derived from the Y-axis scale (±1%).* | X-Axis Tick | Method (Color) | Pass@1 (Circle) | Pass@2 (Square) | Pass@3 (Triangle) | | :--- | :--- | :--- | :--- | :--- | | **0** | Base (Blue) | ~0% | ~0% | ~0% | | | MT (Purple) | ~1% | ~1% | ~1% | | | RL (Red) | ~4% | ~9% | ~12% | | | SFT (Orange) | ~8% | ~13% | ~16% | | **$2^{21}$** | Base (Blue) | ~1% | ~2% | ~3% | | | MT (Purple) | ~5% | ~6% | ~7% | | | SFT (Orange) | ~20% | ~33% | ~38% | | | RL (Red) | ~23% | ~33% | ~39% | | **$2^{23}$** | Base (Blue) | ~16% | ~24% | ~28% | | | MT (Purple) | ~27% | ~36% | ~44% | | | SFT (Orange) | ~27% | ~35% | ~41% | | | RL (Red) | ~33% | ~43% | ~48% | | **$2^{24}$** | Base (Blue) | ~13% | ~22% | ~28% | | | SFT (Orange) | ~20% | ~31% | ~36% | | | MT (Purple) | ~29% | ~41% | ~47% | | | RL (Red) | ~34% | ~42% | ~47% | | **$1.1 \times 2^{25}$** | Base (Blue) | ~12% | ~27% | ~36% | | | MT (Purple) | ~31% | ~46% | ~52% | | | RL (Red) | ~34% | ~45% | ~50% | | | SFT (Orange) | ~35% | ~45% | ~51% | | **$1.1 \times 2^{26}$** | Base (Blue) | ~22% | ~38% | ~45% | | | MT (Purple) | *No Data Plotted* | *No Data Plotted* | *No Data Plotted* | | | SFT (Orange) | ~37% | ~49% | ~55% | | | RL (Red) | ~38% | ~51% | ~58% | | **$1.1 \times 2^{27}$** | Base (Blue) | ~33% | ~48% | ~52% | | | SFT (Orange) | ~44% | ~55% | ~59% | | | RL (Red) | ~44% | ~56% | ~60% | | | MT (Purple) | ~45% | ~55% | ~60% | | **$1.5 \times 2^{28}$** | Base (Blue) | ~36% | ~48% | ~54% | | | MT (Purple) | ~46% | ~55% | ~60% | | | SFT (Orange) | ~48% | ~58% | ~62% | | | RL (Red) | ~49% | ~58% | ~64% | ### Key Observations 1. **Missing Data:** The MT (Purple) series has a distinct gap; there are no data points plotted at the $1.1 \times 2^{26}$ token mark. The dashed lines bridge directly from $1.1 \times 2^{25}$ to $1.1 \times 2^{27}$. 2. **Performance Hierarchy:** Throughout almost the entire chart, RL (Red) and SFT (Orange) are the top-performing methods, often overlapping or tracking very closely together. MT (Purple) generally sits in the middle, while the Base model (Blue) consistently yields the lowest pass rates. 3. **Anomalous Dips:** At the $2^{24}$ token mark, there is a noticeable regression in performance for the Base (Blue) and SFT (Orange) models compared to their performance at $2^{23}$. The Base Pass@1 drops from ~16% to ~13%, and SFT Pass@1 drops significantly from ~27% to ~20%. 4. **Convergence at Scale:** As the token count reaches the maximum ($1.5 \times 2^{28}$), the performance gap between the methods begins to narrow, particularly between RL, SFT, and MT, which all cluster tightly between 55% and 64% for Pass@2 and Pass@3. ### Interpretation This chart demonstrates the efficacy of different training interventions on a language model's ability to successfully complete software engineering tasks (implied by "SWE-Agent"). * **The Value of Multiple Attempts:** The steep solid lines at every interval prove that allowing the agent multiple attempts (Pass@3 vs Pass@1) drastically improves the likelihood of success, regardless of the underlying model or training stage. * **Training Efficacy:** The data clearly shows that fine-tuning (SFT) and Reinforcement Learning (RL) provide massive early advantages over the Base model. For instance, at $2^{21}$ tokens, SFT and RL are already achieving ~40% Pass@3, while the Base model is barely above 0%. * **Scaling Laws:** The overall upward trajectory of the dashed lines confirms a standard scaling law: exposing the model to more SFT tokens generally increases its pass rate. However, the dips at $2^{24}$ suggest that training is not perfectly linear and may experience instability or require learning rate adjustments at certain phases. * **Diminishing Returns:** While performance is still climbing at the far right of the chart ($1.5 \times 2^{28}$), the slope of the dashed lines is beginning to flatten slightly compared to the explosive growth seen between $2^{21}$ and $2^{23}$. This suggests that while more data helps, the marginal utility of each additional token is decreasing, and the models may be approaching an asymptotic performance limit for this specific benchmark.

## Line Chart: Pass Rate vs. SWE-Agent SFT Tokens ### Overview This line chart depicts the relationship between the number of SWE-Agent SFT tokens and the pass rate for different training methods (RL, SFT, MT, and Base). The pass rate is measured in percentage (%). Each training method is further categorized by the "Pass@k" metric, where k represents the number of attempts (1, 2, or 3). ### Components/Axes * **X-axis:** "# SWE-Agent SFT tokens". Scale is logarithmic, with markers at approximately 0, 221, 223, 224, 1.1 x 2²⁵, 1.1 x 2²⁶, 1.1 x 2²⁷, and 1.5 x 2²⁸. * **Y-axis:** "Pass Rate (%)". Scale ranges from 0 to 60, with increments of 10. * **Legend:** Located in the top-right corner, listing the following data series: * RL Pass@1 (Red) * RL Pass@2 (Medium Red) * RL Pass@3 (Light Red) * SFT Pass@1 (Orange) * SFT Pass@2 (Medium Orange) * SFT Pass@3 (Light Orange) * MT Pass@1 (Purple) * MT Pass@2 (Medium Purple) * MT Pass@3 (Light Purple) * Base Pass@1 (Blue) * Base Pass@2 (Medium Blue) * Base Pass@3 (Light Blue) ### Detailed Analysis Here's a breakdown of each data series, noting trends and approximate data points. * **RL Pass@1 (Red):** Starts at approximately 3% at 0 tokens, increases sharply to around 55% at 1.1 x 2²⁶ tokens, and plateaus around 58% at 1.5 x 2²⁸ tokens. * **RL Pass@2 (Medium Red):** Starts at approximately 5% at 0 tokens, increases steadily to around 50% at 1.1 x 2²⁶ tokens, and reaches approximately 55% at 1.5 x 2²⁸ tokens. * **RL Pass@3 (Light Red):** Starts at approximately 7% at 0 tokens, increases rapidly to around 45% at 1.1 x 2²⁶ tokens, and reaches approximately 52% at 1.5 x 2²⁸ tokens. * **SFT Pass@1 (Orange):** Starts at approximately 18% at 0 tokens, increases to around 45% at 1.1 x 2²⁶ tokens, and reaches approximately 50% at 1.5 x 2²⁸ tokens. * **SFT Pass@2 (Medium Orange):** Starts at approximately 15% at 0 tokens, increases to around 40% at 1.1 x 2²⁶ tokens, and reaches approximately 48% at 1.5 x 2²⁸ tokens. * **SFT Pass@3 (Light Orange):** Starts at approximately 12% at 0 tokens, increases to around 35% at 1.1 x 2²⁶ tokens, and reaches approximately 45% at 1.5 x 2²⁸ tokens. * **MT Pass@1 (Purple):** Starts at approximately 2% at 0 tokens, increases to around 25% at 1.1 x 2²⁶ tokens, and reaches approximately 30% at 1.5 x 2²⁸ tokens. * **MT Pass@2 (Medium Purple):** Starts at approximately 3% at 0 tokens, increases to around 20% at 1.1 x 2²⁶ tokens, and reaches approximately 25% at 1.5 x 2²⁸ tokens. * **MT Pass@3 (Light Purple):** Starts at approximately 4% at 0 tokens, increases to around 15% at 1.1 x 2²⁶ tokens, and reaches approximately 20% at 1.5 x 2²⁸ tokens. * **Base Pass@1 (Blue):** Starts at approximately 1% at 0 tokens, increases to around 10% at 1.1 x 2²⁶ tokens, and reaches approximately 15% at 1.5 x 2²⁸ tokens. * **Base Pass@2 (Medium Blue):** Starts at approximately 2% at 0 tokens, increases to around 8% at 1.1 x 2²⁶ tokens, and reaches approximately 12% at 1.5 x 2²⁸ tokens. * **Base Pass@3 (Light Blue):** Starts at approximately 3% at 0 tokens, increases to around 6% at 1.1 x 2²⁶ tokens, and reaches approximately 10% at 1.5 x 2²⁸ tokens. ### Key Observations * RL methods consistently achieve the highest pass rates across all "Pass@k" values. * Increasing the number of attempts ("Pass@k") generally improves the pass rate for each training method. * The pass rate improvement plateaus for all methods as the number of SWE-Agent SFT tokens increases beyond 1.1 x 2²⁶. * The Base methods have the lowest pass rates, indicating they benefit the least from increased SFT tokens. * The SFT methods perform better than the MT and Base methods, but not as well as the RL methods. ### Interpretation The data suggests that Reinforcement Learning (RL) is the most effective training method for improving pass rates, followed by Supervised Fine-Tuning (SFT). The number of SWE-Agent SFT tokens has a significant positive impact on pass rates, but there appears to be a diminishing return as the token count increases. The "Pass@k" metric demonstrates that allowing more attempts improves performance, which is expected. The relatively poor performance of the Base methods suggests that fine-tuning with SFT tokens is crucial for achieving higher pass rates. The logarithmic scale of the x-axis indicates that the initial gains in pass rate are more substantial at lower token counts, and that the rate of improvement slows down as the token count increases. This could be due to the model reaching a point of diminishing returns or saturation. The differences between Pass@1, Pass@2, and Pass@3 for each method show the benefit of allowing multiple attempts, and the magnitude of that benefit varies by method.

## Line Chart: Scaling Behavior of SWE-Agent Training Methods ### Overview This is a multi-series line chart illustrating the performance scaling of four different training methods (RL, SFT, MT, Base) for an AI agent called "SWE-Agent." The chart plots the "Pass Rate (%)" against the number of "SWE-Agent SFT tokens" used for training. Each method is evaluated using three metrics: Pass@1, Pass@2, and Pass@3, resulting in 12 distinct data series. The overall trend shows that performance improves for all methods as the training token count increases, though the rate of improvement and starting points vary significantly. ### Components/Axes * **X-Axis (Horizontal):** Labeled "# SWE-Agent SFT tokens". The scale is logarithmic, with tick marks at the following approximate values: `0`, `2^21` (~2.1 million), `2^23` (~8.4 million), `2^24` (~16.8 million), `1.1 × 2^25` (~36.9 million), `1.1 × 2^26` (~73.8 million), `1.1 × 2^27` (~147.6 million), and `1.5 × 2^28` (~402.7 million). * **Y-Axis (Vertical):** Labeled "Pass Rate (%)". The scale is linear, ranging from 0 to 60 with major gridlines every 10 units. * **Legend:** Positioned on the right side of the chart, outside the plot area. It defines the color and marker shape for each of the 12 data series: * **RL (Red):** Pass@1 (Circle), Pass@2 (Square), Pass@3 (Triangle) * **SFT (Orange):** Pass@1 (Circle), Pass@2 (Square), Pass@3 (Triangle) * **MT (Purple):** Pass@1 (Circle), Pass@2 (Square), Pass@3 (Triangle) * **Base (Blue):** Pass@1 (Circle), Pass@2 (Square), Pass@3 (Triangle) * **Grid:** A light gray dashed grid is present for both axes. ### Detailed Analysis The following table reconstructs the approximate data points for each series at the given token counts. Values are estimated from the chart's visual positioning. | Token Count (Approx.) | RL Pass@1 | RL Pass@2 | RL Pass@3 | SFT Pass@1 | SFT Pass@2 | SFT Pass@3 | MT Pass@1 | MT Pass@2 | MT Pass@3 | Base Pass@1 | Base Pass@2 | Base Pass@3 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **0** | ~4% | ~9% | ~12% | ~8% | ~13% | ~16% | ~0.5% | ~0.5% | ~0.5% | ~0% | ~0% | ~0% | | **2^21** | ~23% | ~33% | ~39% | ~20% | ~33% | ~38% | ~5% | ~6% | ~7% | ~1% | ~2% | ~3% | | **2^23** | ~33% | ~43% | ~48% | ~27% | ~35% | ~41% | ~27% | ~36% | ~44% | ~16% | ~24% | ~28% | | **2^24** | ~34% | ~42% | ~47% | ~20% | ~31% | ~36% | ~29% | ~41% | ~47% | ~13% | ~22% | ~28% | | **1.1 × 2^25** | ~34% | ~45% | ~50% | ~35% | ~46% | ~51% | ~31% | ~46% | ~52% | ~12% | ~28% | ~36% | | **1.1 × 2^26** | ~38% | ~51% | ~58% | ~37% | ~49% | ~55% | ~37% | ~51% | ~58% | ~22% | ~38% | ~45% | | **1.1 × 2^27** | ~44% | ~56% | ~60% | ~44% | ~55% | ~59% | ~45% | ~55% | ~60% | ~33% | ~48% | ~52% | | **1.5 × 2^28** | ~49% | ~58% | ~64% | ~48% | ~58% | ~62% | ~46% | ~55% | ~60% | ~36% | ~48% | ~54% | **Trend Verification per Method:** * **RL (Red Lines):** All three lines show a strong, consistent upward trend. The slope is steep initially (0 to 2^23 tokens) and continues to rise steadily, with RL Pass@3 achieving the highest overall pass rate on the chart. * **SFT (Orange Lines):** Also shows a strong upward trend. Notably, SFT Pass@1 exhibits a significant dip at 2^24 tokens before recovering and continuing its ascent. * **MT (Purple Lines):** Starts very low (near 0%) but demonstrates the most dramatic scaling. The lines have a very steep slope between 2^21 and 2^23 tokens, eventually converging with and sometimes surpassing the SFT lines at higher token counts. * **Base (Blue Lines):** Starts at or near 0% and shows the slowest initial growth. However, it exhibits a strong, consistent upward trend from 2^24 tokens onward, though it remains the lowest-performing group at every data point. ### Key Observations 1. **Performance Hierarchy:** At every token count, the performance order within each method is consistently Pass@3 > Pass@2 > Pass@1. This indicates that allowing the agent more attempts (k in Pass@k) reliably increases the success rate. 2. **Method Comparison:** RL and SFT methods start with a significant performance advantage over MT and Base at low token counts (0 to 2^21). MT shows a "catch-up" phenomenon, scaling rapidly to match SFT performance at higher token volumes. Base is consistently the lowest-performing method. 3. **Scaling Efficiency:** The most dramatic performance gains for all methods occur in the range between `2^21` and `1.1 × 2^25` tokens. After `1.1 × 2^26` tokens, the rate of improvement begins to plateau slightly for most series. 4. **Notable Anomaly:** The SFT Pass@1 series shows a clear performance drop at `2^24` tokens (from ~27% to ~20%) before recovering. This is the most pronounced deviation from the general upward trend in the chart. ### Interpretation This chart demonstrates the **scaling laws** for different training paradigms applied to the SWE-Agent. The data suggests that: * **Data Quantity is Critical:** All methods benefit from more training data (tokens), confirming that scale is a primary driver of performance for this agent. * **Training Method Matters:** The choice of training method (RL, SFT, MT, Base) has a profound impact on data efficiency. RL and SFT are highly data-efficient, achieving decent performance with relatively few tokens. MT is less efficient initially but scales very effectively. The Base method is the least data-efficient, requiring orders of magnitude more data to achieve comparable results. * **Metric Sensitivity:** The consistent gap between Pass@1, Pass@2, and Pass@3 highlights that the agent's "first-attempt" success rate is substantially lower than its success rate when given multiple chances. This is a crucial consideration for real-world deployment where the cost of multiple attempts may be high. * **Practical Implication:** For resource-constrained scenarios (limited training data/compute), RL or SFT would be preferable. If massive data is available, MT becomes a competitive option. The Base method appears to be a weak baseline, likely representing a model without specialized training for the SWE-Agent task. The dip in SFT Pass@1 at `2^24` tokens could indicate a point of instability or overfitting in that specific training run, warranting further investigation.

# Technical Document Analysis: Pass Rate vs. SWE-Agent SFT Tokens ## Chart Overview This line chart illustrates the relationship between the number of SWE-Agent SFT tokens and pass rates for various AI models. The x-axis uses a logarithmic scale to represent token counts, while the y-axis shows pass rates as percentages. ## Axis Labels - **X-axis**: `# SWE-Agent SFT tokens` (logarithmic scale: 0 → 1.5 × 10²⁸) - **Y-axis**: `Pass Rate (%)` (0 → 60%) ## Legend Located in the top-right corner, the legend maps colors/markers to models and metrics: | Color/Marker | Label | |--------------|-------------------| | Red circle | RL Pass@1 | | Red square | RL Pass@2 | | Red triangle | RL Pass@3 | | Orange circle| SFT Pass@1 | | Orange square| SFT Pass@2 | | Orange triangle| SFT Pass@3 | | Purple circle| MT Pass@1 | | Purple square| MT Pass@2 | | Purple triangle| MT Pass@3 | | Blue circle | Base Pass@1 | | Blue square | Base Pass@2 | | Blue triangle| Base Pass@3 | ## Key Trends 1. **RL Models** (red lines): - Steep upward slope across all pass@ metrics - Pass@3 consistently outperforms pass@1 and pass@2 - Example: At 1.5 × 10²⁸ tokens, RL Pass@3 reaches ~65% 2. **SFT Models** (orange lines): - Gradual increase with plateauing at higher token counts - Pass@3 maintains highest performance - Example: At 1.5 × 10²⁸ tokens, SFT Pass@3 reaches ~62% 3. **MT Models** (purple lines): - Moderate upward trajectory with diminishing returns - Pass@3 shows strongest improvement - Example: At 1.5 × 10²⁸ tokens, MT Pass@3 reaches ~58% 4. **Base Models** (blue lines): - Slow initial growth followed by plateau - Pass@3 marginally outperforms lower metrics - Example: At 1.5 × 10²⁸ tokens, Base Pass@3 reaches ~52% ## Data Points (Selected) | Token Count | RL Pass@1 | RL Pass@2 | RL Pass@3 | SFT Pass@1 | SFT Pass@2 | SFT Pass@3 | MT Pass@1 | MT Pass@2 | MT Pass@3 | Base Pass@1 | Base Pass@2 | Base Pass@3 | |-------------------|-----------|-----------|-----------|------------|------------|------------|-----------|-----------|-----------|-------------|-------------|-------------| | 1e28 | 58% | 56% | 65% | 59% | 57% | 62% | 55% | 53% | 58% | 52% | 50% | 52% | | 1.1 × 10²⁷ | 45% | 43% | 54% | 46% | 44% | 53% | 42% | 40% | 45% | 40% | 38% | 40% | | 1.1 × 10²⁶ | 35% | 33% | 44% | 36% | 34% | 43% | 31% | 29% | 34% | 30% | 28% | 30% | | 2.1 × 10²⁵ | 25% | 23% | 34% | 26% | 24% | 33% | 21% | 19% | 24% | 20% | 18% | 20% | | 2.3 × 10²⁴ | 15% | 13% | 24% | 16% | 14% | 23% | 11% | 9% | 14% | 10% | 8% | 10% | | 2.1 × 10²³ | 5% | 3% | 14% | 6% | 4% | 13% | 5% | 3% | 7% | 3% | 1% | 3% | | 1e21 | 2% | 1% | 2% | 3% | 1% | 2% | 1% | 0% | 1% | 0% | 0% | 0% | ## Spatial Grounding - Legend positioned at [x: 0.95, y: 0.95] (top-right corner) - All line colors/markers match legend entries exactly - No overlapping data series observed ## Trend Verification - All lines exhibit upward trajectories (confirmed visually) - RL/SFT models show steeper slopes than MT/Base models - Pass@3 metrics consistently outperform pass@1 and pass@2 across all models ## Component Isolation 1. **Header**: No explicit title present 2. **Main Chart**: - 12 distinct data series (3 metrics × 4 models) - Logarithmic x-axis enables visualization of wide token range 3. **Footer**: Legend provides model/metric mapping ## Critical Observations 1. RL models demonstrate strongest performance scaling with token count 2. SFT models maintain highest absolute pass rates at maximum token count 3. Base models show minimal improvement beyond 1.1 × 10²⁷ tokens 4. Pass@3 metrics consistently outperform lower metrics by 5-15 percentage points *Note: All numerical values extracted from visual inspection of the chart. No textual data present in the diagram.*