# SWE-Universe: Scale Real-World Verifiable Environments to Millions
**Authors**:
- Junyang Lin, Binyuan Hui (Qwen Team, Alibaba Group, Zhejiang University)
> Work done during an internship at Alibaba Qwen.
Abstract
We propose SWE-Universe, a scalable and efficient framework for automatically constructing real-world software engineering (SWE) verifiable environments from GitHub pull requests (PRs). To overcome the prevalent challenges of automatic building, such as low production yield, weak verifiers, and prohibitive cost, our framework utilizes a building agent powered by an efficient custom-trained model. This agent employs iterative self-verification and in-loop hacking detection to ensure the reliable generation of high-fidelity, verifiable tasks. Using this method, we scale the number of real-world multilingual SWE environments to a million scale (807,693). We demonstrate the profound value of our environments through large-scale agentic mid-training and reinforcement learning. Finally, we applied this technique to Qwen3-Max-Thinking and achieved a score of 75.3% on SWE-Bench Verified. Our work provides both a critical resource and a robust methodology to advance the next generation of coding agents.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Bar Chart: Number of Real-World Verifiable SWE Instances
### Overview
The chart compares the number of real-world verifiable software engineering (SWE) instances across different datasets/tools, categorized by Python-only and Multilingual implementations. The data is presented as grouped bars, with Python-only in blue and Multilingual in orange. The y-axis represents instance counts, while the x-axis lists specific SWE frameworks/datasets.
### Components/Axes
- **Title**: "Number of Real-World Verifiable SWE Instances"
- **X-Axis Labels**:
- SWE-Bench
- SWE-Gym
- Multi-SWE-RL
- SWE-rebench
- DeepSeek-V3.2
- CWM
- MiMo-V2-Flash
- SWE-Universe (Ours)
- **Y-Axis**: Instance counts (logarithmic scale implied by spacing)
- **Legend**:
- Blue = Python-only
- Orange = Multilingual
- **Bar Colors**:
- Python-only: Blue
- Multilingual: Orange
### Detailed Analysis
| Dataset/Tool | Python-only | Multilingual |
|-----------------------|-------------|--------------|
| SWE-Bench | 2,294 | 2,438 |
| SWE-Gym | 2,438 | 4,723 |
| Multi-SWE-RL | 4,723 | 21,000 |
| SWE-rebench | 21,000 | 24,667 |
| DeepSeek-V3.2 | 24,667 | 35,000 |
| CWM | 35,000 | 90,000 |
| MiMo-V2-Flash | 90,000 | 807,693 |
| SWE-Universe (Ours) | N/A | 807,693 |
### Key Observations
- **Multilingual dominance**: Multilingual instances consistently outnumber Python-only across all categories, with ratios increasing exponentially (e.g., 1.06x for SWE-Bench to 22.5x for SWE-Universe).
- **Exponential growth**: The largest gap appears in SWE-Universe (Ours), where Multilingual instances reach 807,693, dwarfing all prior categories.
- **Outlier**: SWE-Universe (Ours) is an extreme outlier, with Multilingual instances exceeding the previous highest (MiMo-V2-Flash) by 8.97x.
### Interpretation
The data suggests that Multilingual SWE implementations are significantly more prevalent or effective in real-world scenarios compared to Python-only approaches. The SWE-Universe (Ours) category demonstrates a breakthrough in scalability, achieving 807,693 Multilingual instancesâan order of magnitude higher than prior tools. This implies that Multilingual frameworks may better address diverse linguistic requirements in software engineering tasks. The absence of Python-only data for SWE-Universe could indicate either a lack of Python support or a strategic focus on Multilingual capabilities in this dataset.
</details>
Figure 1: A comparison of the number of instances in real-world SWE instances. Our multilingual SWE-Universe is significantly larger than other recent multilingual efforts like MiMo-V2-Flash (Xiao et al., 2026), DeepSeek-V3.2 (DeepSeek-AI, 2025), and Multi-SWE-RL (Zan et al., 2025), as well as prominent Python-only datasets including SWE-rebench (Badertdinov et al., 2025), SWE-Gym (Pan et al., 2024), CWM (Copet et al., 2025), and SWE-Bench (Jimenez et al., 2024).
1 Introduction
Training large language models (LLMs) (Yang et al., 2025a; Liu et al., 2025) as coding agents has gained substantial attention in software engineering (SWE). Progress in this direction is critically dependent on large-scale, high-quality environments with reliable verification signals. An ideal source for such data lies within the vast ecosystem of real-world software development, specifically public GitHub repositories where pull requests (PRs) are linked to corresponding issues. Each such PR defines a self-contained training environment: the issue serves as a clear problem statement, the code patch offers an expert reference solution, and the accompanying tests can be repurposed into a verifier for agent proposed fixes. This formulation effectively turns each PR into a âgymâ for training and evaluating coding agents. Benchmarks such as SWE-bench (Jimenez et al., 2024) operationalized this pipeline by curating real-world issues and standardizing the evaluation, serving as a common testbed for subsequent work.
However, scaling the construction of verifiable real-world SWE environments while maintaining diversity and reproducibility remains a major challenge. A vast body of work, including the original SWE-bench and its many derivatives and extensions (Pan et al., 2024; Badertdinov et al., 2025; Zhang et al., 2025c; Guo et al., 2025; Zeng et al., 2026; Wang et al., 2025; Aleithan et al., 2024; Copet et al., 2025) has primarily focused on Python. While this line of work leverages the low barrier to entry for Python environment configuration, it nonetheless restricts the development of agents with true cross-lingual generalization capabilities. Efforts to create multi-lingual datasets, such as Multi-SWE-bench (Zan et al., 2025) and SWE-PolyBench (Rashid et al., 2025), rely on labor-intensive manual environment setup and remain limited in scale. Despite recent efforts of industrial LLM providers that scale SWE instances to the $10^{4}-10^{5}$ magnitude (Xiao et al., 2026; DeepSeek-AI, 2025), the technical details are undisclosed.
We argue that scaling the generation of such verifiable environments to the massive scale (e.g., $10^{6}$ ) hinges on overcoming three fundamental challenges:
- Low Production Yield: The intricate and heterogeneous nature of real-world repositories with their complex dependencies, platform-specific configurations, and bespoke build toolchains, yielding a low conversion rate from repositories to runnable instances. This results in significant computational waste and makes large-scale generation impractical.
- Weak Verifier: Issues, PR patches, and test suites exhibit substantial variance in quality. A naive extraction pipeline can produce low-fidelity instances, and can also admit verifier vulnerabilities that allow solutions to pass via shallow heuristics (e.g., string matching with grep) rather than compiling and executing the intended code. Such failure modes create spurious training signals and distort evaluation.
- Prohibitive Cost and Inefficiency: Many existing pipelines rely on large, expensive LLMs to perform repository-specific reasoning for dependency resolution and build configuration. The resulting cost and latency per instance make massive-scale generation economically and operationally impractical.
To systematically address these challenges, we introduce SWE-Universe, a scalable, reliable, and efficient framework for automatically constructing million-scale, real-world agentic software engineering environments. At its core is an autonomous building agent that, for each PR, synthesizes a self-contained executable environment together with an executable verifier. To mitigate low production yield, the agent performs self-verification in an iterative validation loop. Concretely, it repeatedly tests the generated verifier against both the buggy and the fixed repository states, diagnoses failure modes, and revises the build procedure accordingly. This process improves the build success rate from 82.6% to 94% on a held-out set. To address the weak verifier, we integrate an in-loop hacking detector that immediately flags and rejects superficial verifiers during the generation process, forcing the agent toward solutions that genuinely execute the code. To achieve high efficiency and low cost, we specially trained an efficient Qwen-Next-80B-A3B model with a Mixture-of-Experts (MoE) framework and hybrid attention. On our custom multi-lingual building benchmark, this model achieves a 78.44% success rate, surpassing top-tier proprietary models like Claude-Opus-4.5, while its efficient architecture dramatically reduces the latency and cost per build.
Leveraging the SWE-Universe framework, we construct 807,693 multilingual, verifiable training instances sourced from over 52,000 unique GitHub repositories. To our knowledge, this dataset is currently the largest and most diverse collection of real-world software engineering tasks with executable verification. We further validate the immense value of this dataset through extensive experiments. First, in a mid-training phase, we show that continued training on our dataset significantly enhances a modelâs performance on standard benchmarks like SWE-Bench Verified (Jimenez et al., 2024), proving its powerful generalization capabilities. Second, we demonstrate that the verifiers generated by our pipeline provide a stable and effective reward signal for Reinforcement Learning (RL). By applying to our flagship model, Qwen3-Max-Thinking (Qwen Team, 2026), we achieved a score of 75.3% on the SWE-Bench Verified. Together, these results validate SWE-Universe as a robust framework for creating large-scale, high-quality agentic training data, paving the way for the development of more capable and versatile coding agents for real-world applications.
2 Methodology: Scalability and Reliability
<details>
<summary>figures/env-build.png Details</summary>
### Visual Description
## Flowchart: Software Engineering Task Automation Pipeline
### Overview
The diagram illustrates an automated software engineering workflow involving pull request handling, code verification, security checks, and task execution. It features feedback loops, conditional branching, and integration with containerized environments.
### Components/Axes
1. **Pull Request Section** (Top-left)
- Label: "Real-world PR"
- Sub-components:
- `sklearn` directory
- `gradient_boosting.py`
- `helper.py`
- `utils` directory
- Status indicators: `+20 -12` (green/red squares)
2. **Building & Verifier Agent** (Center-left)
- Tools: `bash`, `switch-to-bug`, `switch-to-resolved`
- Connections:
- Input from Pull Request
- Output to Repository Container
- Feedback to Verifier Validation
3. **Hacking Detector** (Bottom-left)
- Evaluation scripts:
- `pytest test_production.py` (â pass)
- `grep -q "keyword" bug.java` (â fail)
- Icon: đľď¸ââď¸ (detective)
4. **Repository Container** (Center)
- Codebase structure:
- `sklearn/`
- `examples/`
- `README.rst`
- `reqs.txt`
- `setup.cfg`
- `setup.py`
5. **Verifier Validation** (Top-right)
- Conditions:
- `switch-to-bug` + non-zero exit code â đ´ fail
- `switch-to-resolved` + 0 exit code â â pass
- Icon: âď¸ (gear with checkmark)
6. **New Software Engineering Task** (Bottom-right)
- Elements:
- Problem statement (đ)
- Docker image (đŚ)
- Evaluation script (đ)
- Icon: đ (smiling face)
### Detailed Analysis
- **Pull Request Flow**:
- PR â Building Agent â Repository Container
- Verifier Validation â New Task (if pass) or Retry (if fail)
- **Agentic Loop**:
- Orange/blue arrows indicate iterative testing/verification cycles
- Red dashed "retry" path for failed evaluations
- **Security Checks**:
- Hacking Detector analyzes evaluation scripts for malicious patterns
- Keyword detection in Java files (`bug.java`)
- **Containerization**:
- Docker image integration for isolated task execution
- File structure mirrors typical Python project layout
### Key Observations
1. **Conditional Branching**:
- Verification outcomes directly control task progression
- Failed security checks trigger retries rather than immediate rejection
2. **Automation Indicators**:
- Agentic loop suggests AI/ML-driven decision making
- Containerized environment enables reproducible testing
3. **Security Integration**:
- Hacking Detector operates in parallel with core verification
- Malicious pattern detection uses both static (grep) and dynamic (pytest) analysis
4. **Exit Code Semantics**:
- Non-zero exit codes trigger bug investigation
- Zero exit codes enable task resolution
### Interpretation
This diagram represents a sophisticated CI/CD pipeline with embedded security and quality assurance mechanisms. The agentic loop suggests autonomous system behavior where:
1. Pull requests trigger automated building and verification
2. Security checks run concurrently with functional testing
3. Failures initiate iterative debugging rather than blocking
4. Successful validations spawn new engineering tasks
The use of containerization (Docker) and repository structure implies cloud-native development practices. The hacking detector's dual approach (static pattern matching + dynamic testing) indicates a layered security strategy. The smiling face icon in the final task suggests positive user experience outcomes when the pipeline succeeds.
</details>
Figure 2: Our SWE-Universe framework for scalable and reliable environment building. The pipeline is built around a building agent that proposes a verifier (evaluation.sh). Two key components ensure quality and yield: an in-loop Hacking Detector that preemptively rejects superficial scripts, and an Iterative Validation loop where the agent self-corrects based on feedback from testing its verifier against both buggy and resolved code states.
Our primary goal is to develop a method for automated verifiable SWE environment construction that is both scalable and reliable. Scalability implies compatibility across diverse programming languages and build toolchains, as well as maximizing the build success rate to minimize computational waste. Reliability requires that the generated verifier (i.e., an evaluation.sh script) accurately provides a reward signal for a given patch, passing only when the SWE requirements are met. To this end, we design an autonomous building agent based on the popular mini-sweagent scaffold.
Our key insight is that while directly verifying the success of a complex building is difficult, assessing the success of a test script is comparatively straightforward: one can simply run it to see if it distinguishes between the pre-patch (buggy) and post-patch (fixed) states, as adopted by DeepSeek-V3.2 (Liu et al., 2025). However, we argue that this condition alone is insufficient. A naive agent might generate a script that âhacksâ the verification by using simple string matching to confirm the patchâs application, rather than actually executing the code. Such a script would correctly distinguish states but would fail to validate the environment or the behavioral correctness of the fix. Therefore, we establish a more robust acceptance criterion for a successfully built task: the verifier must not only correctly distinguish between states but must do so by genuinely executing the code under test. Our methodology, detailed below and sumarized in Figure Ë 2, is designed to meet this twofold objective.
2.1 Key Designs for SWE-Universe
PR Crawling and Patch Separation.
The pipeline begins by sourcing a large corpus of issue-linked pull requests (PRs) from public GitHub repositories. To prevent data contamination and ensure fair evaluation, we meticulously filter out any PRs that overlap with known downstream benchmarks. For each valid PR, we employ a language model to analyze the code modifications and partition them into a test patch (containing test-related changes) and a fix patch (containing the source code fix). PRs lacking a discernible test component are discarded.
Agent-based Environment Building.
Following patch separation, our autonomous agent, equipped with a set of powerful tools, initiates the construction of the environment and its corresponding verifier. The process commences by applying the test patch to the repository. The agent is then tasked with generating a verifier script, designated as evaluation.sh, whose objective is to reliably distinguish between the repositoryâs states based solely on its return code. Guided by the information within the test patch, the agent has the flexibility to adopt one of two strategies: it can either directly invoke the unit tests described in the patch, or, in scenarios where those tests lack a straightforward execution entry point, it can author a new custom test from scratch. This standardized approach, centered on a universal bash interface and its integer return code, is a deliberate design choice. It decouples the verification logic from language-specific conventions (e.g., hardcoding a pytest or cargo run workflow), thereby maximizing the methodâs generalizability and scalability across disparate projects and ecosystems.
Toolset.
We equip the agent with three tools: bash, switch-to-resolved, and switch-to-bug.
- bash: A general-purpose shell for file manipulation, dependency installation, and script generation.
- switch-to-resolved and switch-to-bug: A pair of tools that allow the agent to atomically apply or revert the fix patch, toggling the repository between its fixed and buggy states.
These state-switching tools are fundamental to our approach, as they empower the agent to perform self-verification. By testing its own actions in a closed loop, the agent can diagnose and recover from failures, which is key to improving the overall build success rate and achieving scalability.
Iterative Validation.
To ensure the verifier is reliable, the agent engages in an iterative validation loop. After the agent ends with submitting a candidate evaluation.sh script, we execute it under both repository states using its switching tools. A script is considered functionally correct only if it fails (exits with a non-zero status) in the buggy state and succeeds (exits with a zero status) in the fixed state. If the script fails this validation, the agent receives this negative feedback, discards the faulty script, and is prompted to generate a new, revised version until reaches the maximum turns we predefined (e.g., 100 turns). This iterative process substantially improves reliability: for a set of held-out PRs, the environment-building success rate increases from 82.6% to 94%.
In-loop Hacking Detection.
To meet our second criterionâthat the verifier must genuinely execute the codeâwe integrate a Hacking Detector directly into the agentâs work cycle. This module uses an LLM to inspect the generated evaluation.sh script for âhackingâ patterns, such as using grep or other string-matching utilities to check the contents of source files instead of running a build or test command. Critically, this check is performed within the agentâs loop, not as a post-processing step. If the detector flags a script as a âhackâ, this attempt is immediately considered a failure. This provides timely feedback to the agent, forcing it to abandon superficial strategies and guiding it toward a valid solution that involves actual code execution. This in-loop design improves both the reliability of the final artifact and the efficiency of the agentâs search process.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Screenshot: Code Snippets for Software Testing Methods
### Overview
The image displays three vertically stacked code snippets in a terminal-style interface, each demonstrating different approaches to software testing. The snippets are labeled with their respective methodologies: (1) Human-written unit tests, (2) LLM-written tests, and (3) Hacking via static checking. Each panel contains code in different programming languages (bash, Python, Java) with syntax highlighting.
### Components/Axes
- **Panel 1 (Human-written tests)**:
- **Language**: Bash/Python
- **Key Elements**:
- Bash script for testing a Python module (`glyphs`)
- Python code for mocking dependencies and testing edge cases
- Error handling for missing dependencies and server issues
- **Notable Features**:
- Manual test setup with explicit dependency checks
- Use of `unittest.mock` for patching
- Logging of test results and error patterns
- **Panel 2 (LLM-written tests)**:
- **Language**: Python
- **Key Elements**:
- `unittest.TestCase` class for glyph module testing
- Mocking of `glyphs.builder.hints` and `glyphs.module`
- Test cases for edge cases (e.g., empty glyphs, missing exceptions)
- **Notable Features**:
- Structured test methods (`test_glyphs_module`, `test_no_error_occurred`)
- Use of `unittest.mock.patch.object` for precise mocking
- Focus on edge-case validation
- **Panel 3 (Hacking/static checking)**:
- **Language**: Java
- **Key Elements**:
- Test for `watchCheckTermination` functionality
- JSON response validation for `failAction` field
- Source code change detection logic
- **Notable Features**:
- Use of `TestCase.assertTrue` for validation
- Regex pattern matching for JSON field checks
- Security-focused testing of API endpoints
### Detailed Analysis
1. **Human-written Tests (Panel 1)**:
- Demonstrates manual test setup with explicit dependency management
- Includes error handling for missing modules and server connectivity issues
- Uses `pytest` conventions with `pytest.raises` for exception testing
- Logs test outcomes with color-coded status messages
2. **LLM-written Tests (Panel 2)**:
- Shows AI-generated test structure with comprehensive mocking
- Tests both nominal and edge cases (empty glyphs, missing exceptions)
- Uses `unittest.mock` for precise control over mocked objects
- Includes test cleanup logic (`tearDown` method)
3. **Hacking/static Checking (Panel 3)**:
- Implements security testing for API endpoints
- Validates JSON response structure and field presence
- Uses regex patterns for flexible field matching
- Tests source code change detection mechanisms
### Key Observations
- All three approaches use different testing philosophies:
- Human-written: Manual, explicit, and comprehensive
- LLM-written: Structured but potentially less context-aware
- Hacking: Security-focused with pattern matching
- The code snippets demonstrate progression from basic unit testing to security validation
- Color coding in the terminal interface helps distinguish code elements and test outcomes
### Interpretation
This image illustrates the evolution of testing methodologies in software development:
1. **Human-written tests** represent traditional, thorough testing approaches with explicit control over test conditions
2. **LLM-written tests** show the integration of AI in generating test structures, though potentially requiring human refinement
3. **Hacking/static checking** demonstrates security testing techniques that go beyond standard unit tests
The progression suggests a spectrum of testing approaches from manual verification to automated security checks. The presence of both Python and Java code indicates cross-language testing capabilities, while the terminal interface reflects real-world development environments where these tests would be executed.
</details>
Figure 3: Three types of evaluation.sh. We only accept the first two types of evaluation scripts.
2.2 Further Analysis
Case Study
We examine three representative LLM-generated test scripts to illustrate our verification criteria (Figure 3). Case 1 tests a JavaScript authentication bug by executing existing human-written unit tests in the Jest framework via pnpm exec jest, checking for specific error patterns in the output. Case 2 validates a Python glyphsLib bug by creating LLM-generated mock objects and unit tests that directly exercise the buggy code path and verify execution without exceptions. Case 3 attempts to verify a Scala bug fix using static pattern matching with grep commands to detect the presence of specific code structures, which is flagged as âhackingâ by our hacking detection. We accept Cases 1 and 2 as they employ strong verification through executable tests that validate runtime behavior, while we reject Case 3 because static pattern matching cannot guarantee correctnessâa patch may introduce expected code patterns while still containing logical errors.
Quality Analysis
We find that the resulting data still exhibit several quality issues: (1) some task descriptions are ambiguous or incomplete; (2) certain Docker environments do not fully match the stated requirements; and (3) some unit tests are misaligned with the task descriptions, which can lead to false positives or false negatives. To quantify and mitigate this problem, we develop a quality-judge agent. It takes as input the task description, Docker environment, test scripts, and optionally the ground-truth patch, and automatically evaluates the task quality. On a human-labeled quality-judging benchmark, the agent reaches 78.72% accuracy. After applying it to the entire dataset, as shown in Figure 4, we find our dataset matches SWE-Rebench (Badertdinov et al., 2025) in quality while providing 38Ă more instances.
<details>
<summary>x3.png Details</summary>
### Visual Description
## Scatter Plot: Dataset Size vs Quality Score
### Overview
The image is a scatter plot comparing dataset sizes (logarithmic scale) against quality scores (linear percentage scale). Four labeled data points are plotted, each with distinct colors and positions. The chart emphasizes the relationship between dataset size and perceived quality, with notable outliers and trends.
### Components/Axes
- **Y-Axis (Dataset Size)**: Logarithmic scale from 10² to 10⡠(count).
- **X-Axis (Quality Score)**: Linear scale from 50% to 100%.
- **Legend**: Located on the right, associating colors with labels:
- **Blue**: SWE-Universe
- **Orange**: SWE-rebench
- **Brown**: SWE-Gym
- **Green**: SWE-bench Verified
### Detailed Analysis
1. **Data Points**:
- **SWE-Universe** (Blue): Positioned at (60%, 10âś). Highest dataset size (1,000,000) with the lowest quality score (60%).
- **SWE-rebench** (Orange): Positioned at (62%, 10â´). Smaller dataset (10,000) with slightly higher quality (62%).
- **SWE-Gym** (Brown): Positioned at (75%, 10Âł). Further reduced dataset size (1,000) with a quality score of 75%.
- **SWE-bench Verified** (Green): Positioned at (87%, 10²). Smallest dataset (100) with the highest quality score (87%).
2. **Trends**:
- **Inverse Relationship**: As dataset size decreases (log scale), quality scores increase. This suggests a trade-off between quantity and quality.
- **Logarithmic Clustering**: Dataset sizes span 5 orders of magnitude (10² to 10âś), while quality scores cluster between 60% and 87%.
### Key Observations
- **SWE-bench Verified** stands out as an outlier: it achieves the highest quality score (87%) with the smallest dataset (100 entries), implying rigorous curation or specialized content.
- **SWE-Universe** dominates in dataset size (10âś) but has the lowest quality score (60%), indicating potential noise or redundancy.
- The progression from SWE-Universe to SWE-bench Verified shows a clear trend of prioritizing quality over quantity.
### Interpretation
The chart highlights a critical insight: **higher-quality datasets are often smaller and more curated**, while larger datasets may sacrifice quality for breadth. The SWE-bench Verified dataset exemplifies this, suggesting it is a gold-standard benchmark. Conversely, SWE-Universeâs large size and low quality score may reflect a "firehose" approach, prioritizing accessibility over refinement. The intermediate datasets (SWE-rebench and SWE-Gym) demonstrate incremental improvements in quality with reduced size, possibly indicating iterative filtering or domain-specific optimization. This pattern underscores the importance of dataset design choices in balancing scalability and reliability.
</details>
Figure 4: Task Quality vs. Dataset Size (Log-Scale). Task quality is measured as the fraction of high-quality samples.
3 Efficient Building and Benchmarking
To optimize both the performance and throughput of our pipeline, we developed a high-capacity yet efficient model and a comprehensive cross-lingual benchmark to evaluate its capabilities.
Model Training and Deployment
To make our end-to-end pipeline efficient at scale, we train a lightweight but strong builder model, Qwen-Next-80A3 (Qwen Team, 2025b), a mixture-of-experts (MoE) model with hybrid attentions including linear attentions and full attentions. The model was trained using rejection sampling on a collection of high-quality building trajectories. This process involved sampling multiple candidate paths for environment construction and filtering for successful, non-hacked outcomes to serve as training data. The model serves as the unified backbone for all tasks in our pipeline, including PR patch splitting, iterative environment building, and the hacking detector. Its efficient architecture allows us to scale the building process across thousands of repositories with significantly lower latency compared to dense models of similar performance.
Benchmark
To rigorously assess the reliability of automated environment construction, we constructed a diverse benchmark consisting of 320 pull requests randomly sampled from GitHub. We remove the repositories used in the training trajectories from the benchmark. To ensure broad representativeness, we selected 40 PRs for each of the eight language categories: Python, JavaScript/TypeScript, Go, Java, Rust, C/C++, C#, and an âOthersâ category (including all the other languages such as PHP and Kotlin). We define two primary success metrics: Success Rate (w/o Hack), which requires the verifier to be functionally correct and pass the hacking detector, and Success Rate (w/ Hack), which includes all scripts that distinguish the bug regardless of the detection outcome.
Table 1: Benchmark results for automated environment building across various models. âSuccess (%) (w/o Hack)â measures the rate of creating a valid, non-hacked verifiable environment, while âSuccess (%) (w/ Hack)â also counts âhackedâ verifiers as successful builds. Our model, Qwen-Next-80A3, achieves the highest non-hacking success rate.
| Model | Success (%) | Success (%) | Success Rate by Language (%) | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (w/o Hack) | (w/ Hack) | C/C++ | C# | Go | Java | JS/TS | Other | Python | Rust | |
| Qwen-Next-80A3 (Ours) | $78.44$ | $82.50$ | $57.50$ | $70.00$ | $80.49$ | $82.50$ | $84.62$ | $83.72$ | $85.37$ | $83.33$ |
| Claude-Opus-4.5 | $77.81$ | $85.00$ | $52.50$ | $57.50$ | $82.93$ | $77.50$ | $89.74$ | $76.74$ | $95.12$ | $91.67$ |
| Claude-Sonnet-4 | $75.62$ | $85.62$ | $67.50$ | $52.50$ | $75.61$ | $72.50$ | $84.62$ | $83.72$ | $82.93$ | $86.11$ |
| Gemini-3-Pro | $69.69$ | $72.50$ | $32.50$ | $57.50$ | $73.17$ | $72.50$ | $87.18$ | $76.74$ | $87.80$ | $69.44$ |
| Claude-Sonnet-4-5 | $66.88$ | $71.56$ | $30.00$ | $50.00$ | $63.41$ | $67.50$ | $87.18$ | $67.44$ | $92.68$ | $77.78$ |
| GLM-4.7 | $58.44$ | $64.06$ | $37.50$ | $50.00$ | $53.66$ | $57.50$ | $66.67$ | $65.12$ | $73.17$ | $63.89$ |
| MiniMax-M2.1 | $54.69$ | $61.88$ | $22.50$ | $35.00$ | $56.10$ | $55.00$ | $74.36$ | $65.12$ | $73.17$ | $55.56$ |
| DeepSeek-v3.2 | $54.06$ | $59.38$ | $15.00$ | $50.00$ | $63.41$ | $47.50$ | $53.85$ | $48.84$ | $78.05$ | $77.78$ |
| Qwen3-Coder-480B | $48.75$ | $55.62$ | $35.00$ | $32.50$ | $39.02$ | $45.00$ | $48.72$ | $69.77$ | $68.29$ | $50.00$ |
Result Analysis
Our evaluation, detailed in Table 1, establishes Qwen-Next-80A3 as the new state-of-the-art model for automated environment building, achieving a 78.44% success rate that surpasses even strong proprietary models like Claude-Opus-4.5 (77.81%). This superior performance extends beyond raw success to overall reliability. We observe a significant gap between the true success rate and the one including âhackedâ verifiers for most general-purpose models (e.g., over 7% for Claude-Opus-4.5), indicating their tendency to find superficial shortcuts. In stark contrast, our model exhibits the smallest gap among top performers (4.06%), demonstrating that our task-specific fine-tuning on collected trajectories effectively discourages these deceptive behaviors. Furthermore, while performance varies by languageâwith C/C++ proving most challenging for all agentsâour model demonstrates more consistent and robust capabilities across the entire linguistic spectrum compared to competitors that show strength only in specific ecosystems. Ultimately, these results validate that a specialized, efficiently trained MoE model not only achieves higher success but also operates with greater fidelity than larger, general-purpose counterparts in this complex domain.
4 Scaling Environments to Millions
In this section, we describe our efforts to scale the environment-building pipeline to a massive corpus of real-world software changes, moving beyond small-scale benchmarks to a million-scale dataset of executable tasks.
Large-scale Data Curation
We harvested a comprehensive dataset of approximately 33.3 million pull requests (PRs) spanning the most recent five years (2021â2025) of GitHubâs history. To extract high-quality tasks from this raw pool, we applied a series of rigorous heuristic filters. Specifically, we removed PRs with excessive file changes or line counts to avoid overly complex or noisy tasks, and discarded any entries that did not contain a discernible test patch. Furthermore, we prioritized PRs that were explicitly linked to at least one GitHub issue, as these provide the most reliable ground-truth problem statements. This filtering process resulted in a candidate set of approximately 1 million high-quality PRs.
Infrastructure for Large-scale Rollouts
To support our massive data generation effort, we implement our pipeline on top of MegaFlow (Zhang et al., 2026), a distributed execution system for orchestrating large numbers of long-running agentic jobs. MegaFlow achieves massive parallelism by dispatching each environment-building task as an independent job to a dedicated Alibaba Cloud Elastic Compute Service (ECS) instance. Within this sandboxed virtual machine, our agent executes the entire build process, culminating in a verified Docker image. Upon completion, successful images are pushed to Alibaba Cloudâs Container Registry (ACR) https://www.alibabacloud.com/en/product/container-registry, where we leverage Dockerâs layer caching to significantly reduce storage costs by reusing common base layers. This highly parallel and resource-efficient architecture was instrumental in processing millions of pull requests concurrently, enabling the practical construction of our million-scale dataset.
Massive Production with Qwen-Next-80A3
We deployed our fine-tuned Qwen-Next-80A3 model to execute the building and verification process for the filtered candidates. Leveraging the modelâs efficiency and the iterative agentic loop described in previous sections, we achieved a non-hacked success rate of 75.9%. This process ultimately yielded 717,122 executable, high-fidelity environments. These tasks are primarily focused on assessing an agentâs ability to resolve real-world software issues. Additionally, to expand the datasetâs coverage, we selected a subset of 2025 PRs that were not linked to issues, utilizing the PR titles and descriptions as the problem statements to synthesize an additional environments, and producing 90,571 environments.
Table 2: Data statistics for our building environments.
| Language | Instances | Repos | Inst/Repo (Avg) | Avg Lines of evaluation.sh |
| --- | --- | --- | --- | --- |
| Python | 202,302 | 13,098 | 15.45 | 25.01 |
| Javascript / Typescript | 175,660 | 11,604 | 15.14 | 27.41 |
| Go | 121,062 | 5,554 | 21.80 | 28.87 |
| Java | 86,105 | 4,700 | 18.32 | 24.75 |
| Rust | 74,180 | 4,445 | 16.69 | 19.31 |
| C / C++ | 37,228 | 3,405 | 10.93 | 45.78 |
| C# | 24,387 | 1,929 | 12.64 | 31.84 |
| Others | 86,769 | 8,225 | 10.55 | 38.89 |
| # Total | 807,693 | 52,960 | 15.25 | 28.21 |
Statistical Analysis
The statistics presented in Table 2 highlight both the scale and diversity of our generated dataset. With 807,693 instances spread across 52,960 unique repositories, this dataset represents an unprecedented resource for training and evaluating software engineering agents. The language distribution largely mirrors the current open-source landscape, with Python and JavaScript/TypeScript constituting the largest shares. Notably, Go repositories exhibit the highest average number of instances per repository (21.80), suggesting a high density of verifiable, issue-linked PRs within its ecosystem, possibly due to strong conventions around testing and development. The âAvg Eval Linesâ metric reveals interesting insights into the complexity of verification across languages. C/C++ instances, as expected, require the longest verifier scripts on average (45.78 lines), reflecting the typical complexity and boilerplate of their build systems. In contrast, Rustâs concise average (19.31 lines) may point to the efficiency and standardization of its cargo toolchain, allowing for more succinct test execution commands. Overall, the dataset provides a rich and varied testbed, capturing a wide array of real-world challenges for coding agents.
5 Evaluation: Large-scale Agentic Training
We now turn to leveraging this resource for large-scale agentic model training. Our goal is to demonstrate that training on this diverse, real-world data can significantly enhance a modelâs capabilities as a software engineering agent.
5.1 Mid-training
We hypothesize that intermediate training on a vast corpus of high-quality agentic trajectories can endow a model with a strong foundation in both software engineering problem-solving. This âmid-trainingâ phase is designed to bridge the gap between the pre-training and post-training on downstream agentic tasks.
Setup
Our mid-training process begins by generating a massive dataset of agentic trajectories. We first employed the Qwen3-Coder-480B-A30B (Qwen Team, 2025a) model to interact with the environments we previously constructed. To ensure a diversity of problem-solving strategies, we conducted these rollouts across five different agentic scaffolds: SWE-agent (Yang et al., 2024), Mini-SWE-agent (SWE-agent Team, 2025), OpenHands (Wang et al., 2024), Claude-Code (Anthropic, 2024), Qwen-Code (Qwen Team, 2025c). The entire rollout process was orchestrated by our MegaFlow system. For each environment, we performed rejection sampling: a trajectory is deemed successful and retained only if the final generated code passes the corresponding evaluation.sh script and clears an additional in-house quality filter. This rigorous filtering process yielded a high-quality dataset of 500K successful trajectories, comprising a total of 30 billion training tokens. We then used this data to conduct intermediate training on a Qwen3-Next-80A3 model, using 256K sequence length and Best-Fit packing (Ding et al., 2024). Critically, we applied no loss mask during this training phase. This strategy ensures that the model learns not only to predict agentic actions but also to internalize the vast amount of code and natural language in the agentâs observations and responses, developing it into a more comprehensive âcoding world modelâ (Copet et al., 2025; Zhang et al., 2025a).
Scaling Trends
To analyze the effectiveness of our large-scale agentic mid-training, we evaluated model checkpoints on two widely used agentic coding benchmarks: SWE-Bench Verified (primarily Python-based) and the more diverse SWE-Bench Multilingual. The performance evolution throughout the training process is depicted in Figure 5(a). The results demonstrate a clear and positive scaling trend, confirming that our mid-training on the real-world instances successfully transfers to standard evaluation benchmarks. On the standard SWE-Bench Verified, the modelâs performance exhibits a steady climb, starting from an already strong baseline of 50.3% and consistently improving to a final score of over 61% after 2,000 training steps. More notably, the trend on the more challenging SWE-Bench Multilingual benchmark shows even more dramatic improvement. Starting from a significantly lower baseline of approximately 31%, the modelâs performance surges to over 46% by the end of the training. This substantial gain of over 15 percentage points underscores the critical value of our datasetâs linguistic diversity. It proves that training on a massive, diverse corpus of real-world, multi-language software issues is essential for creating agents that can generalize beyond a single programming ecosystem.
5.2 Agentic Reinforcement Learning
Beyond training via rejection sampling, our collection of executable environments is suited for Reinforcement Learning (RL), where the binary pass/fail signal from the evaluation.sh script serves as a direct and reliable reward. To demonstrate this, we conducted agentic RL experiments on different models with different sizes and structures.
Setup
We validate the effectiveness of the data on agentic reinforcement learning. Prior to training, we perform rollouts with the base model to filter out queries that are either too difficult or too easy. During training, we set the maximum number of interaction turns to 200 and the context length to 128k. The training process is powered by our asynchronous RL framework, which natively supports agentic workflows. This architecture mitigates data skewness while facilitating seamless multi-turn interactions without framework-induced overhead, achieving a 2xâ4x speedup compared to existing RL infrastructures.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Line Graph: SWE-Bench Model Performance vs. Training Tokens
### Overview
The image depicts a line graph comparing the performance of two SWE-Bench models ("Verified" and "Multilingual") across varying amounts of training tokens (in billions). Performance is measured as a percentage on the y-axis, while training tokens are plotted on the x-axis. The graph spans from 10B to 70B training tokens.
### Components/Axes
- **X-axis**: Training Tokens (Billion) â labeled with increments of 10B (10, 20, 30, ..., 70).
- **Y-axis**: Performance (%) â labeled with increments of 5% (30, 35, 40, ..., 60).
- **Legend**: Located in the bottom-right corner, with two entries:
- **Blue line**: "SWE-Bench Verified"
- **Orange line**: "SWE-Bench Multilingual"
### Detailed Analysis
#### SWE-Bench Verified (Blue Line)
- **Trend**: Steadily increases from ~50% at 10B tokens to ~60% at 70B tokens.
- **Key Data Points**:
- 10B: ~50%
- 20B: ~52%
- 30B: ~52%
- 40B: ~54%
- 50B: ~56%
- 60B: ~58%
- 70B: ~60%
#### SWE-Bench Multilingual (Orange Line)
- **Trend**: Initial sharp rise from 30% at 10B tokens to 45% at 50B tokens, followed by a dip to 40% at 60B tokens, then a recovery to 45% at 70B tokens.
- **Key Data Points**:
- 10B: ~30%
- 20B: ~38%
- 30B: ~41%
- 40B: ~40%
- 50B: ~44%
- 60B: ~40%
- 70B: ~45%
### Key Observations
1. **Performance Gap**: The Verified model consistently outperforms the Multilingual model across all token ranges.
2. **Multilingual Model Volatility**: The Multilingual model shows a peak at 50B tokens (44%) followed by a sharp decline to 40% at 60B tokens, suggesting potential overfitting or instability at higher training scales.
3. **Scalability**: The Verified model demonstrates linear improvement with increased training tokens, while the Multilingual modelâs gains plateau or regress.
### Interpretation
The data suggests that the **SWE-Bench Verified** model benefits significantly from additional training data, achieving near-linear performance gains. In contrast, the **SWE-Bench Multilingual** model exhibits diminishing returns and instability at higher token counts, possibly due to challenges in handling multilingual data diversity or overfitting to specific language subsets. The Multilingual modelâs dip at 60B tokens may indicate architectural limitations or data quality issues when scaled beyond a certain threshold. These findings highlight the importance of model verification and targeted training strategies for multilingual applications.
</details>
(a)
<details>
<summary>x5.png Details</summary>
### Visual Description
## Line Graph: SWE-Bench Multilingual (%) Performance Over Training Steps
### Overview
The image depicts a line graph illustrating the performance of a multilingual system (SWE-Bench) across training steps. The y-axis represents performance percentage (%), while the x-axis represents training steps. The graph shows an overall upward trend with fluctuations, culminating in a peak performance of 42.0% marked with a star.
### Components/Axes
- **Y-Axis**: "SWE-Bench Multilingual (%)" with increments of 2% (32â42).
- **X-Axis**: "Training Steps" (no specific numerical labels, only categorical progression).
- **Legend**: Located at the top-right corner, labeled "SWE-Bench Multilingual (%)" in orange.
- **Title**: "SWE-Bench Multilingual (%)" positioned at the top-right corner.
### Detailed Analysis
- **Data Points**:
- Initial value: ~32.3% (bottom-left).
- First peak: ~37.8% (early training steps).
- Dip: ~36.1% (mid-training steps).
- Further dip: ~35.0% (mid-late training steps).
- Recovery: ~37.0% (late training steps).
- Final peak: 42.0% (marked with a star, top-right).
- **Trend**: The line exhibits an overall upward trajectory with two notable dips. The final value (42.0%) is the highest, marked with a star, suggesting a target or optimal performance threshold.
### Key Observations
1. **Initial Growth**: Sharp increase from 32.3% to 37.8% in early training steps.
2. **Mid-Training Fluctuations**: Two dips (36.1% â 35.0%) indicate potential instability or optimization challenges.
3. **Final Surge**: Steep rise from 37.0% to 42.0% in later steps, suggesting successful model refinement or data adaptation.
4. **Star Annotation**: The 42.0% peak is explicitly highlighted, possibly denoting a performance milestone.
### Interpretation
The graph demonstrates the system's ability to improve multilingual performance over training iterations, despite mid-process instability. The final peak (42.0%) implies successful optimization, potentially through architectural adjustments, data augmentation, or hyperparameter tuning. The dips may reflect temporary setbacks, such as overfitting or data quality issues during specific training phases. The star annotation emphasizes the significance of the 42.0% threshold, likely representing a target or state-of-the-art benchmark. The absence of specific training step labels (e.g., epochs or iterations) limits granular analysis but highlights the general progression from initialization (~32%) to peak performance (~42%).
</details>
(b)
Figure 5: (a) Performance scaling trends of the Qwen3-Next-80A3 model during mid-training. (b) Reinforcement learning curve of the Qwen3-30B-A3B for SWE-Bench Multilingual.
Results on Qwen3-30B-A3B
The training curve in Figure 5(b) shows the effectiveness of agentic reinforcement training. The Qwen3-30a3 model shows a remarkable improvement on SWE-Bench Multilingual, surging from a baseline of approximately 32% to a peak of 42.0%. This substantial 10-point absolute gain underscores the power of RL to enhance generalization across diverse, multilingual tasks.
Applied to Qwen3-Max-Thinking
To push the boundaries of agentic coding performance, we applied our methodology to our flagship Qwen3-Max-Thinking model. The resulting model achieved a performance of 75.3% on SWE-Bench Verified. This result validates the effectiveness of our large-scale data generation pipeline at a production level, showcasing its ability to elevate state-of-the-art models.
6 Related Work
Agentic and Synthetic Environment Generation
A significant line of research focuses on generating software engineering tasks synthetically to create scalable training data, often without relying on authentic historical issues. For instance, SWE-smith (Yang et al., 2025b) procedurally generates tasks by artificially injecting bugs, while SWE-Flow (Zhang et al., 2025b) leverages test documentation to synthesize novel problems. Other works generate complex bugs from scratch (Sonwane et al., 2025) or create interactive tasks from bug reports (Jin et al., 2023). The scope of synthesis extends to generating command-line tasks from natural language (Lin et al., 2018; Gandhi et al., 2026). Similarly, RepoST (Xie et al., 2024) employs synthetic tests to generate training data. While these methods provide scalable training data, they do not fully capture the complexity and long-tail challenges of real-world software issues. In contrast, our work focuses exclusively on building executable environments for real-world software engineering problems derived directly from public repositories.
Real-World Software Environment Setup
Much recent work focuses purely on the environment configuration capability itself, proposing scripted, agentic, or expert-driven methods to create executable environments, but without providing verifiers for specific software issues (Hu et al., 2025; Milliken et al., 2025; Bouzenia & Pradel, 2025; Horton & Parnin, 2019; Jain et al., 2024; Vergopoulos et al., 2025; Guo et al., 2026; Arora et al., 2025; Eliseeva et al., 2025; Kuang et al., 2025; Fu et al., 2025). Another line of work provides end-to-end verification for real software issues. The seminal SWE-bench (Jimenez et al., 2024) established this paradigm with over 2,400 real-world Python issues, each with a test script to verify a fix. This Python-centric approach was significantly scaled by SWE-rebench (Badertdinov et al., 2025), which developed a fully automated pipeline to generate over 21,000 verifiable tasks. Other works like SWE-Gym (Pan et al., 2024), SWE-bench-Live (Zhang et al., 2025c), SWE-Factory (Guo et al., 2025), daVinci-Dev (Zeng et al., 2026), SWE-Bench++ (Wang et al., 2025), and SWE-bench+ (Aleithan et al., 2024) have also aimed to expand data generation, primarily within the Python ecosystem. A few recent efforts have created multi-language benchmarks, such as Multi-SWE-bench (Zan et al., 2025) and SWE-PolyBench (Rashid et al., 2025); however, these have generally been limited in scale. Our work systematically overcomes these limitations by presenting a highly scalable and reliable pipeline that operates across arbitrary languages and repositories, successfully generating over million-scale executable environments with a validated, task-specific verifier.
Building Code Verifier
Automated code verifiers are typically constructed by executing test suites to validate solutions. Traditional test generation methods, including probability-based approaches (Pacheco et al., 2007), constraint-based (Xiao et al., 2013), and search-based (Harman & McMinn, 2010; Lukasczyk & Fraser, 2022), often suffer from limited coverage and poor readability, and are typically restricted to regression or implicit oracles (Barr et al., 2015). Recent work leverages LLMs to generate unit tests for verification (Alagarsamy et al., 2024; Chen et al., 2024b; Schäfer et al., 2024; Yuan et al., 2024; Chen et al., 2024a), though the resulting tests can be unreliable due to confidently incorrect assertions. CodeRM (Ma et al., 2025) improves reward signal quality by scaling the number of generated tests and dynamically adapting test counts to problem difficulty. In this work, we leverage a building agent to automatically generate verifier for complext repository-level software engineering issues.
7 Conclusion
In this paper, we introduced SWE-Universe, a scalable framework designed to overcome the critical bottlenecks of low yield, inconsistent quality, and prohibitive cost in generating real-world software engineering environments. Our autonomous building agent, powered by a custom-trained model and equipped with iterative self-verification and in-loop hacking detection, successfully constructed a massive dataset of over 800,000 executable, multi-lingual tasksâthe largest of its kind. We demonstrated the profound value of this resource through large-scale agentic training, showing that it provides a powerful signal for both supervised learning and reinforcement learning, ultimately enabling our Qwen3-Max-Thinking model to achieve a score of 75.3% on SWE-Bench Verified.
References
- Alagarsamy et al. (2024) Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. A3test: Assertion-augmented automated test case generation. Inf. Softw. Technol., 176(C), November 2024. ISSN 0950-5849. doi: 10.1016/j.infsof.2024.107565. URL https://doi.org/10.1016/j.infsof.2024.107565.
- Aleithan et al. (2024) Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992, 2024.
- Anthropic (2024) Anthropic. Claude code: An agentic coding assistant. https://github.com/anthropics/claude-code, 2024. Accessed: 2026-01-22.
- Arora et al. (2025) Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing software engineering agentsâ ability to bootstrap development environments. arXiv preprint arXiv:2507.09063, 2025.
- Badertdinov et al. (2025) Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents, 2025. URL https://arxiv.org/abs/2505.20411.
- Barr et al. (2015) Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507â525, 2015. doi: 10.1109/TSE.2014.2372785.
- Bouzenia & Pradel (2025) Islem Bouzenia and Michael Pradel. You name it, i run it: An llm agent to execute tests of arbitrary projects. Proc. ACM Softw. Eng., 2(ISSTA), June 2025. doi: 10.1145/3728922. URL https://doi.org/10.1145/3728922.
- Chen et al. (2024a) Mouxiang Chen, Zhongxin Liu, He Tao, Yusu Hong, David Lo, Xin Xia, and Jianling Sun. B4: Towards optimal assessment of plausible code solutions with plausible tests. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE â24, pp. 1693â1705, New York, NY, USA, 2024a. Association for Computing Machinery. ISBN 9798400712487. doi: 10.1145/3691620.3695536. URL https://doi.org/10.1145/3691620.3695536.
- Chen et al. (2024b) Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pp. 572â576, New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400706585. doi: 10.1145/3663529.3663801. URL https://doi.org/10.1145/3663529.3663801.
- Copet et al. (2025) Jade Copet, Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, et al. Cwm: An open-weights llm for research on code generation with world models. arXiv preprint arXiv:2510.02387, 2025.
- DeepSeek-AI (2025) DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models. CoRR, abs/2512.02556, 2025. doi: 10.48550/ARXIV.2512.02556. URL https://doi.org/10.48550/arXiv.2512.02556.
- Ding et al. (2024) Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. Fewer truncations improve language modeling. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=kRxCDDFNpp.
- Eliseeva et al. (2025) Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. Envbench: A benchmark for automated environment setup. arXiv preprint arXiv:2503.14443, 2025.
- Fu et al. (2025) Kelin Fu, Tianyu Liu, Zeyu Shang, Yingwei Ma, Jian Yang, Jiaheng Liu, and Kaigui Bian. Multi-docker-eval: Ashovel of the gold rushâbenchmark on automatic environment building for software engineering. arXiv preprint arXiv:2512.06915, 2025.
- Gandhi et al. (2026) Kanishk Gandhi, Shivam Garg, Noah D Goodman, and Dimitris Papailiopoulos. Endless terminals: Scaling rl environments for terminal agents. arXiv preprint arXiv:2601.16443, 2026.
- Guo et al. (2025) Lianghong Guo, Yanlin Wang, Caihua Li, Pengyu Yang, Jiachi Chen, Wei Tao, Yingtian Zou, Duyu Tang, and Zibin Zheng. Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks. arXiv preprint arXiv:2506.10954, 2025.
- Guo et al. (2026) Xinshuai Guo, Jiayi Kuang, Linyue Pan, Yinghui Li, Yangning Li, Hai-Tao Zheng, Ying Shen, Di Yin, and Xing Sun. Evoconfig: Self-evolving multi-agent systems for efficient autonomous environment configuration. arXiv preprint arXiv:2601.16489, 2026.
- Harman & McMinn (2010) Mark Harman and Phil McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering, 36(2):226â247, 2010. doi: 10.1109/TSE.2009.71.
- Horton & Parnin (2019) Eric Horton and Chris Parnin. Dockerizeme: Automatic inference of environment dependencies for python code snippets. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 328â338. IEEE, 2019.
- Hu et al. (2025) Ruida Hu, Chao Peng, Xinchen Wang, Junjielong Xu, and Cuiyun Gao. Repo2run: Automated building executable environment for code repository at scale. arXiv preprint arXiv:2502.13681, 2025.
- Jain et al. (2024) Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: turning any github repository into a programming agent environment. In Proceedings of the 41st International Conference on Machine Learning, ICMLâ24. JMLR.org, 2024.
- Jimenez et al. (2024) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66.
- Jin et al. (2023) Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering, pp. 1646â1656, 2023.
- Kuang et al. (2025) Jiayi Kuang, Yinghui Li, Xin Zhang, Yangning Li, Di Yin, Xing Sun, Ying Shen, and Philip S Yu. Process-level trajectory evaluation for environment configuration in software engineering agents. arXiv preprint arXiv:2510.25694, 2025.
- Lin et al. (2018) Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
- Liu et al. (2025) Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025.
- Lukasczyk & Fraser (2022) Stephan Lukasczyk and Gordon Fraser. Pynguin: automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ICSE â22, pp. 168â172, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392235. doi: 10.1145/3510454.3516829. URL https://doi.org/10.1145/3510454.3516829.
- Ma et al. (2025) Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6917â6935, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.343. URL https://aclanthology.org/2025.acl-long.343/.
- Milliken et al. (2025) Louis Milliken, Sungmin Kang, and Shin Yoo. Beyond pip Install: Evaluating LLM Agents for the Automated Installation of Python Projects . In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 1â11, Los Alamitos, CA, USA, March 2025. IEEE Computer Society. doi: 10.1109/SANER64311.2025.00009. URL https://doi.ieeecomputersociety.org/10.1109/SANER64311.2025.00009.
- Pacheco et al. (2007) Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSEâ07), pp. 75â84, 2007. doi: 10.1109/ICSE.2007.37.
- Pan et al. (2024) Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024.
- Qwen Team (2025a) Qwen Team. Qwen3-coder: Agentic coding in the world. https://qwen.ai/blog?id=qwen3-coder, 2025a. Accessed: 2025-07-22.
- Qwen Team (2025b) Qwen Team. Qwen3-next: Towards ultimate training & inference efficiency. https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, 2025b. Accessed: 2025-09-11.
- Qwen Team (2025c) Qwen Team. qwen-code. https://github.com/QwenLM/qwen-code, 2025c.
- Qwen Team (2026) Qwen Team. Pushing qwen3-max-thinking beyond its limits. https://qwen.ai/blog?id=qwen3-max-thinking, 2026. Accessed: 2026-01-26.
- Rashid et al. (2025) Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703, 2025.
- Schäfer et al. (2024) Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85â105, 2024. doi: 10.1109/TSE.2023.3334955.
- Sonwane et al. (2025) Atharv Sonwane, Isadora White, Hyunji Lee, Matheus Pereira, Lucas Caccia, Minseon Kim, Zhengyan Shi, Chinmay Singh, Alessandro Sordoni, Marc-Alexandre CĂ´tĂŠ, et al. Bugpilot: Complex bug generation for efficient learning of swe skills. arXiv preprint arXiv:2510.19898, 2025.
- SWE-agent Team (2025) SWE-agent Team. mini-swe-agent: The 100 line ai agent thatâs actually useful. https://github.com/SWE-agent/mini-swe-agent, 2025. Accessed: 2026-01-22.
- Vergopoulos et al. (2025) Konstantinos Vergopoulos, Mark Niklas MĂźller, and Martin Vechev. Automated benchmark generation for repository-level coding tasks. arXiv preprint arXiv:2503.07701, 2025.
- Wang et al. (2025) Lilin Wang, Lucas Ramalho, Alan Celestino, Phuc Anthony Pham, Yu Liu, Umang Kumar Sinha, Andres Portillo, Onassis Osunwa, and Gabriel Maduekwe. Swe-bench++: A framework for the scalable generation of software engineering benchmarks from open-source repositories. arXiv preprint arXiv:2512.17419, 2025.
- Wang et al. (2024) Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024.
- Xiao et al. (2026) Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026.
- Xiao et al. (2013) Xusheng Xiao, Sihan Li, Tao Xie, and Nikolai Tillmann. Characteristic studies of loop problems for structural test generation via symbolic execution. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 246â256, 2013. doi: 10.1109/ASE.2013.6693084.
- Xie et al. (2024) Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. Codebenchgen: Creating scalable execution-based code generation benchmarks, 2024.
- Yang et al. (2025a) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
- Yang et al. (2024) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528â50652, 2024.
- Yang et al. (2025b) John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025b. URL https://arxiv.org/abs/2504.21798.
- Yuan et al. (2024) Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. Evaluating and improving chatgpt for unit test generation. Proc. ACM Softw. Eng., 1(FSE), July 2024. doi: 10.1145/3660783. URL https://doi.org/10.1145/3660783.
- Zan et al. (2025) Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: A multilingual benchmark for issue resolving, 2025. URL https://arxiv.org/abs/2504.02605.
- Zeng et al. (2026) Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yaxing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, et al. davinci-dev: Agent-native mid-training for software engineering. arXiv preprint arXiv:2601.18418, 2026.
- Zhang et al. (2025a) Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience. arXiv preprint arXiv:2510.08558, 2025a.
- Zhang et al. (2025b) Lei Zhang, Jiaxi Yang, Min Yang, Jian Yang, Mouxiang Chen, Jiajun Zhang, Zeyu Cui, Binyuan Hui, and Junyang Lin. Synthesizing software engineering data in a test-driven manner. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 76518â76540. PMLR, 13â19 Jul 2025b. URL https://proceedings.mlr.press/v267/zhang25cn.html.
- Zhang et al. (2026) Lei Zhang, Mouxiang Chen, Ruisheng Cao, Jiawei Chen, Fan Zhou, Yiheng Xu, Jiaxi Yang, Zeyao Ma, Liang Chen, Changwei Luo, Kai Zhang, Fan Yan, KaShun Shum, Jiajun Zhang, Zeyu Cui, Feng Hu, Junyang Lin, Binyuan Hui, and Min Yang. Megaflow: Large-scale distributed orchestration system for the agentic era, 2026. URL https://arxiv.org/abs/2601.07526.
- Zhang et al. (2025c) Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025c.