2602.02361

# SWE-Universe: Scale Real-World Verifiable Environments to Millions **Authors**: - Junyang Lin, Binyuan Hui (Qwen Team, Alibaba Group, Zhejiang University) > Work done during an internship at Alibaba Qwen. Abstract We propose SWE-Universe, a scalable and efficient framework for automatically constructing real-world software engineering (SWE) verifiable environments from GitHub pull requests (PRs). To overcome the prevalent challenges of automatic building, such as low production yield, weak verifiers, and prohibitive cost, our framework utilizes a building agent powered by an efficient custom-trained model. This agent employs iterative self-verification and in-loop hacking detection to ensure the reliable generation of high-fidelity, verifiable tasks. Using this method, we scale the number of real-world multilingual SWE environments to a million scale (807,693). We demonstrate the profound value of our environments through large-scale agentic mid-training and reinforcement learning. Finally, we applied this technique to Qwen3-Max-Thinking and achieved a score of 75.3% on SWE-Bench Verified. Our work provides both a critical resource and a robust methodology to advance the next generation of coding agents. <details> <summary>x1.png Details</summary>  ### Visual Description ## Bar Chart: Number of Real-World Verifiable SWE Instances ### Overview The image is a bar chart comparing the number of real-world verifiable SWE (Software Weakness Enumeration) instances across different benchmarks and models. The chart compares "Python-only" instances against "Multilingual" instances. The x-axis represents the different SWE benchmarks/models, and the y-axis represents the number of instances. ### Components/Axes * **Title:** Number of Real-World Verifiable SWE Instances * **X-axis:** Categorical labels for different SWE benchmarks/models: SWE-Bench, SWE-Gym, Multi-SWE-RL, SWE-rebench, DeepSeek-V3.2, CWM, MiMo-V2-Flash, SWE-Universe (Ours) * **Y-axis:** (Implicit) Number of instances. The values are displayed above each bar. * **Legend:** Located in the top-left corner. * Blue: Python-only * Orange: Multilingual ### Detailed Analysis The chart presents the number of verifiable SWE instances for each benchmark/model, separated by Python-only and Multilingual. * **SWE-Bench:** Python-only: 2,294 * **SWE-Gym:** Python-only: 2,438 * **Multi-SWE-RL:** Multilingual: 4,723 * **SWE-rebench:** Python-only: 21,000 * **DeepSeek-V3.2:** Python-only: 24,667 * **CWM:** Python-only: 35,000 * **MiMo-V2-Flash:** Multilingual: 90,000 * **SWE-Universe (Ours):** Multilingual: 807,693 ### Key Observations * SWE-Universe (Ours) has a significantly higher number of multilingual instances (807,693) compared to all other benchmarks/models. * The number of Python-only instances varies across different benchmarks, ranging from 2,294 (SWE-Bench) to 35,000 (CWM). * Multi-SWE-RL and MiMo-V2-Flash only have Multilingual instances reported. ### Interpretation The chart highlights the performance of different benchmarks/models in terms of the number of verifiable SWE instances. The SWE-Universe (Ours) model demonstrates a substantially higher number of multilingual instances, suggesting it is more effective at identifying weaknesses in multilingual code compared to the other benchmarks/models. The data also shows the distribution of Python-only instances across different benchmarks, providing insights into their performance on Python-specific code. The large difference in the number of instances between SWE-Universe and other models suggests a significant improvement or difference in methodology. </details> Figure 1: A comparison of the number of instances in real-world SWE instances. Our multilingual SWE-Universe is significantly larger than other recent multilingual efforts like MiMo-V2-Flash (Xiao et al., 2026), DeepSeek-V3.2 (DeepSeek-AI, 2025), and Multi-SWE-RL (Zan et al., 2025), as well as prominent Python-only datasets including SWE-rebench (Badertdinov et al., 2025), SWE-Gym (Pan et al., 2024), CWM (Copet et al., 2025), and SWE-Bench (Jimenez et al., 2024). 1 Introduction Training large language models (LLMs) (Yang et al., 2025a; Liu et al., 2025) as coding agents has gained substantial attention in software engineering (SWE). Progress in this direction is critically dependent on large-scale, high-quality environments with reliable verification signals. An ideal source for such data lies within the vast ecosystem of real-world software development, specifically public GitHub repositories where pull requests (PRs) are linked to corresponding issues. Each such PR defines a self-contained training environment: the issue serves as a clear problem statement, the code patch offers an expert reference solution, and the accompanying tests can be repurposed into a verifier for agent proposed fixes. This formulation effectively turns each PR into a “gym” for training and evaluating coding agents. Benchmarks such as SWE-bench (Jimenez et al., 2024) operationalized this pipeline by curating real-world issues and standardizing the evaluation, serving as a common testbed for subsequent work. However, scaling the construction of verifiable real-world SWE environments while maintaining diversity and reproducibility remains a major challenge. A vast body of work, including the original SWE-bench and its many derivatives and extensions (Pan et al., 2024; Badertdinov et al., 2025; Zhang et al., 2025c; Guo et al., 2025; Zeng et al., 2026; Wang et al., 2025; Aleithan et al., 2024; Copet et al., 2025) has primarily focused on Python. While this line of work leverages the low barrier to entry for Python environment configuration, it nonetheless restricts the development of agents with true cross-lingual generalization capabilities. Efforts to create multi-lingual datasets, such as Multi-SWE-bench (Zan et al., 2025) and SWE-PolyBench (Rashid et al., 2025), rely on labor-intensive manual environment setup and remain limited in scale. Despite recent efforts of industrial LLM providers that scale SWE instances to the $10^{4}-10^{5}$ magnitude (Xiao et al., 2026; DeepSeek-AI, 2025), the technical details are undisclosed. We argue that scaling the generation of such verifiable environments to the massive scale (e.g., $10^{6}$ ) hinges on overcoming three fundamental challenges: - Low Production Yield: The intricate and heterogeneous nature of real-world repositories with their complex dependencies, platform-specific configurations, and bespoke build toolchains, yielding a low conversion rate from repositories to runnable instances. This results in significant computational waste and makes large-scale generation impractical. - Weak Verifier: Issues, PR patches, and test suites exhibit substantial variance in quality. A naive extraction pipeline can produce low-fidelity instances, and can also admit verifier vulnerabilities that allow solutions to pass via shallow heuristics (e.g., string matching with grep) rather than compiling and executing the intended code. Such failure modes create spurious training signals and distort evaluation. - Prohibitive Cost and Inefficiency: Many existing pipelines rely on large, expensive LLMs to perform repository-specific reasoning for dependency resolution and build configuration. The resulting cost and latency per instance make massive-scale generation economically and operationally impractical. To systematically address these challenges, we introduce SWE-Universe, a scalable, reliable, and efficient framework for automatically constructing million-scale, real-world agentic software engineering environments. At its core is an autonomous building agent that, for each PR, synthesizes a self-contained executable environment together with an executable verifier. To mitigate low production yield, the agent performs self-verification in an iterative validation loop. Concretely, it repeatedly tests the generated verifier against both the buggy and the fixed repository states, diagnoses failure modes, and revises the build procedure accordingly. This process improves the build success rate from 82.6% to 94% on a held-out set. To address the weak verifier, we integrate an in-loop hacking detector that immediately flags and rejects superficial verifiers during the generation process, forcing the agent toward solutions that genuinely execute the code. To achieve high efficiency and low cost, we specially trained an efficient Qwen-Next-80B-A3B model with a Mixture-of-Experts (MoE) framework and hybrid attention. On our custom multi-lingual building benchmark, this model achieves a 78.44% success rate, surpassing top-tier proprietary models like Claude-Opus-4.5, while its efficient architecture dramatically reduces the latency and cost per build. Leveraging the SWE-Universe framework, we construct 807,693 multilingual, verifiable training instances sourced from over 52,000 unique GitHub repositories. To our knowledge, this dataset is currently the largest and most diverse collection of real-world software engineering tasks with executable verification. We further validate the immense value of this dataset through extensive experiments. First, in a mid-training phase, we show that continued training on our dataset significantly enhances a model’s performance on standard benchmarks like SWE-Bench Verified (Jimenez et al., 2024), proving its powerful generalization capabilities. Second, we demonstrate that the verifiers generated by our pipeline provide a stable and effective reward signal for Reinforcement Learning (RL). By applying to our flagship model, Qwen3-Max-Thinking (Qwen Team, 2026), we achieved a score of 75.3% on the SWE-Bench Verified. Together, these results validate SWE-Universe as a robust framework for creating large-scale, high-quality agentic training data, paving the way for the development of more capable and versatile coding agents for real-world applications. 2 Methodology: Scalability and Reliability <details> <summary>figures/env-build.png Details</summary>  ### Visual Description ## Diagram: Automated Bug Detection and Resolution Workflow ### Overview The image illustrates an automated workflow for detecting and resolving bugs in a software repository using a pull request system. The process involves a building and verifier agent, a hacking detector, and a verifier validation step, with a feedback loop for retrying failed validations. ### Components/Axes The diagram consists of the following key components: 1. **Pull Request:** Represents an incoming code change request. * "Real-world PR" * Files: sklearn, gradient\_boosting.py, helper.py, utils * "+20 -12" (Indicates lines added and removed) * Github logo 2. **Oracle Patch:** Represents a potential fix or modification to the code. 3. **Building & Verifier Agent:** Builds and prepares the code for verification. * Tools: bash, switch-to-bug, switch-to-resolved 4. **Repository Container:** Contains the codebase. * Codebase: sklearn/, examples/, README.rst, reqs.txt, setup.cfg, setup.py 5. **Hacking Detector:** Detects potential security vulnerabilities or bugs. * Checks: * `# evaluation.sh; pytest test_production.py` (Pass) * `# evaluation.sh; grep -q "keyword" bug.java` (Fail) 6. **Verifier Validation:** Validates the code changes. * switch-to-bug + TEST -> non-zero exitcode * switch-to-resolved + TEST -> 0 exitcode 7. **New Software Engineering Task:** Represents the final outcome of the process. * Components: problem statement (TXT), docker image, eval script (TEST) ### Detailed Analysis The workflow proceeds as follows: 1. A pull request is submitted. 2. An oracle patch is applied. 3. The Building & Verifier Agent builds the code using tools like bash and switches between bug and resolved versions. 4. The agent submits an evaluation script (`evaluation.sh`). 5. The Hacking Detector observes the code and runs checks. 6. The Verifier Validation step checks the code. * If the validation fails (non-zero exitcode), a retry mechanism is triggered. * If the validation passes (0 exitcode), the process moves to the New Software Engineering Task. 7. The agentic loop represents the iterative process of evaluation and refinement. 8. The Repository Container holds the codebase and related files. ### Key Observations * The diagram highlights the importance of automated testing and validation in software development. * The feedback loop ensures that failed validations are retried, improving the reliability of the process. * The Hacking Detector plays a crucial role in identifying potential security vulnerabilities. ### Interpretation The diagram illustrates a robust and automated system for bug detection and resolution. The system uses a combination of building, verification, and hacking detection to ensure the quality and security of the codebase. The iterative nature of the process, with the agentic loop and retry mechanism, allows for continuous improvement and refinement of the code. The ultimate goal is to produce a new software engineering task that is free of bugs and vulnerabilities. </details> Figure 2: Our SWE-Universe framework for scalable and reliable environment building. The pipeline is built around a building agent that proposes a verifier (evaluation.sh). Two key components ensure quality and yield: an in-loop Hacking Detector that preemptively rejects superficial scripts, and an Iterative Validation loop where the agent self-corrects based on feedback from testing its verifier against both buggy and resolved code states. Our primary goal is to develop a method for automated verifiable SWE environment construction that is both scalable and reliable. Scalability implies compatibility across diverse programming languages and build toolchains, as well as maximizing the build success rate to minimize computational waste. Reliability requires that the generated verifier (i.e., an evaluation.sh script) accurately provides a reward signal for a given patch, passing only when the SWE requirements are met. To this end, we design an autonomous building agent based on the popular mini-sweagent scaffold. Our key insight is that while directly verifying the success of a complex building is difficult, assessing the success of a test script is comparatively straightforward: one can simply run it to see if it distinguishes between the pre-patch (buggy) and post-patch (fixed) states, as adopted by DeepSeek-V3.2 (Liu et al., 2025). However, we argue that this condition alone is insufficient. A naive agent might generate a script that “hacks” the verification by using simple string matching to confirm the patch’s application, rather than actually executing the code. Such a script would correctly distinguish states but would fail to validate the environment or the behavioral correctness of the fix. Therefore, we establish a more robust acceptance criterion for a successfully built task: the verifier must not only correctly distinguish between states but must do so by genuinely executing the code under test. Our methodology, detailed below and sumarized in Figure ˜ 2, is designed to meet this twofold objective. 2.1 Key Designs for SWE-Universe PR Crawling and Patch Separation. The pipeline begins by sourcing a large corpus of issue-linked pull requests (PRs) from public GitHub repositories. To prevent data contamination and ensure fair evaluation, we meticulously filter out any PRs that overlap with known downstream benchmarks. For each valid PR, we employ a language model to analyze the code modifications and partition them into a test patch (containing test-related changes) and a fix patch (containing the source code fix). PRs lacking a discernible test component are discarded. Agent-based Environment Building. Following patch separation, our autonomous agent, equipped with a set of powerful tools, initiates the construction of the environment and its corresponding verifier. The process commences by applying the test patch to the repository. The agent is then tasked with generating a verifier script, designated as evaluation.sh, whose objective is to reliably distinguish between the repository’s states based solely on its return code. Guided by the information within the test patch, the agent has the flexibility to adopt one of two strategies: it can either directly invoke the unit tests described in the patch, or, in scenarios where those tests lack a straightforward execution entry point, it can author a new custom test from scratch. This standardized approach, centered on a universal bash interface and its integer return code, is a deliberate design choice. It decouples the verification logic from language-specific conventions (e.g., hardcoding a pytest or cargo run workflow), thereby maximizing the method’s generalizability and scalability across disparate projects and ecosystems. Toolset. We equip the agent with three tools: bash, switch-to-resolved, and switch-to-bug. - bash: A general-purpose shell for file manipulation, dependency installation, and script generation. - switch-to-resolved and switch-to-bug: A pair of tools that allow the agent to atomically apply or revert the fix patch, toggling the repository between its fixed and buggy states. These state-switching tools are fundamental to our approach, as they empower the agent to perform self-verification. By testing its own actions in a closed loop, the agent can diagnose and recover from failures, which is key to improving the overall build success rate and achieving scalability. Iterative Validation. To ensure the verifier is reliable, the agent engages in an iterative validation loop. After the agent ends with submitting a candidate evaluation.sh script, we execute it under both repository states using its switching tools. A script is considered functionally correct only if it fails (exits with a non-zero status) in the buggy state and succeeds (exits with a zero status) in the fixed state. If the script fails this validation, the agent receives this negative feedback, discards the faulty script, and is prompted to generate a new, revised version until reaches the maximum turns we predefined (e.g., 100 turns). This iterative process substantially improves reliability: for a set of held-out PRs, the environment-building success rate increases from 82.6% to 94%. In-loop Hacking Detection. To meet our second criterion—that the verifier must genuinely execute the code—we integrate a Hacking Detector directly into the agent’s work cycle. This module uses an LLM to inspect the generated evaluation.sh script for “hacking” patterns, such as using grep or other string-matching utilities to check the contents of source files instead of running a build or test command. Critically, this check is performed within the agent’s loop, not as a post-processing step. If the detector flags a script as a “hack”, this attempt is immediately considered a failure. This provides timely feedback to the agent, forcing it to abandon superficial strategies and guiding it toward a valid solution that involves actual code execution. This in-loop design improves both the reliability of the final artifact and the efficiency of the agent’s search process. <details> <summary>x2.png Details</summary>  ### Visual Description ## Code Snippets: Testing and Hacking ### Overview The image presents three code snippets, each representing a different approach to testing or debugging software. The snippets are displayed as if in a terminal window with a dark background and colored syntax highlighting. The three approaches are: (1) Executing Human-written unit test, (2) Executing LLM-written tests, and (3) Hacking: using static checking. ### Components/Axes **Snippet 1: Executing Human-written unit test** * Language: Shell script (Bash) * Commands: * `set -e` * `cd /testbed` * `pnpm install` (if node_modules is not present) * `cat > jest-setup.cjs << JEST_SETUP` (creates a CommonJS version of jest-setup.js) * Defines a server object with `listen`, `resetHandlers`, and `close` methods. * Calls `beforeAll`, `afterEach`, and `afterAll` to manage the server lifecycle. * `pnpm exec jest --config=jest.config.temp.js --coverage=false --no-cache 2>&1 | tee test_output.log` (runs the tests and captures the output) * `grep -q "Property 'persistUserToken' does not exist" test_output.log` (checks for a specific error pattern) * Prints "Bug detected: persistUserToken function is missing" if the error pattern is found. * Prints "Tests passed or other error occurred" if the error pattern is not found. **Snippet 2: Executing LLM-written tests** * Language: Python * Imports: `sys`, `glyphsLib.builder.hints`, `glyphsLib.classes` * Classes: `MockUGlyph`, `MockSelf` * Functions: `to_glyphs_hints` * Logic: * Defines classes to mock glyph-related objects. * Calls `to_glyphs_hints` with mocked objects. * Prints "SUCCESS: No error occurred - bug appears to be fixed" if no exception occurs. * Prints "ERROR: (type(e).__name__}: {e}" if an exception occurs. **Snippet 3: Hacking: using static checking** * Language: Shell script (Bash) * Commands: * `export JAVA_HOME=/opt/jdk-17.0.2` * `export PATH=$JAVA_HOME/bin:$PATH` * `echo "Testing watchOnTermination functionality by checking source code changes..."` * `grep -q "var failAction: Option\[Watch.Action\]" /testbed/main/src/main/scala/sbt/internal/Continuous.scala` (checks for the presence of "failAction" field) * Prints "Found failAction field - fix is present" if the field is found. * `grep -q "exitWatchShared.*" /testbed/main/src/main/scala/sbt/internal/Continuous.scala` (checks for the presence of "exitWatchShared" function) * Prints "Found exitWatchShared refactoring - fix is complete" if the function is found. * Prints "exitWatchShared not refactored - fix is incomplete" if the function is not found. * Prints "failAction field not found - bug is present" if the field is not found. ### Detailed Analysis or ### Content Details **Snippet 1:** * The script sets up a testing environment for a JavaScript project, installs dependencies, and runs Jest tests. It then checks the output for a specific error related to a missing `persistUserToken` function. **Snippet 2:** * This snippet uses Python to test glyph-related functionality. It mocks glyph objects and calls a function (`to_glyphs_hints`) to check for errors. The script handles exceptions and prints success or error messages accordingly. **Snippet 3:** * This script uses static checking to verify the presence of specific code elements in a Scala file. It checks for the `failAction` field and the `exitWatchShared` function, printing messages indicating whether the expected elements are found. ### Key Observations * The three snippets demonstrate different approaches to software testing and debugging. * Snippet 1 uses human-written unit tests to check for specific errors in a JavaScript project. * Snippet 2 uses LLM-written tests to check glyph-related functionality in Python. * Snippet 3 uses static checking to verify the presence of specific code elements in a Scala file. ### Interpretation The image illustrates a range of techniques used in software development to ensure code quality and identify bugs. The human-written unit test focuses on specific error patterns, while the LLM-written test uses a more general approach to check for exceptions. The static checking script verifies the presence of specific code elements, providing a quick way to ensure that certain features are implemented correctly. The combination of these approaches can help developers catch a wide range of errors and improve the overall quality of their code. </details> Figure 3: Three types of evaluation.sh. We only accept the first two types of evaluation scripts. 2.2 Further Analysis Case Study We examine three representative LLM-generated test scripts to illustrate our verification criteria (Figure 3). Case 1 tests a JavaScript authentication bug by executing existing human-written unit tests in the Jest framework via pnpm exec jest, checking for specific error patterns in the output. Case 2 validates a Python glyphsLib bug by creating LLM-generated mock objects and unit tests that directly exercise the buggy code path and verify execution without exceptions. Case 3 attempts to verify a Scala bug fix using static pattern matching with grep commands to detect the presence of specific code structures, which is flagged as “hacking” by our hacking detection. We accept Cases 1 and 2 as they employ strong verification through executable tests that validate runtime behavior, while we reject Case 3 because static pattern matching cannot guarantee correctness—a patch may introduce expected code patterns while still containing logical errors. Quality Analysis We find that the resulting data still exhibit several quality issues: (1) some task descriptions are ambiguous or incomplete; (2) certain Docker environments do not fully match the stated requirements; and (3) some unit tests are misaligned with the task descriptions, which can lead to false positives or false negatives. To quantify and mitigate this problem, we develop a quality-judge agent. It takes as input the task description, Docker environment, test scripts, and optionally the ground-truth patch, and automatically evaluates the task quality. On a human-labeled quality-judging benchmark, the agent reaches 78.72% accuracy. After applying it to the entire dataset, as shown in Figure 4, we find our dataset matches SWE-Rebench (Badertdinov et al., 2025) in quality while providing 38× more instances. <details> <summary>x3.png Details</summary>  ### Visual Description ## Scatter Plot: Dataset Size vs. Quality Score ### Overview The image is a scatter plot comparing the dataset size (count) on a logarithmic scale to the quality score (percentage) for four different datasets: SWE-Universe, SWE-rebench, SWE-Gym, and SWE-bench Verified. The plot shows the relationship between the size of the dataset and its quality score. ### Components/Axes * **X-axis:** Quality Score (%), ranging from 50.0% to 100.0% with tick marks at 50.0, 62.5, 75.0, 87.5, and 100.0. * **Y-axis:** Dataset Size (count), on a logarithmic scale ranging from 10^2 to 10^7. Tick marks are present at each power of 10. * **Data Points:** Four data points, each representing a dataset. * SWE-Universe (Blue) * SWE-rebench (Orange) * SWE-Gym (Brown) * SWE-bench Verified (Green) * **Gridlines:** Vertical and horizontal gridlines are present, corresponding to the tick marks on both axes. ### Detailed Analysis Here's a breakdown of the data points: * **SWE-Universe (Blue):** Located at approximately (60%, 10^6). Quality Score is approximately 60%. Dataset Size is approximately 1,000,000. * **SWE-rebench (Orange):** Located at approximately (61%, 10^4). Quality Score is approximately 61%. Dataset Size is approximately 10,000. * **SWE-Gym (Brown):** Located at approximately (75%, 10^3). Quality Score is approximately 75%. Dataset Size is approximately 1,000. * **SWE-bench Verified (Green):** Located at approximately (93%, 10^2). Quality Score is approximately 93%. Dataset Size is approximately 100. ### Key Observations * There is a general trend of decreasing dataset size as the quality score increases. * SWE-Universe has the largest dataset size but the lowest quality score among the four datasets. * SWE-bench Verified has the highest quality score but the smallest dataset size. * SWE-rebench and SWE-Gym fall in between, with SWE-rebench having a larger dataset size and slightly lower quality score than SWE-Gym. ### Interpretation The scatter plot suggests an inverse relationship between dataset size and quality score for these four datasets. This could indicate that as more effort is put into verifying and improving the quality of the data, the size of the resulting dataset tends to decrease. This could be due to the removal of noisy or incorrect data points during the verification process. The SWE-Universe dataset, with its large size and relatively low quality score, might represent a dataset collected with less stringent quality control measures. Conversely, SWE-bench Verified, with its small size and high quality score, likely represents a dataset that has undergone rigorous verification. </details> Figure 4: Task Quality vs. Dataset Size (Log-Scale). Task quality is measured as the fraction of high-quality samples. 3 Efficient Building and Benchmarking To optimize both the performance and throughput of our pipeline, we developed a high-capacity yet efficient model and a comprehensive cross-lingual benchmark to evaluate its capabilities. Model Training and Deployment To make our end-to-end pipeline efficient at scale, we train a lightweight but strong builder model, Qwen-Next-80A3 (Qwen Team, 2025b), a mixture-of-experts (MoE) model with hybrid attentions including linear attentions and full attentions. The model was trained using rejection sampling on a collection of high-quality building trajectories. This process involved sampling multiple candidate paths for environment construction and filtering for successful, non-hacked outcomes to serve as training data. The model serves as the unified backbone for all tasks in our pipeline, including PR patch splitting, iterative environment building, and the hacking detector. Its efficient architecture allows us to scale the building process across thousands of repositories with significantly lower latency compared to dense models of similar performance. Benchmark To rigorously assess the reliability of automated environment construction, we constructed a diverse benchmark consisting of 320 pull requests randomly sampled from GitHub. We remove the repositories used in the training trajectories from the benchmark. To ensure broad representativeness, we selected 40 PRs for each of the eight language categories: Python, JavaScript/TypeScript, Go, Java, Rust, C/C++, C#, and an “Others” category (including all the other languages such as PHP and Kotlin). We define two primary success metrics: Success Rate (w/o Hack), which requires the verifier to be functionally correct and pass the hacking detector, and Success Rate (w/ Hack), which includes all scripts that distinguish the bug regardless of the detection outcome. Table 1: Benchmark results for automated environment building across various models. “Success (%) (w/o Hack)” measures the rate of creating a valid, non-hacked verifiable environment, while “Success (%) (w/ Hack)” also counts “hacked” verifiers as successful builds. Our model, Qwen-Next-80A3, achieves the highest non-hacking success rate. | Model | Success (%) | Success (%) | Success Rate by Language (%) | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | (w/o Hack) | (w/ Hack) | C/C++ | C# | Go | Java | JS/TS | Other | Python | Rust | | | Qwen-Next-80A3 (Ours) | $78.44$ | $82.50$ | $57.50$ | $70.00$ | $80.49$ | $82.50$ | $84.62$ | $83.72$ | $85.37$ | $83.33$ | | Claude-Opus-4.5 | $77.81$ | $85.00$ | $52.50$ | $57.50$ | $82.93$ | $77.50$ | $89.74$ | $76.74$ | $95.12$ | $91.67$ | | Claude-Sonnet-4 | $75.62$ | $85.62$ | $67.50$ | $52.50$ | $75.61$ | $72.50$ | $84.62$ | $83.72$ | $82.93$ | $86.11$ | | Gemini-3-Pro | $69.69$ | $72.50$ | $32.50$ | $57.50$ | $73.17$ | $72.50$ | $87.18$ | $76.74$ | $87.80$ | $69.44$ | | Claude-Sonnet-4-5 | $66.88$ | $71.56$ | $30.00$ | $50.00$ | $63.41$ | $67.50$ | $87.18$ | $67.44$ | $92.68$ | $77.78$ | | GLM-4.7 | $58.44$ | $64.06$ | $37.50$ | $50.00$ | $53.66$ | $57.50$ | $66.67$ | $65.12$ | $73.17$ | $63.89$ | | MiniMax-M2.1 | $54.69$ | $61.88$ | $22.50$ | $35.00$ | $56.10$ | $55.00$ | $74.36$ | $65.12$ | $73.17$ | $55.56$ | | DeepSeek-v3.2 | $54.06$ | $59.38$ | $15.00$ | $50.00$ | $63.41$ | $47.50$ | $53.85$ | $48.84$ | $78.05$ | $77.78$ | | Qwen3-Coder-480B | $48.75$ | $55.62$ | $35.00$ | $32.50$ | $39.02$ | $45.00$ | $48.72$ | $69.77$ | $68.29$ | $50.00$ | Result Analysis Our evaluation, detailed in Table 1, establishes Qwen-Next-80A3 as the new state-of-the-art model for automated environment building, achieving a 78.44% success rate that surpasses even strong proprietary models like Claude-Opus-4.5 (77.81%). This superior performance extends beyond raw success to overall reliability. We observe a significant gap between the true success rate and the one including “hacked” verifiers for most general-purpose models (e.g., over 7% for Claude-Opus-4.5), indicating their tendency to find superficial shortcuts. In stark contrast, our model exhibits the smallest gap among top performers (4.06%), demonstrating that our task-specific fine-tuning on collected trajectories effectively discourages these deceptive behaviors. Furthermore, while performance varies by language—with C/C++ proving most challenging for all agents—our model demonstrates more consistent and robust capabilities across the entire linguistic spectrum compared to competitors that show strength only in specific ecosystems. Ultimately, these results validate that a specialized, efficiently trained MoE model not only achieves higher success but also operates with greater fidelity than larger, general-purpose counterparts in this complex domain. 4 Scaling Environments to Millions In this section, we describe our efforts to scale the environment-building pipeline to a massive corpus of real-world software changes, moving beyond small-scale benchmarks to a million-scale dataset of executable tasks. Large-scale Data Curation We harvested a comprehensive dataset of approximately 33.3 million pull requests (PRs) spanning the most recent five years (2021–2025) of GitHub’s history. To extract high-quality tasks from this raw pool, we applied a series of rigorous heuristic filters. Specifically, we removed PRs with excessive file changes or line counts to avoid overly complex or noisy tasks, and discarded any entries that did not contain a discernible test patch. Furthermore, we prioritized PRs that were explicitly linked to at least one GitHub issue, as these provide the most reliable ground-truth problem statements. This filtering process resulted in a candidate set of approximately 1 million high-quality PRs. Infrastructure for Large-scale Rollouts To support our massive data generation effort, we implement our pipeline on top of MegaFlow (Zhang et al., 2026), a distributed execution system for orchestrating large numbers of long-running agentic jobs. MegaFlow achieves massive parallelism by dispatching each environment-building task as an independent job to a dedicated Alibaba Cloud Elastic Compute Service (ECS) instance. Within this sandboxed virtual machine, our agent executes the entire build process, culminating in a verified Docker image. Upon completion, successful images are pushed to Alibaba Cloud’s Container Registry (ACR) https://www.alibabacloud.com/en/product/container-registry, where we leverage Docker’s layer caching to significantly reduce storage costs by reusing common base layers. This highly parallel and resource-efficient architecture was instrumental in processing millions of pull requests concurrently, enabling the practical construction of our million-scale dataset. Massive Production with Qwen-Next-80A3 We deployed our fine-tuned Qwen-Next-80A3 model to execute the building and verification process for the filtered candidates. Leveraging the model’s efficiency and the iterative agentic loop described in previous sections, we achieved a non-hacked success rate of 75.9%. This process ultimately yielded 717,122 executable, high-fidelity environments. These tasks are primarily focused on assessing an agent’s ability to resolve real-world software issues. Additionally, to expand the dataset’s coverage, we selected a subset of 2025 PRs that were not linked to issues, utilizing the PR titles and descriptions as the problem statements to synthesize an additional environments, and producing 90,571 environments. Table 2: Data statistics for our building environments. | Language | Instances | Repos | Inst/Repo (Avg) | Avg Lines of evaluation.sh | | --- | --- | --- | --- | --- | | Python | 202,302 | 13,098 | 15.45 | 25.01 | | Javascript / Typescript | 175,660 | 11,604 | 15.14 | 27.41 | | Go | 121,062 | 5,554 | 21.80 | 28.87 | | Java | 86,105 | 4,700 | 18.32 | 24.75 | | Rust | 74,180 | 4,445 | 16.69 | 19.31 | | C / C++ | 37,228 | 3,405 | 10.93 | 45.78 | | C# | 24,387 | 1,929 | 12.64 | 31.84 | | Others | 86,769 | 8,225 | 10.55 | 38.89 | | # Total | 807,693 | 52,960 | 15.25 | 28.21 | Statistical Analysis The statistics presented in Table 2 highlight both the scale and diversity of our generated dataset. With 807,693 instances spread across 52,960 unique repositories, this dataset represents an unprecedented resource for training and evaluating software engineering agents. The language distribution largely mirrors the current open-source landscape, with Python and JavaScript/TypeScript constituting the largest shares. Notably, Go repositories exhibit the highest average number of instances per repository (21.80), suggesting a high density of verifiable, issue-linked PRs within its ecosystem, possibly due to strong conventions around testing and development. The “Avg Eval Lines” metric reveals interesting insights into the complexity of verification across languages. C/C++ instances, as expected, require the longest verifier scripts on average (45.78 lines), reflecting the typical complexity and boilerplate of their build systems. In contrast, Rust’s concise average (19.31 lines) may point to the efficiency and standardization of its cargo toolchain, allowing for more succinct test execution commands. Overall, the dataset provides a rich and varied testbed, capturing a wide array of real-world challenges for coding agents. 5 Evaluation: Large-scale Agentic Training We now turn to leveraging this resource for large-scale agentic model training. Our goal is to demonstrate that training on this diverse, real-world data can significantly enhance a model’s capabilities as a software engineering agent. 5.1 Mid-training We hypothesize that intermediate training on a vast corpus of high-quality agentic trajectories can endow a model with a strong foundation in both software engineering problem-solving. This “mid-training” phase is designed to bridge the gap between the pre-training and post-training on downstream agentic tasks. Setup Our mid-training process begins by generating a massive dataset of agentic trajectories. We first employed the Qwen3-Coder-480B-A30B (Qwen Team, 2025a) model to interact with the environments we previously constructed. To ensure a diversity of problem-solving strategies, we conducted these rollouts across five different agentic scaffolds: SWE-agent (Yang et al., 2024), Mini-SWE-agent (SWE-agent Team, 2025), OpenHands (Wang et al., 2024), Claude-Code (Anthropic, 2024), Qwen-Code (Qwen Team, 2025c). The entire rollout process was orchestrated by our MegaFlow system. For each environment, we performed rejection sampling: a trajectory is deemed successful and retained only if the final generated code passes the corresponding evaluation.sh script and clears an additional in-house quality filter. This rigorous filtering process yielded a high-quality dataset of 500K successful trajectories, comprising a total of 30 billion training tokens. We then used this data to conduct intermediate training on a Qwen3-Next-80A3 model, using 256K sequence length and Best-Fit packing (Ding et al., 2024). Critically, we applied no loss mask during this training phase. This strategy ensures that the model learns not only to predict agentic actions but also to internalize the vast amount of code and natural language in the agent’s observations and responses, developing it into a more comprehensive “coding world model” (Copet et al., 2025; Zhang et al., 2025a). Scaling Trends To analyze the effectiveness of our large-scale agentic mid-training, we evaluated model checkpoints on two widely used agentic coding benchmarks: SWE-Bench Verified (primarily Python-based) and the more diverse SWE-Bench Multilingual. The performance evolution throughout the training process is depicted in Figure 5(a). The results demonstrate a clear and positive scaling trend, confirming that our mid-training on the real-world instances successfully transfers to standard evaluation benchmarks. On the standard SWE-Bench Verified, the model’s performance exhibits a steady climb, starting from an already strong baseline of 50.3% and consistently improving to a final score of over 61% after 2,000 training steps. More notably, the trend on the more challenging SWE-Bench Multilingual benchmark shows even more dramatic improvement. Starting from a significantly lower baseline of approximately 31%, the model’s performance surges to over 46% by the end of the training. This substantial gain of over 15 percentage points underscores the critical value of our dataset’s linguistic diversity. It proves that training on a massive, diverse corpus of real-world, multi-language software issues is essential for creating agents that can generalize beyond a single programming ecosystem. 5.2 Agentic Reinforcement Learning Beyond training via rejection sampling, our collection of executable environments is suited for Reinforcement Learning (RL), where the binary pass/fail signal from the evaluation.sh script serves as a direct and reliable reward. To demonstrate this, we conducted agentic RL experiments on different models with different sizes and structures. Setup We validate the effectiveness of the data on agentic reinforcement learning. Prior to training, we perform rollouts with the base model to filter out queries that are either too difficult or too easy. During training, we set the maximum number of interaction turns to 200 and the context length to 128k. The training process is powered by our asynchronous RL framework, which natively supports agentic workflows. This architecture mitigates data skewness while facilitating seamless multi-turn interactions without framework-induced overhead, achieving a 2x–4x speedup compared to existing RL infrastructures. <details> <summary>x4.png Details</summary>  ### Visual Description ## Line Chart: Performance vs. Training Tokens ### Overview The image is a line chart comparing the performance of two models, "SWE-Bench Verified" and "SWE-Bench Multilingual," against the number of training tokens (in billions). The chart shows how performance (in percentage) changes as the models are trained with more data. ### Components/Axes * **X-axis:** Training Tokens (Billion). Scale ranges from approximately 5 to 70, with markers at 10, 20, 30, 40, 50, 60, and 70. * **Y-axis:** Performance (%). Scale ranges from 30 to 60, with markers at 30, 35, 40, 45, 50, 55, and 60. * **Legend:** Located in the bottom-right of the chart. * Blue line: SWE-Bench Verified * Orange line: SWE-Bench Multilingual ### Detailed Analysis * **SWE-Bench Verified (Blue Line):** * Trend: Generally slopes upward, indicating improved performance with more training tokens. * Data Points: * At 5 Billion Tokens: ~50.5% * At 10 Billion Tokens: ~52.5% * At 20 Billion Tokens: ~52.7% * At 30 Billion Tokens: ~52.3% * At 40 Billion Tokens: ~54.2% * At 50 Billion Tokens: ~56.0% * At 60 Billion Tokens: ~58.5% * At 65 Billion Tokens: ~58.5% * At 70 Billion Tokens: ~61.2% * **SWE-Bench Multilingual (Orange Line):** * Trend: Initially slopes upward, then fluctuates, but generally increases overall. * Data Points: * At 5 Billion Tokens: ~31.0% * At 10 Billion Tokens: ~38.5% * At 20 Billion Tokens: ~39.5% * At 30 Billion Tokens: ~41.2% * At 40 Billion Tokens: ~40.2% * At 50 Billion Tokens: ~44.0% * At 60 Billion Tokens: ~40.5% * At 65 Billion Tokens: ~40.0% * At 70 Billion Tokens: ~46.5% ### Key Observations * The "SWE-Bench Verified" model consistently outperforms the "SWE-Bench Multilingual" model across all training token values. * Both models show improvement in performance as the number of training tokens increases, but the "SWE-Bench Verified" model demonstrates a more stable and consistent upward trend. * The "SWE-Bench Multilingual" model shows more fluctuation in performance, particularly between 40 and 65 billion training tokens. ### Interpretation The data suggests that the "SWE-Bench Verified" model is more effective or better suited for the task being measured, as it consistently achieves higher performance levels than the "SWE-Bench Multilingual" model. The upward trends indicate that both models benefit from increased training data, but the "SWE-Bench Verified" model appears to leverage the data more efficiently. The fluctuations in the "SWE-Bench Multilingual" model's performance could indicate sensitivity to specific data subsets or potential overfitting issues. The performance gap between the two models widens as the number of training tokens increases, suggesting that the "SWE-Bench Verified" model scales better with more data. </details> (a) <details> <summary>x5.png Details</summary>  ### Visual Description ## Line Chart: SWE-Bench Multilingual Performance vs. Training Steps ### Overview The image is a line chart showing the performance of a SWE-Bench Multilingual model as a function of training steps. The y-axis represents the SWE-Bench Multilingual score in percentage, and the x-axis represents the training steps. The chart displays a single data series in orange, showing how the model's performance changes over the course of training. ### Components/Axes * **Y-axis:** "SWE-Bench Multilingual (%)". The scale ranges from 32 to 42, with gridlines at intervals of 2. * **X-axis:** "Training Steps". The x-axis is not explicitly scaled, but it represents an increasing number of training steps. * **Data Series:** A single orange line represents the model's performance. The final data point is marked with an orange star and the value "42.0". ### Detailed Analysis The orange line represents the SWE-Bench Multilingual performance. * **Initial Performance:** The line starts at approximately 32.3%. * **Initial Increase:** The line increases sharply to approximately 37.7%. * **Plateau/Decrease:** The line then plateaus and decreases slightly to approximately 37.3%, then decreases further to approximately 36.0%, and then decreases again to approximately 35.0%. * **Final Increase:** The line then increases to approximately 37.0%, then increases sharply to approximately 40.0%, and finally reaches 42.0%, marked with a star. ### Key Observations * The model's performance initially increases rapidly with training. * There is a period of stagnation or slight decrease in performance after the initial increase. * The model's performance increases significantly towards the end of the training steps. * The peak performance is 42.0%, indicated by the star. ### Interpretation The chart illustrates the training progress of a SWE-Bench Multilingual model. The initial rapid increase suggests that the model quickly learns the basic patterns in the data. The subsequent plateau or decrease could be due to overfitting or the model getting stuck in a local minimum. The final significant increase indicates that the model eventually overcomes these challenges and achieves a higher level of performance. The star at 42.0% highlights the best performance achieved during the training process. The data suggests that continued training, perhaps with adjustments to the learning rate or model architecture, could potentially lead to even better results. </details> (b) Figure 5: (a) Performance scaling trends of the Qwen3-Next-80A3 model during mid-training. (b) Reinforcement learning curve of the Qwen3-30B-A3B for SWE-Bench Multilingual. Results on Qwen3-30B-A3B The training curve in Figure 5(b) shows the effectiveness of agentic reinforcement training. The Qwen3-30a3 model shows a remarkable improvement on SWE-Bench Multilingual, surging from a baseline of approximately 32% to a peak of 42.0%. This substantial 10-point absolute gain underscores the power of RL to enhance generalization across diverse, multilingual tasks. Applied to Qwen3-Max-Thinking To push the boundaries of agentic coding performance, we applied our methodology to our flagship Qwen3-Max-Thinking model. The resulting model achieved a performance of 75.3% on SWE-Bench Verified. This result validates the effectiveness of our large-scale data generation pipeline at a production level, showcasing its ability to elevate state-of-the-art models. 6 Related Work Agentic and Synthetic Environment Generation A significant line of research focuses on generating software engineering tasks synthetically to create scalable training data, often without relying on authentic historical issues. For instance, SWE-smith (Yang et al., 2025b) procedurally generates tasks by artificially injecting bugs, while SWE-Flow (Zhang et al., 2025b) leverages test documentation to synthesize novel problems. Other works generate complex bugs from scratch (Sonwane et al., 2025) or create interactive tasks from bug reports (Jin et al., 2023). The scope of synthesis extends to generating command-line tasks from natural language (Lin et al., 2018; Gandhi et al., 2026). Similarly, RepoST (Xie et al., 2024) employs synthetic tests to generate training data. While these methods provide scalable training data, they do not fully capture the complexity and long-tail challenges of real-world software issues. In contrast, our work focuses exclusively on building executable environments for real-world software engineering problems derived directly from public repositories. Real-World Software Environment Setup Much recent work focuses purely on the environment configuration capability itself, proposing scripted, agentic, or expert-driven methods to create executable environments, but without providing verifiers for specific software issues (Hu et al., 2025; Milliken et al., 2025; Bouzenia & Pradel, 2025; Horton & Parnin, 2019; Jain et al., 2024; Vergopoulos et al., 2025; Guo et al., 2026; Arora et al., 2025; Eliseeva et al., 2025; Kuang et al., 2025; Fu et al., 2025). Another line of work provides end-to-end verification for real software issues. The seminal SWE-bench (Jimenez et al., 2024) established this paradigm with over 2,400 real-world Python issues, each with a test script to verify a fix. This Python-centric approach was significantly scaled by SWE-rebench (Badertdinov et al., 2025), which developed a fully automated pipeline to generate over 21,000 verifiable tasks. Other works like SWE-Gym (Pan et al., 2024), SWE-bench-Live (Zhang et al., 2025c), SWE-Factory (Guo et al., 2025), daVinci-Dev (Zeng et al., 2026), SWE-Bench++ (Wang et al., 2025), and SWE-bench+ (Aleithan et al., 2024) have also aimed to expand data generation, primarily within the Python ecosystem. A few recent efforts have created multi-language benchmarks, such as Multi-SWE-bench (Zan et al., 2025) and SWE-PolyBench (Rashid et al., 2025); however, these have generally been limited in scale. Our work systematically overcomes these limitations by presenting a highly scalable and reliable pipeline that operates across arbitrary languages and repositories, successfully generating over million-scale executable environments with a validated, task-specific verifier. Building Code Verifier Automated code verifiers are typically constructed by executing test suites to validate solutions. Traditional test generation methods, including probability-based approaches (Pacheco et al., 2007), constraint-based (Xiao et al., 2013), and search-based (Harman & McMinn, 2010; Lukasczyk & Fraser, 2022), often suffer from limited coverage and poor readability, and are typically restricted to regression or implicit oracles (Barr et al., 2015). Recent work leverages LLMs to generate unit tests for verification (Alagarsamy et al., 2024; Chen et al., 2024b; Schäfer et al., 2024; Yuan et al., 2024; Chen et al., 2024a), though the resulting tests can be unreliable due to confidently incorrect assertions. CodeRM (Ma et al., 2025) improves reward signal quality by scaling the number of generated tests and dynamically adapting test counts to problem difficulty. In this work, we leverage a building agent to automatically generate verifier for complext repository-level software engineering issues. 7 Conclusion In this paper, we introduced SWE-Universe, a scalable framework designed to overcome the critical bottlenecks of low yield, inconsistent quality, and prohibitive cost in generating real-world software engineering environments. Our autonomous building agent, powered by a custom-trained model and equipped with iterative self-verification and in-loop hacking detection, successfully constructed a massive dataset of over 800,000 executable, multi-lingual tasks—the largest of its kind. We demonstrated the profound value of this resource through large-scale agentic training, showing that it provides a powerful signal for both supervised learning and reinforcement learning, ultimately enabling our Qwen3-Max-Thinking model to achieve a score of 75.3% on SWE-Bench Verified. References - Alagarsamy et al. (2024) Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. A3test: Assertion-augmented automated test case generation. Inf. Softw. Technol., 176(C), November 2024. ISSN 0950-5849. doi: 10.1016/j.infsof.2024.107565. URL https://doi.org/10.1016/j.infsof.2024.107565. - Aleithan et al. (2024) Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992, 2024. - Anthropic (2024) Anthropic. Claude code: An agentic coding assistant. https://github.com/anthropics/claude-code, 2024. Accessed: 2026-01-22. - Arora et al. (2025) Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing software engineering agents’ ability to bootstrap development environments. arXiv preprint arXiv:2507.09063, 2025. - Badertdinov et al. (2025) Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents, 2025. URL https://arxiv.org/abs/2505.20411. - Barr et al. (2015) Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507–525, 2015. doi: 10.1109/TSE.2014.2372785. - Bouzenia & Pradel (2025) Islem Bouzenia and Michael Pradel. You name it, i run it: An llm agent to execute tests of arbitrary projects. Proc. ACM Softw. Eng., 2(ISSTA), June 2025. doi: 10.1145/3728922. URL https://doi.org/10.1145/3728922. - Chen et al. (2024a) Mouxiang Chen, Zhongxin Liu, He Tao, Yusu Hong, David Lo, Xin Xia, and Jianling Sun. B4: Towards optimal assessment of plausible code solutions with plausible tests. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE ’24, pp. 1693–1705, New York, NY, USA, 2024a. Association for Computing Machinery. ISBN 9798400712487. doi: 10.1145/3691620.3695536. URL https://doi.org/10.1145/3691620.3695536. - Chen et al. (2024b) Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pp. 572–576, New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400706585. doi: 10.1145/3663529.3663801. URL https://doi.org/10.1145/3663529.3663801. - Copet et al. (2025) Jade Copet, Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, et al. Cwm: An open-weights llm for research on code generation with world models. arXiv preprint arXiv:2510.02387, 2025. - DeepSeek-AI (2025) DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models. CoRR, abs/2512.02556, 2025. doi: 10.48550/ARXIV.2512.02556. URL https://doi.org/10.48550/arXiv.2512.02556. - Ding et al. (2024) Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. Fewer truncations improve language modeling. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=kRxCDDFNpp. - Eliseeva et al. (2025) Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. Envbench: A benchmark for automated environment setup. arXiv preprint arXiv:2503.14443, 2025. - Fu et al. (2025) Kelin Fu, Tianyu Liu, Zeyu Shang, Yingwei Ma, Jian Yang, Jiaheng Liu, and Kaigui Bian. Multi-docker-eval: Ashovel of the gold rush’benchmark on automatic environment building for software engineering. arXiv preprint arXiv:2512.06915, 2025. - Gandhi et al. (2026) Kanishk Gandhi, Shivam Garg, Noah D Goodman, and Dimitris Papailiopoulos. Endless terminals: Scaling rl environments for terminal agents. arXiv preprint arXiv:2601.16443, 2026. - Guo et al. (2025) Lianghong Guo, Yanlin Wang, Caihua Li, Pengyu Yang, Jiachi Chen, Wei Tao, Yingtian Zou, Duyu Tang, and Zibin Zheng. Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks. arXiv preprint arXiv:2506.10954, 2025. - Guo et al. (2026) Xinshuai Guo, Jiayi Kuang, Linyue Pan, Yinghui Li, Yangning Li, Hai-Tao Zheng, Ying Shen, Di Yin, and Xing Sun. Evoconfig: Self-evolving multi-agent systems for efficient autonomous environment configuration. arXiv preprint arXiv:2601.16489, 2026. - Harman & McMinn (2010) Mark Harman and Phil McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering, 36(2):226–247, 2010. doi: 10.1109/TSE.2009.71. - Horton & Parnin (2019) Eric Horton and Chris Parnin. Dockerizeme: Automatic inference of environment dependencies for python code snippets. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 328–338. IEEE, 2019. - Hu et al. (2025) Ruida Hu, Chao Peng, Xinchen Wang, Junjielong Xu, and Cuiyun Gao. Repo2run: Automated building executable environment for code repository at scale. arXiv preprint arXiv:2502.13681, 2025. - Jain et al. (2024) Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: turning any github repository into a programming agent environment. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. - Jimenez et al. (2024) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. - Jin et al. (2023) Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. Inferfix: End-to-end program repair with llms. In Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering, pp. 1646–1656, 2023. - Kuang et al. (2025) Jiayi Kuang, Yinghui Li, Xin Zhang, Yangning Li, Di Yin, Xing Sun, Ying Shen, and Philip S Yu. Process-level trajectory evaluation for environment configuration in software engineering agents. arXiv preprint arXiv:2510.25694, 2025. - Lin et al. (2018) Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018. - Liu et al. (2025) Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. - Lukasczyk & Fraser (2022) Stephan Lukasczyk and Gordon Fraser. Pynguin: automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, ICSE ’22, pp. 168–172, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392235. doi: 10.1145/3510454.3516829. URL https://doi.org/10.1145/3510454.3516829. - Ma et al. (2025) Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6917–6935, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.343. URL https://aclanthology.org/2025.acl-long.343/. - Milliken et al. (2025) Louis Milliken, Sungmin Kang, and Shin Yoo. Beyond pip Install: Evaluating LLM Agents for the Automated Installation of Python Projects . In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 1–11, Los Alamitos, CA, USA, March 2025. IEEE Computer Society. doi: 10.1109/SANER64311.2025.00009. URL https://doi.ieeecomputersociety.org/10.1109/SANER64311.2025.00009. - Pacheco et al. (2007) Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07), pp. 75–84, 2007. doi: 10.1109/ICSE.2007.37. - Pan et al. (2024) Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024. - Qwen Team (2025a) Qwen Team. Qwen3-coder: Agentic coding in the world. https://qwen.ai/blog?id=qwen3-coder, 2025a. Accessed: 2025-07-22. - Qwen Team (2025b) Qwen Team. Qwen3-next: Towards ultimate training & inference efficiency. https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd, 2025b. Accessed: 2025-09-11. - Qwen Team (2025c) Qwen Team. qwen-code. https://github.com/QwenLM/qwen-code, 2025c. - Qwen Team (2026) Qwen Team. Pushing qwen3-max-thinking beyond its limits. https://qwen.ai/blog?id=qwen3-max-thinking, 2026. Accessed: 2026-01-26. - Rashid et al. (2025) Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703, 2025. - Schäfer et al. (2024) Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85–105, 2024. doi: 10.1109/TSE.2023.3334955. - Sonwane et al. (2025) Atharv Sonwane, Isadora White, Hyunji Lee, Matheus Pereira, Lucas Caccia, Minseon Kim, Zhengyan Shi, Chinmay Singh, Alessandro Sordoni, Marc-Alexandre Côté, et al. Bugpilot: Complex bug generation for efficient learning of swe skills. arXiv preprint arXiv:2510.19898, 2025. - SWE-agent Team (2025) SWE-agent Team. mini-swe-agent: The 100 line ai agent that’s actually useful. https://github.com/SWE-agent/mini-swe-agent, 2025. Accessed: 2026-01-22. - Vergopoulos et al. (2025) Konstantinos Vergopoulos, Mark Niklas Müller, and Martin Vechev. Automated benchmark generation for repository-level coding tasks. arXiv preprint arXiv:2503.07701, 2025. - Wang et al. (2025) Lilin Wang, Lucas Ramalho, Alan Celestino, Phuc Anthony Pham, Yu Liu, Umang Kumar Sinha, Andres Portillo, Onassis Osunwa, and Gabriel Maduekwe. Swe-bench++: A framework for the scalable generation of software engineering benchmarks from open-source repositories. arXiv preprint arXiv:2512.17419, 2025. - Wang et al. (2024) Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024. - Xiao et al. (2026) Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. - Xiao et al. (2013) Xusheng Xiao, Sihan Li, Tao Xie, and Nikolai Tillmann. Characteristic studies of loop problems for structural test generation via symbolic execution. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 246–256, 2013. doi: 10.1109/ASE.2013.6693084. - Xie et al. (2024) Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. Codebenchgen: Creating scalable execution-based code generation benchmarks, 2024. - Yang et al. (2025a) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. - Yang et al. (2024) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. - Yang et al. (2025b) John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025b. URL https://arxiv.org/abs/2504.21798. - Yuan et al. (2024) Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. Evaluating and improving chatgpt for unit test generation. Proc. ACM Softw. Eng., 1(FSE), July 2024. doi: 10.1145/3660783. URL https://doi.org/10.1145/3660783. - Zan et al. (2025) Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: A multilingual benchmark for issue resolving, 2025. URL https://arxiv.org/abs/2504.02605. - Zeng et al. (2026) Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yaxing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, et al. davinci-dev: Agent-native mid-training for software engineering. arXiv preprint arXiv:2601.18418, 2026. - Zhang et al. (2025a) Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience. arXiv preprint arXiv:2510.08558, 2025a. - Zhang et al. (2025b) Lei Zhang, Jiaxi Yang, Min Yang, Jian Yang, Mouxiang Chen, Jiajun Zhang, Zeyu Cui, Binyuan Hui, and Junyang Lin. Synthesizing software engineering data in a test-driven manner. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 76518–76540. PMLR, 13–19 Jul 2025b. URL https://proceedings.mlr.press/v267/zhang25cn.html. - Zhang et al. (2026) Lei Zhang, Mouxiang Chen, Ruisheng Cao, Jiawei Chen, Fan Zhou, Yiheng Xu, Jiaxi Yang, Zeyao Ma, Liang Chen, Changwei Luo, Kai Zhang, Fan Yan, KaShun Shum, Jiajun Zhang, Zeyu Cui, Feng Hu, Junyang Lin, Binyuan Hui, and Min Yang. Megaflow: Large-scale distributed orchestration system for the agentic era, 2026. URL https://arxiv.org/abs/2601.07526. - Zhang et al. (2025c) Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025c.