2505.17826

# : A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models **Authors**: Alibaba Group Abstract Trinity-RFT is a general-purpose, unified and easy-to-use framework designed for reinforcement fine-tuning (RFT) of large language models. It is built with a modular and decoupled design, consisting of (1) an RFT-core that unifies and generalizes synchronous/asynchronous, on-policy/off-policy, and online/offline modes of RFT; (2) seamless integration for agent-environment interaction with high efficiency and robustness; and (3) systematic data pipelines optimized for RFT. Trinity-RFT can be easily adapted for diverse application scenarios, and serves as a unified platform for development and research of advanced reinforcement learning paradigms at both macroscopic and microscopic levels. This technical report outlines the vision, features, design and implementations of Trinity-RFT, accompanied by extensive examples, applications and experiments that demonstrate its functionalities and user-friendliness. footnotetext: Equal contribution. footnotetext: Corresponding author. {chenyanxi.cyx,panxuchen.pxc,daoyuanchen.cdy,yaliang.li,bolin.ding}@alibaba-inc.com GitHub: https://github.com/modelscope/Trinity-RFT Documentation: https://modelscope.github.io/Trinity-RFT Note: Trinity-RFT is currently under active development. This technical report corresponds to commit id 63d4920 (July 14, 2025) of the GitHub repository, and will be continuously updated as the codebase evolves. Comments, suggestions and contributions are welcome! 1 Introduction Reinforcement learning (RL) has achieved remarkable success in the development of large language models (LLMs). Examples include aligning LLMs with human preferences via reinforcement learning from human feedback (RLHF) [24], and training long-CoT reasoning models via RL with rule-based or verifiable rewards (RLVR) [5, 35]. However, such approaches are limited in their abilities to handle dynamic, agentic and continuous learning in the real world. We envision a future where AI agents learn by interacting directly with environments, collecting lagged or complex reward signals, and continuously refining their behavior through RL based on the collected experiences [32]. For example, imagine an AI scientist who designs an experiment, executes it, waits for feedback (while working on other tasks concurrently), and iteratively updates itself based on true environmental rewards and feedback when the experiment is finally finished. This vision motivates us to develop Trinity-RFT, a reinforcement fine-tuning (RFT) framework that aims to offer a path into this future. The modular, decoupled and trinity design of Trinity-RFT illustrated in Figure 1, along with its various features, makes it a promising solution for realizing such a vision. <details> <summary>x2.png Details</summary>  ### Visual Description # Technical Document Extraction: RFT-core Architecture Diagram ## 1. Overview This image is a technical system architecture diagram for **RFT-core**, a framework likely related to Reinforcement Learning from Feedback or similar iterative model training processes. The diagram illustrates the flow of data and synchronization between environment interaction, data storage, and model training. ## 2. Component Segmentation ### A. External Entities (Left & Top-Right) * **Environment & Human:** Represented by a globe and a person icon. This is the source of external feedback and the arena for agent interaction. * **Data Pipelines:** Represented by a magnifying glass over a waveform icon. This represents external data processing streams. ### B. Core System Nodes (The Central Triangle - RFT-core) The core architecture is organized in a triangular relationship, labeled centrally as **RFT-core**. * **Buffer:** (Top) Represented by a database icon. Acts as the central repository for experiences and training data. * **Explorer:** (Bottom-Left) Represented by a robot icon. Responsible for interacting with the environment. * **Trainer:** (Bottom-Right) Represented by a head with gears icon. Responsible for updating the model based on data. ### C. Infrastructure Layer (Bottom) * **LLM Infra:** Represented by an oval at the base. * **Sub-text:** (Training, Inference, Model Sync, ...) ### D. Hierarchical Classification (Right Sidebar) The diagram is divided into three vertical tiers: 1. **High-Level:** Aligned with Buffer, Environment, and Data Pipelines. 2. **Middle-Level:** Aligned with Explorer and Trainer. 3. **Low-Level:** Aligned with LLM Infra. --- ## 3. Data Flow and Process Logic The diagram uses directed arrows to indicate the sequence of operations and data movement: | Source | Destination | Label / Action | Description | | :--- | :--- | :--- | :--- | | **Environment & Human** | **Explorer** | Agent-Env Interaction | Bi-directional curved arrows showing the agent acting and receiving state/reward. | | **Environment & Human** | **Buffer** | Additional Feedback | Direct input of human/environmental feedback into the storage buffer. | | **Data Pipelines** | **Buffer** | Clean/Filter/Prioritize/Synthesize/... | External data is processed and ingested into the buffer. | | **Data Pipelines** | **Trainer** | Process Training Batch | Direct feed of processed data batches to the training component. | | **Explorer** | **Buffer** | Rollout Experiences | The agent sends its interaction history (rollouts) to the buffer. (Accompanied by a document icon). | | **Buffer** | **Trainer** | Training Data | The trainer pulls stored data from the buffer for model updates. (Accompanied by a document icon). | | **Trainer** | **Explorer** | Synchronize Model Weights | A feedback loop (with a sync icon) where the updated model from the trainer is sent back to the explorer. | | **Explorer** | **LLM Infra** | (Solid Arrow) | Indicates the explorer utilizes the underlying LLM infrastructure for inference. | | **Trainer** | **LLM Infra** | (Solid Arrow) | Indicates the trainer utilizes the underlying LLM infrastructure for training operations. | --- ## 4. Technical Summary of Operations The **RFT-core** system operates as a closed-loop learning system: 1. The **Explorer** interacts with the **Environment & Human** layer to generate **Rollout Experiences**. 2. These experiences, along with **Additional Feedback** and external **Data Pipelines**, are collected in the **Buffer**. 3. The **Trainer** consumes **Training Data** from the Buffer and **Training Batches** from the pipelines to optimize the model. 4. The updated parameters are sent back to the Explorer via **Synchronize Model Weights**, completing the cycle. 5. The entire process is supported by a **Low-Level LLM Infra** that handles the heavy lifting of training, inference, and synchronization. </details> Figure 1: The high-level design of Trinity-RFT. 1.1 Key Features of Trinity-RFT Trinity-RFT is a general-purpose, unified, scalable and user-friendly RL framework that can be easily adapted for diverse experimental or real-world scenarios. It integrates both macroscopic and microscopic RL methodologies in one place; roughly speaking, the former deals with natural language and plain text, while the latter handles torch.Tensor (such as token probabilities, gradients, and model weights of LLMs) Many prior RL works for games/control/LLMs focus mainly on the microscopic aspect, e.g., designing policy loss functions or optimization techniques for updating the policy model. On the other hand, pre-trained LLMs, as generative models with rich prior knowledge of natural language and the world, open up numerous opportunities at the macroscopic level, e.g., experience synthesis by reflection or reasoning with environmental feedback [4], leveraging existing text processing methods like deduplication and quality filtering [2], among others.. The key features of Trinity-RFT are presented below, which will be further elaborated in Section 2 and exemplified in Section 3. An RFT-core that unifies and generalizes diverse RL modes. Trinity-RFT implements diverse RL methodologies in a unified manner, supporting synchronous/asynchronous, on-policy/off-policy, and online/offline training. These RL modes can be flexibly generalized, e.g., a hybrid mode that incorporates expert trajectories to accelerate an online RL process [21, 46]. This unification is made possible partly by our decoupled design (which will soon be introduced in Section 2.1) that allows rollout and training to be executed separately and scaled up independently on different devices, while having access to the same stand-alone experience buffer. The efficacy of various RL modes has been validated empirically by our experiments in Section 3.3, which particularly highlight the efficiency gains by off-policy or asynchronous methods. Agent-environment interaction as a first-class citizen. Trinity-RFT allows delayed rewards and environmental feedback in multi-step/time-lagged feedback loop, handles long-tailed latencies and the straggler effect via asynchronous and streaming LLM inference, and deals with environment/agent failures gracefully via dedicated timeout/retry/skip mechanisms. These together ensure efficiency and robustness of continuous agent-environment interaction in complex real-world scenarios. Systematic data pipelines optimized for RFT. Figure 2 illustrates the high-level design of data pipelines in Trinity-RFT, which regard rollout tasks and experiences as dynamic assets to be actively managed throughout the RFT lifecycle. Trinity-RFT empowers users to: (1) curate tasks for curriculum learning, e.g., by prioritizing easier tasks at the beginning of training to stabilize and accelerate the learning process; (2) actively manipulate experience by cleaning, filtering, or synthesizing new experiences, such as repairing failed trajectories or amplifying successful ones; (3) perform online reward shaping by augmenting sparse environmental rewards with dense, computed signals, such as quality or diversity scores; (4) customize interfaces for human-in-the-loop curation and utilize an agentic paradigm for RFT data processing that translates high-level natural language commands (e.g., “improve response diversity and safety for coding scenarios”) into complex data pipelines, powered by established community tools like Data-Juicer [2]. For instance, Section 3.4 presents experiments that demonstrate the efficacy of task prioritization and reward shaping empowered by Trinity-RFT. <details> <summary>x3.png Details</summary>  ### Visual Description # Technical Document Extraction: Data Pipelines Diagram ## 1. Overview This image is a technical flow diagram illustrating a Reinforcement Learning (RL) or machine learning data architecture. It depicts the lifecycle of data from a raw state through task generation, exploration, buffering, and finally to a training module. The entire process is overseen by a centralized "Data Pipelines" management layer. The diagram uses a purple color scheme for all text, shapes, and icons. --- ## 2. Component Isolation ### Header Region - **Title:** Data Pipelines - **Icon:** A magnifying glass containing a pulse/waveform line, positioned to the left of the title. - **Function:** This acts as a central hub. Seven curved arrows originate from a single point beneath the title and point downward to various stages and transitions in the main flow, indicating that the "Data Pipelines" control or monitor every step of the process. ### Main Flow (Left to Right) The core logic follows a linear progression of five main nodes connected by horizontal arrows. | Order | Node Shape | Label | Associated Icon | | :--- | :--- | :--- | :--- | | 1 | Rounded Rectangle | **Raw Dataset** | N/A | | 2 | Rounded Rectangle | **Task Set** | N/A | | 3 | Oval | **Explorer** | A small robot icon (positioned below) | | 4 | Oval | **Buffer** | A database/stack icon (positioned below) | | 5 | Oval | **Trainer** | A human head profile with gears (positioned below) | --- ## 3. Process Flow and Transitions The transitions between nodes are labeled with specific actions. Additionally, the "Data Pipelines" hub provides specific interventions at these stages. ### Stage 1: Data Ingestion * **Transition:** From **Raw Dataset** to **Task Set**. * **Action Label:** `Conversion` * **Pipeline Intervention:** An arrow points to the `Conversion` step. ### Stage 2: Task Management * **Node:** **Task Set** * **Pipeline Intervention:** `Augment Task Set (Synthesis, Prioritization, ...)` * **Transition:** From **Task Set** to **Explorer**. * **Action Label:** `Sample Tasks for Rollout` (An arrow from the Pipeline hub points specifically to this transition). ### Stage 3: Exploration/Data Collection * **Node:** **Explorer** (represented by a robot icon). * **Transition:** From **Explorer** to **Buffer**. * **Action Label:** `Rollout Experiences` ### Stage 4: Data Storage and Refinement * **Node:** **Buffer** (represented by a database icon). * **Pipeline Intervention:** `Experience Shaping / Cleaning / Synthesis` (An arrow from the Pipeline hub points directly to the Buffer node). * **Transition:** From **Buffer** to **Trainer**. * **Action Label:** `Sample Experience Batches for RL Training` (An arrow from the Pipeline hub points to this transition). ### Stage 5: Model Training * **Node:** **Trainer** (represented by a brain/gears icon). * **Pipeline Intervention:** `Local Experience Replay` (An arrow from the Pipeline hub points directly to the Trainer node). --- ## 4. Summary of Data Pipeline Interventions The "Data Pipelines" entity performs the following seven distinct functions (mapped to the seven arrows): 1. **Conversion** of raw data. 2. **Augmentation** of tasks (Synthesis and Prioritization). 3. **Sampling** tasks for the rollout phase. 4. **Monitoring** the rollout of experiences. 5. **Shaping, Cleaning, and Synthesizing** experiences within the buffer. 6. **Sampling** specific batches for the training phase. 7. **Managing** local experience replay within the trainer. </details> Figure 2: The high-level design of data pipelines in Trinity-RFT. User-friendliness as a top priority. For development and research, the modular and decoupled design of Trinity-RFT allows the user to develop new RFT methodologies by adding one or a few small, plug-and-play classes (modified from built-in templates) that implement the essential functionalities of interest, with minimal code duplication or intrusive changes to the codebase. An example can be found in Section 3.2, which shows that three compact python classes (with around 200 lines of code in total) suffice for implementing a hybrid RL process that leverages samples from multiple data sources and updates the policy model with a customized loss function. For applications, the user can adapt Trinity-RFT to a new scenario by simply implementing a single Workflow class that specifies the logic of agent-environment interaction, as will be exemplified in Section 3.1. To further enhance usability, Trinity-RFT incorporates various graphical user interfaces to support low-code usage and development, enhance transparency of the RFT process, and facilitate easy monitoring and tracking. 1.2 Related Works There exist numerous open-source RLHF frameworks, such as veRL [30], OpenRLHF [13], TRL [40], ChatLearn [1], Asynchronous RLHF [23], among others. Some of them have been further adapted for training long-CoT reasoning models or for agentic RL more recently. Concurrent to Trinity-RFT, some recent works on LLM reinforcement learning also advocate a decoupled and/or asynchronous design; examples include StreamRL [50], MiMo [44], AReaL [9], ROLL [37], LlamaRL [43], Magistral [18], AsyncFlow [12], among others. Complementary to this huge number of related works, Trinity-RFT provides the community with a new solution that is powerful, easy-to-use, and unique in certain aspects. In a nutshell, Trinity-RFT aims to be general-purpose and applicable to diverse application scenarios, while unifying various RFT modes, RFT methodologies at macroscopic and microscopic levels, and RFT-core/agent-environment interaction/data pipelines. Such a system-engineering perspective makes Trinity-RFT particularly useful for handling the whole RFT pipeline in one place. We also hope that some specific features of Trinity-RFT, such as data persistence in the experience buffer, and distributed deployment of multiple independent explorers, will open up new opportunities for LLM reinforcement fine-tuning. 2 Design and Implementations The overall design of Trinity-RFT exhibits a trinity consisting of (1) RFT-core, (2) agent-environment interaction, and (3) data pipelines, which are illustrated in Figure 1 and elaborated in this section. 2.1 RFT-Core RFT-core is the component of Trinity-RFT, highlighted at the center of Figure 1, where the core RFT process happens. Its design also exhibits a trinity, consisting of the explorer, buffer, and trainer. - The explorer, powered by a rollout model, takes a task as input and solves it by executing a workflow that specifies the logic of agent-environment interaction, thereby collecting experiences (including rollout trajectories, rewards, and other useful information) to be stored in the buffer. - The buffer stores experiences that can be generated by the explorer or by other sources, such as human experts. It can be realized in various forms, such as a non-persistent ray.Queue or a persistent SQLite database. It also assists with fetching training samples for the trainer, and can be integrated with advanced sampling strategies and post-processing operations. - The trainer, backed by a policy model, samples batches of experiences from the buffer and updates the policy model via RL algorithms. Our implementations allow the explorer and trainer to be deployed on separate machines and act independently. They are only connected via (1) access to the same experience buffer with a customizable sampling strategy, and (2) model weight synchronization by a customizable schedule. See Figure 3 for an illustration. This decoupled design of RFT-core offers support for diverse RFT modes with great flexibility. <details> <summary>x4.png Details</summary>  ### Visual Description # Technical Document Extraction: Reinforcement Learning System Architecture This document provides a comprehensive extraction and analysis of the provided architectural diagram, which illustrates a Reinforcement Learning (RL) pipeline consisting of three primary functional regions: **Buffer**, **Explorer**, and **Trainer**. --- ## 1. Component Isolation (Spatial Segmentation) The diagram is divided into three color-coded regions: * **Top Region (Blue Background):** The **Buffer** system, responsible for task management and data processing. * **Bottom-Left Region (Yellow Background):** The **Explorer**, responsible for environment interaction and data generation. * **Bottom-Right Region (Green Background):** The **Trainer**, responsible for model optimization and weight synchronization. --- ## 2. Detailed Component Extraction ### A. Buffer Region (Top) * **Taskset (Database Icon):** The initial source of tasks. * **Task Data Processor (Purple Box):** Processes data from the Taskset. Accompanied by a magnifying glass icon (indicating monitoring/inspection). * **Raw Experiences (Database Icon):** Stores data generated by the Explorer. * **Experience Data Processor (Purple Box):** Processes raw experiences. Accompanied by a magnifying glass icon. * **Verified Experiences (Database Icon):** Stores the output of the Experience Data Processor, ready for training. ### B. Explorer Region (Bottom-Left) * **Workflow Runner (Large Yellow Container):** * **Agent (Yellow Box):** Sends an "action" to the Environment. * **Environment (Yellow Box):** Sends a "reward" back to the Agent. * **Rollout Model (Yellow Box):** Interfaces with the Agent. * **Reward Model (Yellow Box):** Interfaces with the Environment. * **Output:** Generates "Experience" (indicated by a document icon) which is sent to the **Raw Experiences** database. ### C. Trainer Region (Bottom-Right) * **Reference Model (Green Dashed Box):** Used as a baseline for training. * **Actor Model (Green Solid Box):** The primary model being trained. * **Critic Model (Green Dashed Box):** Evaluates the actions of the Actor. * **Input:** Receives "Experience" (indicated by a document icon) from the **Verified Experiences** database. --- ## 3. Process Flow and Logic The system operates in a continuous loop characterized by the following data movements: 1. **Task Initialization:** * `Taskset` $\rightarrow$ `Task Data Processor` $\rightarrow$ `Workflow Runner`. * The transition is labeled with a clipboard icon and the text **"Task"**. 2. **Experience Generation (The Explorer Loop):** * Inside the `Workflow Runner`, the `Agent` and `Environment` interact via an **action/reward** cycle. * The `Rollout Model` and `Reward Model` support this interaction. * The resulting data is labeled **"Experience"** and sent to the `Raw Experiences` database in the Buffer. 3. **Data Refinement:** * `Raw Experiences` $\rightarrow$ `Experience Data Processor` $\rightarrow$ `Verified Experiences`. * This stage ensures data quality before it reaches the training phase. 4. **Model Training (The Trainer):** * `Verified Experiences` are fed into the `Reference Model`, `Actor Model`, and `Critic Model`. * The `Actor Model` is the central component, flanked by the `Reference` and `Critic` models (indicated by vertical dashed arrows). 5. **Weight Synchronization (The Feedback Loop):** * A thick grey arrow labeled **"Sync. Weights"** points from the `Actor Model` (Trainer) back to the `Rollout Model` (Explorer). * This represents the deployment of the trained model back into the exploration phase to improve future data collection. --- ## 4. Textual Transcription | Category | Transcribed Text | | :--- | :--- | | **Headers/Regions** | Buffer, Explorer, Trainer | | **Data Stores** | Taskset, Raw Experiences, Verified Experiences | | **Processors** | Task Data Processor, Experience Data Processor | | **Explorer Components** | Workflow Runner, Agent, Environment, action, reward, Rollout Model, Reward Model | | **Trainer Components** | Reference Model, Actor Model, Critic Model | | **Data Labels** | Task, Experience, Experience, Sync. Weights | --- ## 5. Summary of Facts * **Language:** English (100%). * **System Type:** Distributed Reinforcement Learning Architecture. * **Key Feature:** Decoupled data collection (Explorer) and model optimization (Trainer) mediated by a processing buffer. * **Monitoring:** The presence of magnifying glass icons suggests that both Task and Experience processing stages are subject to automated or manual validation/monitoring. </details> Figure 3: The architecture of RFT-core in Trinity-RFT. 2.1.1 Unified Support for Diverse RFT Modes We present the RFT modes supported by Trinity-RFT, some of which are demonstrated in Figure 4. <details> <summary>x5.png Details</summary>  ### Visual Description # Technical Document Extraction: Reinforcement Learning Training Architectures This document provides a detailed technical extraction of an image illustrating four different architectural patterns for Reinforcement Learning (RL) training workflows. ## 1. Global Legend and Component Definitions The following components and color codes are consistent across all four diagrams. | Component | Color | Description | | :--- | :--- | :--- | | **Rollout** | Yellow (Rounded Rect) | Data collection phase performed by the Explorer. | | **Train** | Green (Rounded Rect) | Model optimization phase performed by the Trainer. | | **Sync. weight (NCCL)** | Purple (Thin Rect) | Fast weight synchronization using NCCL (NVIDIA Collective Communications Library). | | **Sync. weight (Checkpoint)** | Red (Thin Rect) | Weight synchronization via checkpointing (typically slower/asynchronous). | | **Buffer** | Light Blue (Long Rect) | Shared storage for experiences (Replay Buffer). | | **Data Flow** | Curved Arrows | Arrows from Explorer to Buffer indicate data storage; arrows from Buffer to Trainer indicate data sampling. | | **Document Icon** | Grey Icon | Represents the transfer of experience data/logs. | --- ## 2. Architectural Analysis by Quadrant The image is divided into four quadrants (a through d), representing a progression from synchronous to fully asynchronous multi-agent systems. ### (a) Synchronous * **Components:** Explorer, Buffer, Trainer. * **Workflow:** 1. The Explorer completes two Rollout blocks. 2. Data is pushed to the Buffer. 3. The Trainer remains idle initially, labeled with the text: *"Wait for experiences"*. 4. Once data is available, the Trainer performs two Train blocks. 5. A **Sync. weight (NCCL)** (purple) event occurs simultaneously for both Explorer and Trainer, halting both processes until the update is complete. * **Trend:** Sequential execution with significant idle time for the Trainer. ### (b) One-Step Off-Policy * **Components:** Explorer, Buffer, Trainer. * **Workflow:** 1. The Trainer starts with a delay, labeled: *"One step offset"*. 2. The Explorer performs Rollouts continuously, punctuated by **Sync. weight (NCCL)** (purple) events. 3. The Trainer processes data from the Buffer. 4. A specific synchronization point is highlighted with a downward arrow and the text: *"Wait for synchronization"*. This indicates the Trainer must wait for the Explorer to reach a sync point before it can proceed with the next training step. * **Trend:** Overlapped execution but still constrained by rigid synchronization points. ### (c) Fully Asynchronous * **Components:** Explorer, Buffer, Trainer. * **Workflow:** 1. The Explorer and Trainer operate independently. 2. Synchronization is handled via **Sync. weight (Checkpoint)** (red). 3. The Explorer does not stop for the Trainer; it continues Rollouts and updates its weights from checkpoints whenever they become available. 4. The Trainer continues training and periodically saves checkpoints. * **Trend:** Maximum hardware utilization; no "Wait" states are visible in the timeline. ### (d) Multi-Explorer Asynchronous * **Components:** Explorer1, Explorer2, Buffer, Trainer. * **Workflow:** 1. Two independent Explorer lines (Explorer1 and Explorer2) perform Rollouts at different cadences. 2. Both Explorers push data into a single shared **Buffer**. 3. The Trainer samples from this aggregated Buffer. 4. Weight synchronization is performed via **Sync. weight (Checkpoint)** (red) across all three active entities (Explorer1, Explorer2, and Trainer). * **Trend:** Scaled-out data collection. The Trainer is constantly fed by multiple sources, and synchronization is decoupled from the execution loops. --- ## 3. Spatial Grounding and Textual Labels * **Header Labels:** * (a) Synchronous [Top-Left] * (b) One-Step Off-Policy [Top-Right] * (c) Fully Asynchronous [Bottom-Left] * (d) Multi-Explorer Asynchronous [Bottom-Right] * **Y-Axis Labels (Internal to each quadrant):** * Explorer / Explorer1 / Explorer2 * Buffer * Trainer * **Legend Position:** Located in the bottom-left quadrant, contained within a dashed rounded box. * **Annotations:** * *"Wait for experiences"* (Quadrant a, under Trainer line) * *"One step offset"* (Quadrant b, under Trainer start) * *"Wait for synchronization"* (Quadrant b, between Trainer blocks) ## 4. Summary of Data Flow In all models, the **Explorer** is the producer of data (Yellow $\rightarrow$ Blue Buffer), and the **Trainer** is the consumer of data (Blue Buffer $\rightarrow$ Green). The primary differentiator between the four diagrams is the **temporal alignment** and the **mechanism of weight feedback** (Purple/NCCL vs. Red/Checkpoint). </details> Figure 4: A visualization of diverse RFT modes supported by Trinity-RFT, including: (a) synchronous mode, with sync_interval=2; (b) one-step off-policy mode, with sync_interval=1 and sync_offset=1; (c) fully asynchronous mode, with sync_interval=2; (d) multi-explorer asynchronous mode, with sync_interval=2. The buffer supports, in principle, arbitrary management and sampling strategies for experiences. Synchronous mode. In the synchronous mode shown in Figure 4 (a), the explorer and trainer get launched simultaneously, work in close coordination, and synchronize their model weights once every sync_interval training steps. Within each synchronization period, the explorer continuously generates sync_interval batches of rollout experiences and stores them in the buffer, which are then retrieved and utilized by the trainer for updating the policy model. If sync_interval=1, this is a strictly on-policy RL process, whereas if sync_interval>1, it becomes off-policy (akin to the mode adopted in [35]) and can be accelerated by pipeline parallelism between the explorer and trainer. This mode can be activated by setting the configuration parameter mode=both. One-step off-policy mode. This mode, demonstrated in Figure 4 (b), closely resembles the synchronous mode, except for an offset of one batch between the explorer and trainer. This allows the trainer to sample experiences from the buffer immediately after model weight synchronization, thereby streamlining the execution of explorer and trainer with smaller pipeline bubbles, at the cost of slight off-policyness. The visualization in Figure 4 (b) corresponds to configuration parameters sync_interval=1 and sync_offset=1, both of which can take more general values in Trinity-RFT. Asynchronous mode. In the fully asynchronous mode shown in Figure 4 (c), the explorer and trainer act almost independently. The explorer continuously generates rollout experiences and stores them in the buffer, while the trainer continuously samples experiences from the buffer and uses them for training the policy model. External experiences, e.g., those generated by expert models or humans, can be continuously incorporated into the buffer as well. The explorer and trainer independently load or save model weights from the checkpoint directory every sync_interval steps, keeping the distribution of rollout experiences up to date. This mode can be activated by setting mode=explore/train and launching the explorer and trainer separately on different GPUs. Multi-explorer asynchronous mode. One benefit brought by the decoupled design is that explorers and trainers can scale up independently on separate devices. As a proof-of-concept, Trinity-RFT offers support for a multi-explorer asynchronous mode, as demonstrated in Figure 4 (d), where multiple explores send the generated rollout experiences to the same buffer. Scaling up the number of independent and distributed explorers can be particularly useful for resolving data scarcity and speeding up the generation of experiences in real-world scenarios where rollout trajectories have to be sampled via interaction with the physical world, or in an environment with sparse and lagged feedback. Another by-product of this multi-explorer mode is 24-hour non-interrupted service for real-world online serving situations: since the explorers can pause and update model weights at different moments, it can be guaranteed that there is always one explorer ready to serve an incoming request immediately whenever it arrives. This is in contrast to a single-explorer mode, where online service has to be paused when the explorer is updating its model weights. Benchmark mode. Trinity-RFT supports a benchmark mode that allows the user to evaluate one or multiple checkpoints on arbitrary benchmarks, after the RFT training process has finished. To activate this mode, the user simply needs to set mode=bench and specify the paths for the evaluation datasets in the configurations. This mode can be particularly useful for experimental purposes; for example, the user might want to try out different RFT techniques or configurations quickly (with limited evaluation on hold-out data) during training, identify which RFT trials have achieved stable convergence and high rewards, and then conduct more thorough evaluations only for the checkpoints of these successful trials. Train-only mode. In certain scenarios, the user would like to train the policy model without further exploration, using experiences that have already been collected and stored in the buffer. This train-only mode can be activated by setting the configuration parameter mode $=$ train and launching the trainer alone. Offline methods like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) [25] can be regarded as special cases of such scenarios, both of which are natively supported by Trinity-RFT. For another example, consider an online RFT process that expands over a long period, where the explorer alone is launched during the daytime for serving human users and collecting experiences, while the trainer alone is launched at night for updating the policy model (which will be thoroughly validated and evaluated before it can be actually deployed as the rollout model for the next day). Discussions. We conclude this subsection with two remarks. (1) Given the unified implementation of various RFT modes, it is easy to design and implement a hybrid mode with Trinity-RFT that combines multiple modes into a single learning process. One example is learning with both online rollout data and offline-collected expert data, via jointly optimizing two loss terms corresponding to these two data sources. Section 3.2 illustrates how to implement this conveniently in Trinity-RFT. (2) We take a system-algorithm co-design perspective in the development of Trinity-RFT, aiming to unify and generalize diverse RFT methodologies in this framework. RFT-core provides the necessary infrastructure for achieving this goal. This technical report focuses on the system perspective, and we refer interested readers to the literature for recent algorithmic developments in off-policy / asynchronous RL for LLMs [21, 6, 26, 35, 7, 23, 42, 45, 46, 47]. 2.1.2 Implementations of RFT-Core We present some implementation details of RFT-core in the following. Inference and training engines. The current version of Trinity-RFT leverages vLLM [15] as the inference engine for the explorer, which offers features including paged attention, continuous batching [49], asynchronous and concurrent inference for multiple rollout trajectories, among others. Trinity-RFT also leverages verl [30] as the training engine for the trainer, which gracefully handles model placement (for the policy, critic and reference models) and incorporates various performance optimizations for training (such as dynamic batching, management of padding and unpadding, etc.). Trinity-RFT stands on the shoulders of these excellent open-source projects, and will continue to benefit from their future development. Experience buffer. Trinity-RFT supports multiple types of experience buffers, ranging from a non-persistent ray.Queue to persistent SQLite or Redis databases. While using a basic first-in-first-out queue is the most straightforward approach, data persistence with a database opens up many new opportunities (e.g., advanced sampling strategies), as discussed throughout this report. Trinity-RFT has provided dedicated read/write control to prevent any conflict in accessing the buffer. Model weight synchronization. Trinity-RFT supports model weight synchronization between the explorer and trainer by NCCL [22], or by checkpoint saving and loading. The former is faster (when available), while the latter is generally more flexible and widely applicable, especially for asynchronous RFT modes. 2.2 Agent-Environment Interaction To adapt Trinity-RFT to a new downstream scenario, the user mainly needs to define and register a customized workflow (by inheriting the base class Workflow or MultiTurnWorkflow) where the logic of agent-environment interaction for this particular scenario is implemented. Advanced methods for experience synthesis with environmental feedback [4] can be implemented in the same way as well. See Section 3.1 for detailed examples. The workflow will then be executed by workflow runners within the explorer for generating experiences, as shown in Figure 3. Numerous challenges arise when one tries to build an RFT framework that can efficiently and robustly handle real-world interaction between the LLM-powered agent and the environment. These include long-tailed latencies, agent/environment failures, and lagged reward signals, among others. Trinity-RFT regards agent-environment interaction as a first-class citizen and incorporates various solutions to tackle these challenges, for example: - The workflow runners in Trinity-RFT support asynchronous and streaming generation of rollout trajectories for multiple tasks. This helps mitigate the straggler effect caused by the long-tailed latencies in rollout generation and agent-environment interaction, thereby accelerating the RFT process. Load balancing among multiple LLM inference engines within one RFT training course is also taken care of, and would be one direction for further optimizing the utilization of computational resources. - Trinity-RFT incorporates various timeout/retry/skip mechanisms for fault tolerance and robustness, which ensure that continuous rollout generation would not be interrupted or blocked by individual failures in certain rounds of agent-environment interaction. This is crucial for stable and efficient learning in real-world scenarios, e.g., when the agent interacts with a large number of MCP services [17] that differ vastly in quality and availability. - Trinity-RFT is built to provide native support for asynchronous RFT modes, which allow great flexibility in the paces of the explorer and trainer. This can boost the overall efficiency of the RFT process, compared to synchronous modes where the slower one among the explorer and trainer can block the progress of the other and cause waste of computational resources. - For lagged reward signals, the trinity design of RFT-core offers a natural solution. As soon as the rollout trajectory (without reward values) has been generated, it is saved into the experience buffer, but marked as “not ready for training”. The explorer is now free from this task and may continue to collect experiences for other tasks. When the reward signals from the environment finally arrive, they are written to the buffer, and the corresponding experience is now marked as “ready for training”. - For multi-turn conversations and ReAct-style workflows [48], Trinity-RFT supports concatenating multiple rounds of agent-environment interaction compactly into a single sequence, with proper masking that indicates which tokens need to be incorporated into the training objective of RL algorithms. This avoids unnecessary recomputation and thus improves training efficiency, compared to a vanilla approach that represents a $K$ -turn rollout trajectory with $K$ separate samples. - As another performance optimization, the implementation of Trinity-RFT allows resetting the environment in a workflow, rather than re-initializing it every time. This is especially useful for scenarios where setting up the environment is costly. 2.3 Data Pipelines The data pipelines in Trinity-RFT aim to address fundamental challenges in RFT scenarios, such as managing heterogeneous data dynamics across interaction workflows, enabling delayed reward integration, and facilitating continuous data curation. Our solutions center on four core aspects: end-to-end data transformation, task curation, active experience shaping, and human-in-the-loop curation, each corresponding to key requirements identified in our development of RFT-core (Section 2.1). 2.3.1 End-to-end Data Transformation To support the diverse RFT modes (e.g., synchronous or asynchronous) in Trinity-RFT, we establish a service-oriented data pipeline architecture as illustrated in Figure 5. It decouples data pipeline logic from procedure control to enable flexible RL-oriented data transformations with two key modules: - The Formatter Module unifies disparate data sources into RFT-compatible formats, providing convenient conversion between raw inputs (e.g., meta-prompts, domain-specific corpora, and QA pairs with tagged rewards) and structured RFT representations. For efficient RFT workloads, we utilize buffer-based persistent storage (Section 2.1) to support different data models, such as ExperienceModel for prioritized rollout trajectories and DPODataModel for preference pairs. The conversion logic and data models are highly customizable to meet diverse requirements for managing experience data. This flexibility enables robust metadata recording and field normalization, which is essential for advanced scenarios such as asynchronous RFT in trainer-explorer environments, agent self-evolution from a cold start using meta-prompts, and knowledge injection from structurally complex domain-specific corpora. - The Controller Module manages the complete data pipeline lifecycle through distributed server initialization, declarative configuration, and automated dataset persistence. It implements dynamic control mechanisms for asynchronous scenarios and protection against resource exhaustion, with configurable termination conditions based on compute quota or data quantity. This modular design enables Trinity-RFT to handle data transformations flexibly while maintaining consistency across different RFT modes. The Formatter-Controller duality mirrors the explorer-trainer decoupling in RFT-core, enabling parallel data ingestion and model updating. This design also allows Trinity-RFT to handle delayed rewards through version-controlled experience updates while maintaining low-latency sampling for the trainer. <details> <summary>x6.png Details</summary>  ### Visual Description # Technical Document Extraction: Data Processing and Training Pipeline This image illustrates a complex, two-stage data pipeline for machine learning, specifically focusing on task curation and experience shaping for an autonomous agent. The system is divided into two primary functional blocks connected by a shared "Buffer" and an "Explorer/Trainer" loop. ## 1. Component Isolation The diagram is organized into four main regions: * **Left Block (Purple Dashed Border):** Task Curation & Prioritization. * **Right Block (Purple Dashed Border):** Experience Shaping. * **Central Horizontal Band (Light Blue Background):** The Buffer, containing data storage cylinders. * **Bottom Row:** The active agents (Explorer and Trainer) and feedback loops. --- ## 2. Detailed Component Analysis ### Region A: Task Curation & Prioritization (Left) This section describes the ingestion and refinement of raw data into actionable tasks. * **Header Label:** Task Curation & Prioritization * **Data Flow:** 1. **Raw Data (Cylinder):** Located in the Buffer. 2. **Upward Path:** An arrow leads from Raw Data to the Data Processor. It is marked with a black "clipboard/list" icon. 3. **Data Processor (Purple Box):** Contains a magnifying glass/graph icon. * *Functions listed:* * Convert format * Clean & augment * Online Scoring * ... (indicates additional unspecified steps) 4. **Downward Path:** An arrow leads from the Data Processor to the Taskset. It is marked with a purple "clipboard/list" icon. 5. **Taskset (Cylinder):** Located in the Buffer. ### Region B: Experience Shaping (Right) This section describes how raw experiences are processed into refined training data. * **Header Label:** Experience Shaping * **Data Flow:** 1. **Raw Experience (Cylinder):** Located in the Buffer. 2. **Upward Path:** An arrow leads from Raw Experience to the Data Processor. It is marked with a black "document" icon. 3. **Data Processor (Purple Box):** Contains a magnifying glass/graph icon. * *Functions listed:* * Dense rewards * Human-in-the-loop * Counterfactual, dynamic synthesis * ... (indicates additional unspecified steps) 4. **Downward Path:** An arrow leads from the Data Processor to Experience. It is marked with a purple "document" icon. 5. **Experience (Cylinder):** Located in the Buffer. ### Region C: The Buffer (Center) A light blue shaded region that acts as the persistent storage layer for the pipeline. It contains four data cylinders: 1. **Raw Data** 2. **Taskset** 3. **Raw Experience** 4. **Experience** ### Region D: Execution & Training (Bottom) This region shows the interaction between the stored data and the active components. * **Explorer (Yellow Box):** Represented by a robot icon. * **Input:** Receives data from the **Taskset** (marked with a purple clipboard icon). * **Output:** Sends data to **Raw Experience** (marked with a black document icon). * **Feedback:** Sends "Environment Feedback" (dotted arrow) back toward the Taskset/Raw Data area. * **Trainer (Green Box):** Represented by a head/gears icon. * **Input:** Receives data from **Experience** (marked with a black document icon). * **Feedback:** Sends "Model Feedback" (dotted arrow) back toward the Raw Experience/Experience area. --- ## 3. Process Flow Summary 1. **Task Generation:** Raw Data is pulled from the buffer, processed (cleaned/scored) by the Data Processor, and stored as a Taskset. 2. **Exploration:** The **Explorer** agent takes the Taskset and interacts with an environment. This interaction generates **Raw Experience**. 3. **Experience Refinement:** Raw Experience is pulled from the buffer, processed (rewarded/synthesized) by the Data Processor, and stored as **Experience**. 4. **Training:** The **Trainer** uses the refined Experience to update the model. 5. **Feedback Loops:** Both the Explorer and Trainer provide feedback (Environment and Model respectively) that presumably informs the curation and shaping processes in subsequent iterations. ## 4. Text Transcription | Category | Transcribed Text | | :--- | :--- | | **Headers** | Task Curation & Prioritization; Experience Shaping | | **Storage (Buffer)** | Raw Data; Taskset; Raw Experience; Experience; Buffer | | **Processors** | Data Processor (x2) | | **Processor 1 Tasks** | Convert format; Clean & augment; Online Scoring; ... | | **Processor 2 Tasks** | Dense rewards; Human-in-the-loop; Counterfactual, dynamic synthesis; ... | | **Agents** | Explorer; Trainer | | **Feedback** | Environment Feedback; Model Feedback | </details> Figure 5: The interaction of data processor and data buffers in Trinity-RFT, divided into two key stages. Left: Task Curation & Prioritization prepares the initial tasks for the explorer. Right: Experience Shaping processes the collected trajectories from the explorer before they are used by the trainer. The data processor is a central component that operates on different buffers at different stages. 2.3.2 Task Curation and Prioritization Before the RFT loop begins, it is crucial to prepare a high-quality set of initial tasks. This stage, depicted on the left side of Figure 5, transforms raw data into an optimized task set for the explorer. The process begins with raw data sources (e.g., prompts, domain corpora), which are ingested into a buffer. The Data Processor, powered by over 100 operators from Data-Juicer [2], reads from this buffer to perform various curation tasks. It provides composable building blocks for experience cleaning (e.g., length filters, duplication removal), safety alignment (e.g., toxicity detection, ethics checks), and preference data synthesis (e.g., critique-conditioned augmentation). By treating Data-Juicer as a modular data processing operator pool rather than a central dependency, Trinity-RFT provides RL-specific abstractions and coherence, while benefiting from well-established data tools. The processed data is then organized into a structured task buffer. This stage effectively implements a form of curriculum learning by allowing users to prioritize tasks (e.g., from easy to hard), guiding the explorer towards a more efficient and stable learning trajectory from the outset. This entire workflow is managed by a service-oriented architecture that decouples data logic from procedural control, ensuring flexibility and scalability, especially in asynchronous and distributed settings. 2.3.3 Active Experience Shaping Once the explorer begins interacting with the environment, it generates a continuous stream of experience data. To maximize learning efficiency, this raw experience must be actively shaped before it reaches the trainer. This stage is shown on the right side of Figure 5. Generated experiences are first collected in a buffer. The Data Processor is applied again with a series of transformations to clean, augment, or synthesize these experiences. This is where the core of RFT data intelligence lies. Key capabilities include: - Agent-Driven Data Processing: Trinity-RFT introduces a powerful agentic paradigm for data manipulation. Users can define complex processing pipelines through high-level objectives, specified as either natural language commands (e.g., “improve safety” or “increase response diversity”) or explicit Data-Juicer configurations. The framework automatically translates these commands into executable workflows backed by its modular components like DataCleaner and DataSynthesizer. This design provides a user-friendly abstraction layer over the underlying Data-Juicer operators, making advanced processing functionalities accessible to both RFT users familiar with Data-Juicer and those who are not. It also facilitates the flexible injection of user-defined inductive biases into the learning process, unlocking new research directions for self-evolving agents, as we will discuss later in Section 2.3.5. - Online Reward Shaping: The data processor can dynamically augment the reward signal. Instead of relying on a single, often sparse, task-completion reward, users can add dense rewards based on quality, diversity, or safety scores computed on the fly. This enriched feedback provides a much stronger learning signal for the trainer. - Prioritized Experience Replay: Experiences are not treated equally. Trinity-RFT allows for flexible, multi-dimensional utility scoring to prioritize the most valuable samples for training. The DataActiveIterator supports version-controlled experience reuse and cross-task data lineage tracking, ensuring that the trainer always learns from the most informative data available. This mechanism is also critical for handling delayed rewards, as experience utilities can be updated asynchronously as new feedback arrives. 2.3.4 Human-AI Collaboration In scenarios where human feedback is irreplaceable, Trinity-RFT establishes a bi-directional human-AI collaboration loop that provides first-class support for human annotations, based on Label Studio [39] and Data-Juicer’s HumanOPs. - Multi-stage annotation. Trinity-RFT implements configurable procedures combining automatic pre-screening and human verification. Typical stages include preference annotation (comparative assessment of model responses), quality auditing (human verification of automated cleaning/synthesis results), and cold-start bootstrapping (initial dataset curation through expert demonstrations). - Native asynchronism support. As the collection of human feedback is generally slower than AI/model feedback, we provide dedicated capabilities to handle both synchronous and asynchronous feedback modes, with configurable timeout and polling parameters. The feedback collaboration is based on an event-driven design, with automatic task creation upon data state changes, configurable notifications via email/Slack/webhook, and an atomic transaction model for annotation batches. - Customization. Different applications may involve humans in heterogeneous ways. We thus prioritize flexibility in both the interaction-interface and service levels. Examples include rich built-in interfaces that can be extended in a visualized style with XML-like tags provided by Label Studio, fine-grained quality scoring for reward shaping, free-form feedback attachment for dataset shaping, among others. Moreover, for easy deployment, we provide local Label Studio instance management with automatic environment setup via Docker/pip; optimized SDK interactions with batch request coalescing; unified logging across annotation tools and ML services; and concurrent annotation campaigns through priority-based task routing, while maintaining full data lineage preserved via LineageTracker. The decoupled design of Trinity-RFT, and the presence of a standalone experience buffer in particular, enable human feedback to participate in RL loops without breaking the asynchronous execution model. For instance, human-verified samples can be prioritized for training while fresh experiences are being collected, which is a critical capability for real-world deployment scenarios with mixed feedback sources. Further details for human-AI collaboration in Trinity-RFT will be illustrated in Section 3.5. 2.3.5 Discussion: Unlocking New Research & Development Directions The modular design of our data pipelines and the powerful data processor open up promising research and development avenues to be further explored. One direction is about effective management of experience data. While prior RFT works often treat the experience as a static log, Trinity-RFT enables a more sophisticated, full-lifecycle approach to data, from selective acquisition to efficient representation: - Intelligent Perception and Collection: In an open-ended environment, what experience is “worth” recording? Storing everything creates a low signal-to-noise ratio and burdens the trainer. Trinity-RFT ’s architecture allows researchers to implement active collection strategies. For instance, one could design a data processor operator that evaluates incoming experiences from the explorer based on metrics like surprise, uncertainty, or information gain, and only commits the most salient trajectories to the replay buffer. This transforms data collection from passive logging into a targeted, intelligent process. - Adaptive Representation: Raw experience is often high-dimensional and redundant (e.g., long dialogues, complex code generation traces). How can this be distilled into a format that an agent can efficiently learn from? The data processor in Trinity-RFT acts as a powerful transformation engine. Researchers can use it to explore various representation learning techniques, such as automatically summarizing trajectories, extracting causal chains from tool usage, or converting a multi-turn dialogue into a structured preference pair. This not only makes training more efficient but also opens the door to building meta-experience (more abstract and reusable knowledge) from raw interaction data. - Agentic Workflows: Trinity-RFT ’s agent-driven processing enables the research and development of self-improving agents, e.g., by configuring the policy agent to also serve as the “processing agent” for LLM-based Data-Juicer operators. Such an agent could perform its own critique and dynamically curate its own training data, creating a truly autonomous learning and data management loop. Another direction is about synthetic and counterfactual experience processing. The integration of synthesis operators enables research into creating “better-than-real” data. Instead of relying solely on the agent’s own trial-and-error, our framework facilitates exploring questions like: - Dynamic and Composable Rewarding: With our framework, researchers can move beyond static, hand-crafted rewards. It is now possible to investigate dynamic reward shaping, where auxiliary signals like novelty, complexity, or alignment scores are automatically extracted from trajectories and composed into a dense reward function. How to define “good” experience and how can we learn the optimal combination of these reward components as the agent’s policy evolves? - Experience Reorganization: Can successful sub-trajectories from different tasks be “spliced” together to solve a novel, composite task? For example, can an agent that has learned to “open a door” and “pick up a cup” synthesize a new trajectory to "enter the room and fetch the cup"? - Failure Repair: Can the data processor identify where errors occur in a failed trajectory, and synthesize a corrected version for the trainer to learn from, effectively turning failures into valuable lessons? - Success Amplification: Can a single successful experience be augmented into multiple diverse yet successful variants, thereby improving the generalization and robustness of the learned policy? By providing dedicated capabilities for such advanced data and reward manipulation, Trinity-RFT aims to facilitates flexible processing of “experience data” for the next generation of self-evolving LLMs. 2.4 User-Friendliness Trinity-RFT has been designed with user-friendliness as a top priority. For development and research: The modular and decoupled design of Trinity-RFT allows users to develop a new algorithm for a specific aspect of RFT by adding one or a few new classes that implement the essential functionalities of interest, without concerning about other aspects of RFT or intrusive modifications of the original codebase. In addition, we include a monitor (built upon Wandb [41] and TensorBoard [38]) that makes it easy to track the progress of an RFT process, both quantitatively (e.g., via learning curves for rewards and other metrics) and qualitatively (e.g., via concrete examples of rollout trajectories generated at different RL steps). See Figure 6 for an example snapshot of the monitor. For RFT applications: Trinity-RFT offers extensive graphical user interfaces to support low-code usage of the framework, and to maximize transparency of the RFT process. For example, we implement a configuration manager, as shown in Figure 7, that allows the user to create configuration files conveniently via a front-end interface. We also provide Trinity-Studio, an all-in-one unified UI (including the aforementioned monitor and configuration manager) that allows the user to configure and run data inspection, data processing, RFT learning process, etc., all by clicking the mouse and filling forms, without writing any code. An example for using Trinity-Studio will be introduced in Section 3.6. Such functionalities, of course, can be useful not only for applications but also for development and research. <details> <summary>figs/wandb_screencut.png Details</summary>  ### Visual Description # Technical Document Extraction: Model Training Dashboard This document provides a comprehensive extraction of the data and visual components from the provided image, which appears to be a machine learning experiment tracking dashboard (likely Weights & Biases). ## 1. Header and Navigation * **Top Left Label:** `Tables 1` * **Top Right Actions:** `Add panel`, `Export as CSV`, `Columns...`, `Reset Table` * **Data Source Identifier:** `runs.summary["rollout_examples"]` --- ## 2. Main Data Table: `rollout_examples` This table displays specific model outputs (rollouts) during training. | Index | step | reward | prompt | response | | :--- | :--- | :--- | :--- | :--- | | **1** | 45 | 1.1 | `<|im_start|>system` <br> A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning | `<think>` Let's denote the current experience of Bill as \( B \) years and the current experience of Joan as \( J \) years. <br> 5 years ago, Joan's experience was \( J - 5 \) and | | **2** | 45 | -0.1 | `<|im_start|>system` <br> A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning | `<think>` Let's break it down step by step: <br> 1. Total points scored by the team: 311 <br> 2. Points combined by some players: 188 <br> 3. Points left for Lisa, Jessie, and Devin: 311 - 188 | | **3** | 46 | 1.1 | `<|im_start|>system` <br> A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning | `<think>` First, we need to calculate the total area of the courtyard. The courtyard is 10 feet by 25 feet, so the area is 10 * 25 = 250 square feet. Jackson needs 4 tiles per square foot, so he | **Table Footer Information:** * Pagination: `1 - 3 of 4` (indicating one additional record is not visible). --- ## 3. Selected Metrics (Charts) The bottom section, labeled `selected 2`, contains two line charts tracking performance over training steps. ### Chart A: `eval/accuracy/mean` * **X-Axis:** `Step` (Markers: 10, 15, 20, 25, 30, 35, 40) * **Y-Axis:** Accuracy (Markers: 0.55, 0.6, 0.65, 0.7) * **Line Color:** Blue * **Trend Verification:** The line shows a consistent upward trend with minor fluctuations. It starts at approximately 0.53 at step 10, dips slightly, then climbs steadily to a peak of approximately 0.71 at step 36, before leveling off slightly at step 40. * **Key Data Points (Approximate):** * Step 10: ~0.53 * Step 12: ~0.51 (Local minimum) * Step 20: ~0.63 * Step 28: ~0.68 * Step 36: ~0.71 (Peak) * Step 40: ~0.695 (Final point marked with a dot) ### Chart B: `critic/rewards/mean` * **X-Axis:** `Step` (Markers: 0, 10, 20, 30, 40) * **Y-Axis:** Reward Value (Markers: 0, 0.2, 0.4, 0.6, 0.8) * **Line Color:** Red/Brown * **Trend Verification:** The line shows a rapid initial increase followed by a plateau with high volatility. It starts near 0.5, climbs to 0.8 by step 15, and fluctuates between 0.8 and 0.95 for the remainder of the run. * **Key Data Points (Approximate):** * Step 0: ~0.52 * Step 5: ~0.45 (Local minimum) * Step 15: ~0.80 * Step 19: ~0.92 (Local peak) * Step 33: ~0.95 (Highest peak) * Step 40: ~0.85 (Final point marked with a dot) --- ## 4. Component Isolation Summary * **Header:** Contains table controls and the specific data key being queried. * **Main Content (Table):** Shows that the model is using a "Chain of Thought" (`<think>`) prompting style. Rewards vary between positive (1.1) and negative (-0.1) for different reasoning paths. * **Footer (Charts):** Indicates that as training steps increase, both the mean accuracy and the critic's mean reward are improving, suggesting successful model convergence. </details> Figure 6: A snapshot of the monitor implemented in Trinity-RFT. <details> <summary>figs/config_manager_beginner.jpg Details</summary>  ### Visual Description # Technical Document: Trinity-RFT Config Generator Interface This document provides a comprehensive extraction of the user interface for the **Trinity-RFT Config Generator**, a web-based configuration tool for machine learning experiments. ## 1. Header Section * **Main Title:** Trinity-RFT Config Generator * **Interactive Element:** A "link" icon is located to the right of the main title. * **Mode Selection (Tabs/Buttons):** * **Beginner Mode:** Currently active (indicated by a red border and red text). * **Expert Mode:** Inactive (indicated by a grey border and black text). ## 2. Main Configuration Section: "Essential Configs" This section contains the primary parameters required to generate a configuration file. ### Project Identification | Field Label | Input Value / Status | | :--- | :--- | | **Project** | `Trinity-RFT` | | **Experiment Name** | `qwen2.5-1.5B` | ### Path Configurations These fields currently contain validation warnings, indicating they are mandatory and currently empty. * **Model Path** * *Input Status:* Empty. * *Warning Message:* "Please input model path." (Displayed in a yellow alert box). * **Checkpoint Path** * *Input Status:* Empty. * *Warning Message:* "Please input checkpoint path." (Displayed in a yellow alert box). * **Taskset Path** * *Input Status:* Empty. * *Warning Message:* "Please input taskset path." (Displayed in a yellow alert box). ## 3. Parameter Settings (Footer Region) The bottom row contains specific algorithmic and monitoring settings. | Parameter Label | Control Type | Current Value | | :--- | :--- | :--- | | **Algorithm Type** | Dropdown Menu | `ppo` | | **SFT Warmup Steps** | Numeric Stepper | `0` (with `-` and `+` controls) | | **Monitor Type** | Dropdown Menu | `tensorboard` | --- **Technical Summary:** The interface is designed for a Reinforcement Learning from Fine-Tuning (RFT) workflow, specifically targeting the `qwen2.5-1.5B` model. The "Beginner Mode" restricts the user to "Essential Configs," focusing on file paths, the PPO (Proximal Policy Optimization) algorithm, and logging via TensorBoard. The presence of yellow warning boxes suggests a real-time validation system that prevents configuration generation until all path fields are populated. </details> (a) The “beginner” mode. <details> <summary>figs/config_manager_expert.jpg Details</summary>  ### Visual Description # Technical Document Extraction: Trinity-RFT Config Generator Interface This document provides a comprehensive extraction of the textual and structural data from the provided image of a web-based configuration interface. ## 1. Header and Mode Selection The top section of the interface contains the application title and global mode toggles. * **Main Title:** Trinity-RFT Config Generator (accompanied by a hyperlink icon). * **Mode Selection Buttons:** * **Beginner Mode:** (Inactive/Unselected) White background with grey border. * **Expert Mode:** (Active/Selected) White background with a red border and red text. --- ## 2. Navigation Tabs The interface uses a tabbed navigation system to categorize configuration settings. The "Model" tab is currently active. * **Model** (Active: Indicated by red text and a red underline). * **Buffer** * **Explorer and Synchronizer** * **Trainer** --- ## 3. Configuration Fields (Model Tab) The main body of the interface consists of several input fields, dropdowns, and numerical incrementors. ### A. Project Identification | Field Label | Current Value / Placeholder | | :--- | :--- | | **Project** | Trinity-RFT | | **Experiment Name** | qwen2.5-1.5B | ### B. Path Configurations These fields are currently empty and display validation warnings. * **Model Path:** [Empty Input Field] * *Warning Message:* "Please input model path." (Displayed in a yellow alert box). * **Critic Model Path (defaults to `model_path`):** [Empty Input Field] * **Checkpoint Path:** [Empty Input Field] * *Warning Message:* "Please input checkpoint path." (Displayed in a yellow alert box). ### C. Hardware and Monitoring This section uses a mix of dropdown menus and numerical steppers. * **Monitor Type:** `tensorboard` (Dropdown menu). * **Node Num:** `1` (Numerical stepper with `-` and `+` controls). * **GPU Per Node:** `8` (Numerical stepper with `-` and `+` controls). ### D. Token Limits Numerical constraints for the model's processing. * **Max Prompt Tokens:** `1024` (Numerical stepper with `-` and `+` controls). * **Max Response Tokens:** `1024` (Numerical stepper with `-` and `+` controls). --- ## 4. Component Analysis & Logic * **Language:** The interface is entirely in **English**. * **Validation Logic:** The interface employs real-time validation. The "Model Path" and "Checkpoint Path" are mandatory fields, as evidenced by the yellow warning blocks appearing immediately below the empty inputs. * **Defaulting Logic:** The "Critic Model Path" field explicitly states a fallback logic: if left blank, it inherits the value provided in the "Model Path" field. * **Visual Hierarchy:** The use of red for the "Expert Mode" and the active "Model" tab indicates the current state of the user's workflow. The numerical steppers suggest that the values for Nodes, GPUs, and Tokens are constrained to integer increments. </details> (b) The “expert” mode. Figure 7: Snapshots of the configuration manager. 3 Examples, Applications, and Experiments This section demonstrates the utilities and user-friendliness of Trinity-RFT and exemplifies some concepts introduced in previous sections, through a diverse range of examples, applications and experiments. Additional step-by-step tutorials can be found on the documentation website https://modelscope.github.io/Trinity-RFT, or the examples folder of the GitHub repository https://github.com/modelscope/Trinity-RFT/tree/main/examples. 3.1 Customizing Agent-Environment Interaction With a modular design, Trinity-RFT can be easily adapted to a new downstream scenario by implementing the logic of agent-environment interaction in a single workflow class, without modifications to other components of the codebase. This approach is also sufficient for macroscopic RL algorithm design that targets high-quality experience synthesis with environmental feedback [4]. We provide some concrete examples in the rest of this subsection. 3.1.1 Single-turn Scenarios In a simple yet common scenario, a user of Trinity-RFT would like to train an LLM for completing single-turn tasks, where the LLM generates one response to each input query. For this purpose, the user mainly needs to (1) define and register a single-turn workflow class (by inheriting the base class Workflow) tailored to the targeted tasks, and (2) specify the tasksets (for training and/or evaluation) and the initial LLM, both of which are compatible with HuggingFace [14] and ModelScope [19] formats. Listing 1 gives a minimal example for implementing a single-turn workflow. Suppose that each task is specified by a <question, answer> tuple. The run() method of ExampleWorkflow calls the LLM once to generate a response for the question, calculates its reward, and returns an Experience instance that consists of the response itself, the reward value, and the log-probabilities of response tokens predicted by the rollout model (which is necessary for certain RL algorithms, such as PPO [28] and GRPO [29]). Some built-in workflows and reward functions have been implemented in Trinity-RFT, e.g., the MathWorkflow class for math-related tasks. In some cases, the user wants to utilize auxiliary LLMs in the workflow, e.g., for computing rewards via LLM-as-a-judge, or for playing the roles of other agents in a multi-agent scenario. For these purposes, the user can specify auxiliary_models via APIs when initializing the workflow. ⬇ 1 @WORKFLOWS. register_module ("example_workflow") 2 class ExampleWorkflow (Workflow): 3 4 def __init__ ( 5 self, 6 model: ModelWrapper, 7 task: Task, 8 auxiliary_models: Optional [List [openai. OpenAI]] = None, 9 ): 10 super (). __init__ (model, task, auxiliary_models) 11 self. question = task. raw_task. get ("question") 12 self. answer = task. raw_task. get ("answer") 13 14 def calculate_reward_by_rule (self, response: str, truth: str) -> float: 15 return 1.0 if response == truth else 0.0 16 17 def calculate_reward_by_llm_judge (self, response: str, truth: str) -> float: 18 judge_model = self. auxiliary_models [0] 19 PROMPT_FOR_JUDGE = "" "Please evaluate..." "" 20 completion = judge_model. chat. completions. create ( 21 model = "gpt-4", # Or another suitable judge model 22 messages =[{"role": "user", "content": PROMPT_FOR_JUDGE}], 23 ) 24 reward_str = completion. choices [0]. message. content. strip () 25 reward = float (reward_str) 26 return reward 27 28 def run (self) -> List [Experience]: 29 response = self. model. chat ( 30 [ 31 { 32 "role": "user", 33 "content": f "Question:\n{self.question}", 34 } 35 ], 36 ** self. rollout_args, 37 ) 38 reward: float = self. calculate_reward_by_rule (response. response_text, self. answer) 39 # reward: float = self.calculate_reward_by_llm_judge(response.response_text, self.answer) 40 return [ 41 Experience ( 42 tokens = response. tokens, 43 prompt_length = response. prompt_length, 44 reward = reward, 45 logprobs = response. logprobs, 46 ) 47 ] Listing 1: A minimal example for implementing a customized workflow. 3.1.2 Multi-turn Scenarios In more advanced cases, the user would like to train an LLM-powered agent that solves multi-turn tasks by repeatedly interacting with the environment. In Trinity-RFT, achieving this is mostly as simple as in the single-turn case, except that the user needs to define and register a multi-turn workflow class by inheriting the base class MultiTurnWorkflow. Listing 2 provides one such example using the ALFWorld dataset [31]. For training efficiency, the process_messages_to_experience() method concatenates multiple rounds of agent-environment interactions compactly into an Experience instance consisting of a single token sequence with proper masking, which can readily be consumed by standard RL algorithms like PPO and GRPO. For more detailed examples of multi-turn cases, please refer to the documentation https://modelscope.github.io/Trinity-RFT/tutorial/example_multi_turn.html. ⬇ 1 @WORKFLOWS. register_module ("alfworld_workflow") 2 class AlfworldWorkflow (MultiTurnWorkflow): 3 "" "A workflow for the ALFWorld task." "" 4 5 def generate_env_inference_samples (self, env, rollout_num) -> List [Experience]: 6 print ("Generating env inference samples...") 7 experience_list = [] 8 for i in range (rollout_num): 9 observation, info = env. reset () 10 final_reward = -0.1 11 memory = [] 12 memory. append ({"role": "system", "content": AlfWORLD_SYSTEM_PROMPT}) 13 for r in range (self. max_env_steps): 14 format_obs = format_observation (observation) 15 memory = memory + [{"role": "user", "content": format_obs}] 16 response_text = self. model. chat (memory, n =1)[0]. response_text 17 memory. append ({"role": "assistant", "content": response_text}) 18 action = parse_action (response_text) 19 observation, reward, done, info = env. step (action) 20 if done: 21 final_reward = reward 22 break 23 experience = self. process_messages_to_experience ( 24 memory, final_reward, {"env_rounds": r, "env_done": 1 if done else 0} 25 ) 26 experience_list. append (experience) 27 # Close the env to save CPU memory 28 env. close () 29 return experience_list 30 31 def run (self) -> List [Experience]: 32 # ... 33 game_file_path = self. task_desc 34 rollout_n = self. repeat_times 35 # ... 36 env = create_environment (game_file_path) 37 return self. generate_env_inference_samples (env, rollout_n) Listing 2: An implementation of a multi-turn workflow for ALFWorld [31]. 3.1.3 Experience Synthesis in Workflows As mentioned in Section 1.1, Trinity-RFT has been designed to streamline RL algorithm design and development at both macroscopic and microscopic levels. One example for the former is experience synthesis: at each RL step, the agent (backed by the rollout LLM) iteratively generates refined responses to a query by incorporating feedback or guidance from the environment, which can be in the form of plain text rather than numerical reward values. The resulting data will then be utilized for updating the policy model, e.g., by a standard SFT or RL loss. Such a macroscopic RL approach is made possible by pre-trained LLMs’ generative nature and rich prior knowledge about natural language. Closely related to this idea is Agent-RLVR [4], a contemporary work that applies such an approach to software engineering scenarios. Within Trinity-RFT, this process of experience synthesis can be regarded as a particular way of agent-environment interaction, and thus can be realized by simply implementing a Workflow class. As a minimal demonstration, suppose that we want to implement this approach for a math reasoning scenario, where the agent generates multiple rollout responses to an input query, receives feedback from the environment regarding correctness of the responses, reflects on the gathered information, and generates a final response to the query. Listing 3 presents an implementation of this approach within Trinity-RFT. ⬇ 1 @WORKFLOWS. register_module ("reflect_once_workflow") 2 class ReflectOnceWorkflow (Workflow): 3 4 def run (self) -> List [Experience]: 5 experiences = [] 6 7 # Stage 1: K-rollout generation 8 rollout_messages = self. create_rollout_messages () 9 responses = self. model. chat ( 10 rollout_messages, 11 n = self. k_rollouts, 12 temperature = self. temperature, 13 logprobs = self. logprobs, 14 max_tokens = self. task. rollout_args. max_tokens, 15 ) 16 rollout_responses = [response. response_text. strip () for response in responses] 17 18 # Stage 2: Verification 19 verification_results = [] 20 for rollout_response in rollout_responses: 21 is_correct = self. verify_answer (rollout_response, self. ground_truth) 22 verification_results. append (is_correct) 23 24 # Stage 3: Reflection 25 reflection_messages = self. create_reflection_messages ( 26 rollout_responses, 27 verification_results, 28 ) 29 reflection_responses = self. model. chat ( 30 reflection_messages, 31 n =1, 32 temperature = self. temperature, 33 logprobs = self. logprobs, 34 max_tokens = self. task. rollout_args. max_tokens, 35 ) 36 reflection_response = reflection_responses [0] 37 38 # Verify the reflection response 39 reflection_text = reflection_response. response_text. strip () 40 reflection_is_correct = self. verify_answer (reflection_text, self. ground_truth) 41 42 if reflection_is_correct: 43 sharegpt_message = [ 44 { 45 "role": "system", 46 "content": self. task. format_args. system_prompt 47 }, 48 { 49 "role": "user", 50 "content": self. question 51 }, 52 { 53 "role": "assistant", 54 "content": reflection_text 55 } 56 ] 57 experience = self. process_messages_to_experience (sharegpt_message) 58 experiences. append (experience) 59 60 # Save experience to file 61 if self. exp_file and sharegpt_message is not None: 62 exp_data = sharegpt_message 63 self. exp_file. write (json. dumps (exp_data, ensure_ascii = False) + "\n") 64 self. exp_file. flush () 65 return experiences Listing 3: A toy implementation of experience synthesis with environmental feedback. 3.2 RL Algorithm Development with Trinity-RFT To support RL algorithm development, Trinity-RFT allows researchers and developers to focus on designing and implementing the essential logic of a new RL algorithm, without the need to care about the internal engineering details about Trinity-RFT. As an example, suppose that we want to implement a MIX algorithm that seamlessly integrates online RL and offline SFT into a single learning process. In its most basic form, the MIX algorithm requires that (1) the trainer samples from two sources of experiences, i.e., the rollout experiences collected online and the high-quality expert trajectories collected offline; and (2) the trainer updates its policy model with a loss function that handles both sources of experiences properly, e.g., a weighted sum of GRPO loss for the on-policy rollout experiences and SFT loss for the expert trajectories. Variants of this MIX algorithm include adaptive weighting of multiple loss terms [10], alternating between RL and SFT [16], incorporating expert trajectories into RL loss [21, 34, 46], or incorporating SFT loss for high-reward rollout trajectories generated by older versions of the rollout model [27]. Such approaches have proved to be effective in accelerating the online RL process with only a small amount of expert experiences, or to enhance stability and plasticity in continual learning. <details> <summary>x7.png Details</summary>  ### Visual Description # Technical Document Extraction: Reinforcement Learning Training Pipeline This image illustrates a technical architecture for a machine learning training pipeline, specifically focusing on a hybrid learning approach combining Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). The diagram is segmented into three primary functional regions: **Explorer**, **Buffer**, and **Trainer**. --- ## 1. Component Isolation ### Region A: Explorer (Left, Yellow Background) This region represents the data generation phase. * **Icon:** A robot head. * **Components:** * **Rollout engine:** A grey rounded rectangular block that executes tasks to generate data. * **Internal Flow:** * Receives a **Task** (represented by a clipboard icon) from the **Taskset** in the Buffer region. * The process of selecting a task is labeled **Sampling**. * Outputs **Experience** (represented by a document icon) to the Buffer region. ### Region B: Buffer (Center, Blue Background) This region acts as the data storage and retrieval layer. * **Icon:** A database/server stack. * **Components:** * **Usual Experiences:** A purple cylinder (database icon). * **Expert Experiences:** A blue cylinder (database icon). * **Taskset:** A white cylinder (database icon) located at the bottom. * **Internal Flow:** * Receives "Experience" from the Explorer into the **Usual Experiences** storage. * Data is retrieved from both "Usual Experiences" and "Expert Experiences" via a process labeled **Sampling**. * The sampled data is represented by two document icons (one purple, one blue) being sent to the Trainer. ### Region C: Trainer (Right, Green Background) This region represents the model optimization and update phase. * **Icon:** A human head with gears inside. * **Components:** * **SFT Loss:** A purple rounded rectangular block. * **GRPO Loss:** A blue rounded rectangular block. * **Summation Operator (+):** A blue circle with a plus sign. * **MIX Loss:** A grey rounded rectangular block. * **Update model:** A grey rounded rectangular block (final output). * **Internal Flow:** * The purple document icon (Usual/Expert data) feeds into the **SFT Loss** calculation. * The blue document icon feeds into the **GRPO Loss** calculation. * Both loss values are combined at the **(+)** operator. * The combined result flows into **MIX Loss**. * The final step is the **Update model** action, triggered by the MIX Loss. --- ## 2. Data Flow and Logic Summary The system operates in a continuous loop or pipeline: 1. **Task Selection:** The **Explorer** samples a **Task** from the **Taskset** in the **Buffer**. 2. **Experience Generation:** The **Rollout engine** processes the task to generate new **Experience**, which is stored in the **Usual Experiences** buffer. 3. **Data Sampling:** The **Trainer** samples data from both the **Usual Experiences** and **Expert Experiences** buffers. 4. **Loss Calculation:** * **SFT Loss** (Supervised Fine-Tuning) is calculated, likely using the Expert or high-quality data. * **GRPO Loss** (Group Relative Policy Optimization) is calculated, typically used for reinforcement learning from feedback. 5. **Optimization:** The two losses are aggregated into a **MIX Loss**, which is then used to **Update model**. --- ## 3. Textual Transcription | Category | Transcribed Text | | :--- | :--- | | **Headers** | Explorer, Buffer, Trainer | | **Process Labels** | Sampling, Sampling | | **Data Objects** | Experience, Task, Usual Experiences, Expert Experiences, Taskset | | **Functional Blocks** | Rollout engine, SFT Loss, GRPO Loss, MIX Loss, Update model | | **Symbols** | + (Addition/Aggregation) | **Language Declaration:** All text in this image is in **English**. </details> Figure 8: A visualization of the MIX algorithm. The MIX algorithm is visualized in Figure 8, where we integrate GRPO loss for usual experiences generated by the rollout model and SFT loss for expert experiences into a unified training pipeline. It requires dealing with different sources of experiences, and two types of loss functions; fortunately, to implement such an algorithm in Trinity-RFT, we only need to define three new classes — MixSampleStrategy, MIXPolicyLossFn, and MIXAlgorithm — as demonstrated in Listing 4. With these components, Trinity-RFT enables a seamless integration of online RL and offline SFT within the same training loop. More details of the MIX algorithm are referred to the documentation https://modelscope.github.io/Trinity-RFT/tutorial/example_mix_algo.html. ⬇ 1 @SAMPLE_STRATEGY. register_module ("mix") 2 class MixSampleStrategy (SampleStrategy): 3 def __init__ (self, buffer_config: BufferConfig, trainer_type: str, ** kwargs): 4 # ... 5 self. usual_exp_buffer = get_buffer_reader ( 6 buffer_config. trainer_input. experience_buffer, usual_buffer_config 7 ) 8 self. expert_exp_buffer = get_buffer_reader ( 9 buffer_config. trainer_input. sft_warmup_dataset, expert_buffer_config 10 ) 11 # ... 12 13 def sample (self, step: int) -> Tuple [Any, Dict, List]: 14 "" "Sample a batch composed of rollout experiences and expert trajectories" "" 15 usual_exp_list = self. usual_exp_buffer. read () 16 expert_exp_list = self. expert_exp_buffer. read () 17 exp_list = usual_exp_list + expert_exp_list 18 exps = Experiences. gather_experiences (exp_list, self. pad_token_id) 19 # ... 20 21 22 @POLICY_LOSS_FN. register_module ("mix") 23 class MIXPolicyLossFn (PolicyLossFn): 24 def __init__ (self, mu: float = 0.1, ...): 25 # ... 26 self. mu = mu 27 self. grpo_loss_fn = PPOPolicyLossFn (...) 28 self. sft_loss_fn = SFTLossFn (...) 29 30 def __call__ ( 31 self, 32 logprob: torch. Tensor, 33 old_logprob: torch. Tensor, 34 action_mask: torch. Tensor, 35 advantages: torch. Tensor, 36 is_expert_mask: torch. Tensor, 37 ** kwargs, 38 ) -> Tuple [torch. Tensor, Dict]: 39 "" "Calculate a weighted sum of GRPO loss and SFT loss" "" 40 # ... 41 grpo_loss, grpo_metrics = self. grpo_loss_fn ( 42 logprob [~ is_expert_mask], 43 old_logprob [~ is_expert_mask], 44 action_mask [~ is_expert_mask], 45 advantages [~ is_expert_mask], 46 ** kwargs, 47 ) 48 sft_loss, sft_metrics = self. sft_loss_fn ( 49 logprob [is_expert_mask], 50 action_mask [is_expert_mask], 51 ) 52 loss = (1 - self. mu) * grpo_loss + self. mu * sft_loss 53 # ... 54 return loss, metrics 55 56 @ALGORITHM_TYPE. register_module ("mix") 57 class MIXAlgorithm (AlgorithmType): 58 "" "MIX algorithm." "" 59 60 use_critic: bool = False 61 use_reference: bool = True 62 use_advantage: bool = True 63 can_balance_batch: bool = True 64 schema: type = ExperienceModel 65 66 @classmethod 67 def default_config (cls) -> Dict: 68 return { 69 "repeat_times": 8, 70 "policy_loss_fn": "mix", # Specify the MIX loss function 71 "advantage_fn": "grpo", 72 "sample_strategy": "mix", # Specify the MIX sampling strategy 73 } Listing 4: An implementation of the MIX algorithm with Trinity-RFT. 3.3 Unified Support for Diverse RL Modes As explained previously in Section 2.1.1, Trinity-RFT offers support for synchronous/asynchronous, on-policy/off-policy, and online/offline RL, controlled by a few configuration parameters. In this subsection, we conduct experiments for comparing the following RL modes: - The synchronous mode: mode=both, sync_interval={1,2,10}, sync_offset=0; - The one-step off-policy mode: mode=both, sync_interval=1, sync_offset=1; - The fully asynchronous mode: the explorer and trainer are launched with mode=explore and train respectively, with sync_interval=10. Our experiments include dummy learning processes (which will soon be explained) for performance profiling, as well as real learning processes with vanilla GRPO [29] in different modes. 3.3.1 Experiments: Performance Profiling Settings. We aim to measure and compare the efficiency of different RL modes under controlled settings. It is noteworthy that, even with all other variables controlled, different RL modes can still result in different trained models — and thus different rollout response lengths — during the learning processes, which have direct impacts on performance metrics like wall-clock time and GPU utilization rate. To mitigate this, we conduct performance profiling with dummy learning processes, where the learning rate is set to zero. A dummy learning process closely resembles a real learning process, in that all necessary computation and communication (e.g., rollout generation, gradient computation, model weight synchronization) are executed; the only difference is that the rollout model (and thus the distribution of rollout trajectories) remains fixed throughout and same across different RL modes. To show the performance of Trinity-RFT in diverse scenarios, we consider both math reasoning task (GSM8k [3]) and multi-turn agentic task (ALFWorld [31]). In the experiments, we use the Qwen2.5-Instruct [36] model series of different sizes (1.5B, 3B, and 7B), and run the GRPO [5] algorithm (with 8 rollout trajectories per task) in all modes. We choose a 100-step training trace to evaluate the performance and report the following metrics: (1) end-to-end wall-clock time and time speedup: the wall-clock time from the start of running the command to the end of finishing 100 training steps; (2) GPU utilization: the GPU utilization in percent for each GPU; (3) GPU power usage: the GPU power usage as a percentage of its power capacity for each GPU. Metrics for GPU utilization and power usage were extracted from WandB https://docs.wandb.ai/guides/models/app/settings-page/system-metrics/ and averaged over all GPUs. We run each experiment for three random trials and report the mean and standard deviation. Unless specified otherwise, each experiment uses 8 NVIDIA A100-80G GPUs. Results for GSM8k. We use the 2/6 GPU partition for explorer/trainer. While this configuration is not the optimal one for all experiments, it is sufficient to show the difference between several RL modes. In our GSM8k experiments, the batch size is 96 tasks, and the temperature is 1.0. The results for both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models are shown in Table 1. We observe that in the synchronous mode (with sync_offset=0), a less frequent synchronization (i.e., a larger sync_interval) effectively improves the efficiency, GPU utilization, and GPU power usage. This is mainly because, as shown in Figure 4 (a), the impacts of pipeline bubbles in this mode can be effectively reduced by using a lower synchronization frequency. Besides, Table 1 shows that one-step off-policy and fully asynchronous modes also accelerate the training process with higher GPU utilization, compared to a strictly on-policy mode. In one-step off-policy mode, the trainer leverages the one-step off-policy data stored in the buffer without needing to wait for new experiences generated by the explorer after weight synchronization, which significantly reduces the GPU idle ratio. In fully asynchronous mode, the explorer and trainer operate almost independently while fully leveraging GPU resources, except when loading or saving model checkpoints. Table 1: Performance profiling for GSM8k with 2/6 GPU partition for explorer/trainer. | Mode | Qwen2.5-1.5B-Instruct | | | | | --- | --- | --- | --- | --- | | Speedup $\uparrow$ | Time (Minutes) $\downarrow$ | GPU Utilization (%) $\uparrow$ | GPU Power Usage (%) $\uparrow$ | | | Sync. (sync_interval=1) | $1.00×$ | $38.70± 0.34$ | $33.64± 2.15$ | $35.85± 1.83$ | | Sync. (sync_interval=2) | $1.24×$ | $31.19± 0.08$ | $36.05± 0.49$ | $38.74± 1.47$ | | Sync. (sync_interval=10) | $1.59×$ | $24.28± 0.16$ | $\mathbf{38.27}± 0.98$ | $\mathbf{44.41}± 0.81$ | | One-step off-policy | $1.25×$ | $30.84± 0.20$ | $32.39± 1.17$ | $39.70± 0.78$ | | Fully async. | $\mathbf{1.61}×$ | $\mathbf{23.97}± 0.03$ | $36.04± 0.61$ | $43.91± 0.48$ | | Mode | Qwen2.5-7B-Instruct | | | | | --- | --- | --- | --- | --- | | Speedup $\uparrow$ | Time (Minutes) $\downarrow$ | GPU Utilization (%) $\uparrow$ | GPU Power Usage (%) $\uparrow$ | | | Sync. (sync_interval=1) | $1.00×$ | $68.71± 0.54$ | $55.61± 0.80$ | $52.88± 0.29$ | | Sync. (sync_interval=2) | $1.31×$ | $52.44± 0.41$ | $64.88± 1.35$ | $61.90± 1.32$ | | Sync. (sync_interval=10) | $\mathbf{1.85}×$ | $\mathbf{37.17}± 0.15$ | $\mathbf{78.44}± 1.03$ | $\mathbf{77.77}± 0.96$ | | One-step off-policy | $1.69×$ | $40.73± 0.57$ | $77.19± 2.26$ | $76.17± 1.56$ | | Fully async. | $1.63×$ | $42.17± 1.06$ | $73.90± 2.00$ | $72.74± 1.82$ | Table 2: Performance profiling for ALFWorld with 4/4 GPU partition for explorer/trainer. | Mode | Batch_Size = 4 | | | | | --- | --- | --- | --- | --- | | Speedup $\uparrow$ | Time (Minutes) $\downarrow$ | GPU Utilization (%) $\uparrow$ | GPU Power Usage (%) $\uparrow$ | | | Sync. (sync_interval=1) | $1.00×$ | $333.68± 0.06$ | $17.19± 0.58$ | $28.44± 0.37$ | | Sync. (sync_interval=2) | $1.70×$ | $196.64± 0.59$ | $21.69± 0.18$ | $31.35± 0.06$ | | Sync. (sync_interval=10) | $5.21×$ | $64.09± 0.39$ | $32.85± 0.18$ | $40.86± 0.58$ | | One-step off-policy | $0.98×$ | $340.12± 3.99$ | $14.63± 1.17$ | $28.21± 0.48$ | | Fully async. | $\mathbf{5.45}×$ | $\mathbf{61.27}± 0.35$ | $\mathbf{36.46}± 0.10$ | $\textbf{42.51}± 0.72$ | | Mode | Batch_Size = 32 | | | | | --- | --- | --- | --- | --- | | Speedup $\uparrow$ | Time (Minutes) $\downarrow$ | GPU Utilization (%) $\uparrow$ | GPU Power Usage (%) $\uparrow$ | | | Sync. (sync_interval=1) | $1.00×$ | $561.21± 2.04$ | $39.37± 0.89$ | $39.93± 0.22$ | | Sync. (sync_interval=2) | $1.13×$ | $496.80± 0.36$ | 37.74 $± 0.39$ | $41.90± 0.44$ | | Sync. (sync_interval=10) | $1.59×$ | $352.11± 0.49$ | $44.50± 0.58$ | $49.95± 0.61$ | | One-step off-policy | $1.14×$ | $494.13± 0.28$ | $34.89± 0.75$ | $43.05± 0.81$ | | Fully async. | $\mathbf{1.65}×$ | $\mathbf{339.51}± 0.24$ | $\mathbf{45.55}± 0.20$ | $\textbf{50.77}± 0.45$ | Results for ALFWorld. A particular feature of ALFWorld is the long-horizon multi-turn interaction with the environment. To accommodate the heavy computational demands in rollout, we use the 4/4 GPU partition for explorer/trainer. In our ALFWorld experiments, we use the Qwen2.5-3B-Instruct model and set the rollout temperature to 1.0. The results with different batch sizes are shown in Table 2. One observation is that, when the batch size is 4 tasks, the one-step off-policy mode exhibits no efficiency advantage over the synchronous mode with sync_interval=1. This phenomenon can be attributed to the computational imbalance between the explorer and trainer. In ALFWorld, the larger computation latency of the explorer emerges primarily from (1) the inherent complexity of multi-turn environment interactions, and (2) the long-tailed latency distribution when certain tasks require extended rollout durations, whose effect is further exacerbated by a small batch size. The one-step off-policy mode cannot eliminate pipeline bubbles caused by long-tailed latencies in the explorer, whereas this can be mitigated by the synchronous mode with a large sync_interval as well as by the asynchronous mode, thanks to the implementation of streaming rollout generation in Trinity-RFT. Another observation, due to the same reason, is that when scaling the batch size from 4 to 32, all modes incur a small increase (much smaller than $8×$ ) in wall-clock time for the same number of training steps, thanks to better GPU usage. 3.3.2 Experiments: Real Learning with Vanilla GRPO Settings. We aim to compare the real learning processes by different RL modes. For simplicity and controlled variability, we use the vanilla GRPO [29] algorithm for all RL modes, without specific algorithm design for asynchronous or off-policy cases. GRPO mainly relies on the mechanism of clipping probability ratio (defaults to the range of $1± 0.2$ ) to handle the off-policyness of experiences. For future works, we will investigate more advanced off-policy or asynchronous RL algorithms, and develop new ones to accommodate diverse RL modes. To encourage the exploration of the rollout model, we disable the Kullback-Leibler (KL) penalty or loss in our experiments. Training. For each RL mode, we fine-tune the Qwen2.5-7B-Instruct model for one epoch on the OpenR1-Math-46k [46] https://huggingface.co/datasets/Elliott/Openr1-Math-46k-8192 dataset, a filtered version of the OpenR1-Math-220k https://huggingface.co/datasets/open-r1/OpenR1-Math-220k dataset. The allocated GPU ratio for the explorer and trainer is 4/4. We set the batch size to 120 tasks, the rollout number per task to 32, and the learning rate to 1e-6. Evaluation. For each RL mode, we save the checkpoint of the rollout model once every 100 steps, evaluate the checkpoints using the bench mode, and report the best results among the checkpoints. The models are evaluated on several math benchmarks, including AIME2024 https://huggingface.co/datasets/math-ai/aime25, AIME2025 https://huggingface.co/datasets/math-ai/aime25, AMC https://huggingface.co/datasets/math-ai/amc23, and MATH500 https://huggingface.co/datasets/HuggingFaceH4/MATH-500. For AIME2024, AIME2025, and AMC, we generate 32 responses (with temperature 0.6) per task and report the average accuracy (Avg@32), to ensure the accuracy of the evaluation process; for MATH500, we report Avg@4 as the dataset is relatively large. For more detailed comparisons, we also plot some training metrics, including reward, response length, gradient norm, and KL distance from the initial LLM, with the wall-clock time as the X-axis. Results. Figure 9 presents the training curves. It is observed that several RL modes show increasing trends in terms of rewards and response lengths. The synchronous mode with sync_interval=1 exhibits longer responses and larger KL divergence than other RL modes, likely because it updates the rollout model most frequently and leverages on-policy data in each step. Table 3 presents the evaluation results. We observe that, for the synchronous mode with sync_offset=0, increasing sync_interval reduces the total training time for one epoch, at the cost of slightly compromising the average evaluation performance. In contrast, the one-step off-policy mode with sync_interval=1 achieves comparable or better performance than the other modes on several benchmarks, while achieving around $2.66×$ speedup in wall-clock time over the strictly on-policy mode. <details> <summary>x8.png Details</summary>  ### Visual Description # Technical Data Extraction: Training Metrics Comparison This document provides a comprehensive extraction of data from a series of four line charts comparing different synchronization intervals and off-policy methods in a machine learning training context. ## 1. Metadata and Global Legend * **Image Type:** Multi-panel line chart (4 subplots). * **Language:** English. * **Legend Location:** Top center, spanning the width of the image. * **Data Series (Legend):** 1. **Blue Line:** `Sync. (sync_interval=1)` 2. **Green Line:** `Sync. (sync_interval=2)` 3. **Red Line:** `Sync. (sync_interval=10)` 4. **Purple Line:** `One-Step Off-Policy` --- ## 2. Component Analysis (Subplots) All subplots share a common X-axis: **Time (hours)**, ranging from 0 to approximately 130. ### Subplot A: Reward * **Y-Axis Range:** 0.40 to 0.50+ * **Trend Analysis:** * **Sync. (interval=1) [Blue]:** Shows the longest duration. It has a volatile upward trend, peaking near 0.53 around hour 120 before a slight dip. * **Sync. (interval=2) [Green]:** Rapid initial climb, stabilizing around 0.48–0.50 before ending at hour 75. * **Sync. (interval=10) [Red]:** Steepest initial climb, reaching ~0.52 by hour 40, then terminating. * **One-Step Off-Policy [Purple]:** Similar trajectory to interval=10, reaching ~0.51 by hour 45. ### Subplot B: Response Length * **Y-Axis Range:** 1000 to 2500 * **Trend Analysis:** * **Sync. (interval=1) [Blue]:** Exhibits significant fluctuations. It rises to 2250 (hour 50), drops to 1750 (hour 75), then climbs to a peak of 2500 (hour 110). * **Sync. (interval=2) [Green]:** Steady, linear-like increase from 800 to ~1750 over 75 hours. * **Sync. (interval=10) [Red]:** Slowest growth, plateauing around 1000 by hour 40. * **One-Step Off-Policy [Purple]:** Moderate growth, reaching ~1300 by hour 45. ### Subplot C: Gradient Norm * **Y-Axis Range:** 0.08 to 0.16 * **Trend Analysis:** * **Sync. (interval=1) [Blue]:** Highly volatile. Starts high (~0.12), fluctuates between 0.08 and 0.13, and ends at its lowest point (~0.08) after hour 100. * **Sync. (interval=2) [Green]:** Fluctuates between 0.08 and 0.12, ending near 0.09 at hour 75. * **Sync. (interval=10) [Red]:** Starts at 0.12, drops and stabilizes around 0.09–0.10. * **One-Step Off-Policy [Purple]:** Starts with a massive spike at 0.16, then settles into the 0.10–0.12 range. ### Subplot D: KL Divergence * **Y-Axis Range:** 0.0 to 0.5 * **Trend Analysis:** * **Sync. (interval=1) [Blue]:** Shows a massive spike to 0.52 at hour 50, followed by a sharp drop and stabilization around 0.25 from hour 100 onwards. * **Sync. (interval=2) [Green]:** Steady upward slope, reaching 0.20 by hour 75. * **Sync. (interval=10) [Red]:** Very low, nearly flat growth, staying below 0.05. * **One-Step Off-Policy [Purple]:** Moderate upward slope, reaching ~0.18 by hour 45. --- ## 3. Summary Data Table (Approximate Values) | Metric | Sync (Int=1) [Blue] | Sync (Int=2) [Green] | Sync (Int=10) [Red] | One-Step Off-Policy [Purple] | | :--- | :--- | :--- | :--- | :--- | | **Max Time (h)** | ~130 | ~75 | ~40 | ~45 | | **Final Reward** | ~0.51 | ~0.49 | ~0.52 | ~0.50 | | **Final Resp. Len** | ~2200 | ~1750 | ~950 | ~1300 | | **Final Grad Norm** | ~0.08 | ~0.09 | ~0.10 | ~0.11 | | **Final KL Div.** | ~0.24 | ~0.20 | ~0.04 | ~0.17 | ## 4. Key Observations 1. **Training Duration:** The `Sync. (sync_interval=1)` configuration is the only one that runs for the full 130-hour duration shown. 2. **Instability:** The `sync_interval=1` (Blue) method shows high instability in Response Length and KL Divergence around the 50-hour mark, suggesting a significant policy shift or instability during that phase of training. 3. **Efficiency:** Higher sync intervals (Red/Green) appear to reach higher rewards faster but were terminated earlier in this visualization. </details> Figure 9: Results of training for one epoch by vanilla GRPO in different RL modes. The results are smoothed using a 40-step moving average for clarity. Table 3: Performance comparison of different RL modes. | | AIME2024 | AIME2025 | AMC | MATH500 | Average | Runtime (Hours) | | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-7B-Instruct | 11.15 | 6.95 | 51.35 | 70.96 | 35.10 | N/A | | Sync. (sync_interval=1) | 14.58 | 14.06 | 61.25 | 76.25 | 41.54 | 130.33 | | Sync. (sync_interval=2) | 15.52 | 11.67 | 57.97 | 75.15 | 40.08 | 73.57 | | Sync. (sync_interval=10) | 14.38 | 12.71 | 57.66 | 75.05 | 39.95 | 44.43 | | One-Step Off-Policy | 16.88 | 12.19 | 59.92 | 74.55 | 40.89 | 48.98 | 3.4 Data Processors for Tasks and Experiences We present practical examples to demonstrate how the data pipeline concepts described in Section 2.3 are applied in Trinity-RFT. These use cases highlight how manipulating data at both the task and experience level directly improves RFT performance and provides granular control over the agent’s learning process. 3.4.1 Static Task Prioritization for Curriculum Learning A common strategy for effective training is to present tasks in increasing order of difficulty. This use case demonstrates how Trinity-RFT facilitates curriculum learning by prioritizing tasks before exploration begins. This is particularly crucial for RFT, as it helps stabilize the initial learning phase of the explorer and prevents it from getting stuck on overly complex tasks, leading to a more efficient exploration path. As shown in Listing 5, a user can configure this pipeline with a simple YAML file The full configuration files can be accessed at baseline_run and priority_run.. Here, we use the GSM8K mathematical reasoning dataset. The user provides a natural language instruction via dj_process_desc: “Please compute difficulty scores for these math questions.”. Trinity-RFT ’s data service then orchestrates a three-phase process: 1. The data processor invokes an LLM (Qwen-Max) to score the difficulty of each math problem. 1. The system prioritizes samples with lower difficulty scores, creating an easy-to-hard ordering (by setting ‘priority_weights["difficulty"]: -1.0’). 1. The curated and prioritized data is formatted into an RL-ready task set for the explorer. As shown in Figure 10, this simple curation strategy yields more stable performance gains. This pattern is highly extensible: users can easily customize the difficulty metric, apply it to their own datasets, or even make the prioritization dynamic by re-ranking tasks periodically based on the agent’s current performance. ⬇ 1 # Core dataset configuration 2 data_processor: 3 data_workflow_url: "http://127.0.0.1:5005/data_workflow" 4 task_pipeline: 5 # I/O buffers 6 input_buffers: 7 - name: "raw_input" 8 path: "openai/gsm8k" 9 storage_type: "file" 10 raw: true 11 output_buffer: 12 name: "raw_output" 13 path: "outputs/task_pipeline_output/prioritized_gsm8k.jsonl" 14 storage_type: "file" 15 format: 16 prompt_key: "question" 17 response_key: "answer" 18 # data active iterator related 19 dj_process_desc: "Please compute difficulty scores for these math questions." 20 agent_model_name: "qwen-max" 21 clean_strategy: "iterative" 22 priority_weights: 23 difficulty: -1.0 # easy-to-hard Listing 5: Data processor configuration, applied on customizable buffers. <details> <summary>figs/data-pipelines/data-flow-static-priority-res.jpg Details</summary>  ### Visual Description # Technical Data Extraction: Training Metrics Dashboard This document provides a detailed extraction of data from three line charts representing machine learning training metrics. The dashboard consists of three panels: Evaluation Accuracy, Entropy Loss, and KL Loss. Each chart contains two data series: * **Red Line:** Series A * **Teal/Blue Line:** Series B --- ## 1. Chart: eval/math-eval/accuracy/mean **Description:** This chart tracks the mean accuracy on a mathematical evaluation set over time. * **X-Axis:** Time/Steps (Markers at 5, 10, 15, 20, 25, 30). * **Y-Axis:** Accuracy (Markers at 0.25, 0.3, 0.35). ### Data Trends and Points * **Teal Line (Series B):** * **Trend:** Rapid initial ascent, peaking early, followed by a plateau and a slight decline. * **Key Points:** Starts near 0.21. Sharp rise to ~0.33 at step 6. Peaks at ~0.35 at step 12. Drops to ~0.32 at step 18 and remains flat until step 30, ending with a slight dip to ~0.31. * **Red Line (Series A):** * **Trend:** Steady initial ascent, overtaking the teal line mid-run, and maintaining a higher final accuracy. * **Key Points:** Starts at ~0.24. Rises to ~0.33 at step 6. Continues a slower climb to peak at ~0.355 at step 18. Maintains ~0.35 until step 25, dips to ~0.335 at step 30, and recovers to ~0.35 at the final step. --- ## 2. Chart: actor/entropy_loss **Description:** This chart tracks the entropy loss of the actor model, typically used to measure the randomness of actions. * **X-Axis:** Time/Steps (Markers at 5, 10, 15, 20, 25, 30). * **Y-Axis:** Loss Value (Markers at 0.05, 0.1, 0.15, 0.2). ### Data Trends and Points * **Teal Line (Series B):** * **Trend:** Sharp initial decrease, followed by a period of oscillation between 0.06 and 0.10, ending with a significant drop to a low stable floor. * **Key Points:** Starts above 0.25 (off-chart). Drops to ~0.13 by step 5. Oscillates around 0.08 between steps 10 and 23. After step 25, it drops sharply to ~0.02 and stabilizes near 0.015. * **Red Line (Series A):** * **Trend:** Sharp initial decrease, followed by persistent oscillation with higher variance than the teal line, maintaining a higher final entropy. * **Key Points:** Starts at ~0.24. Drops to ~0.11 by step 4. Oscillates frequently between 0.05 and 0.12. It shows a notable peak at step 22 (~0.12). Ends the run higher than the teal line at ~0.06. --- ## 3. Chart: actor/kl_loss **Description:** This chart tracks the Kullback–Leibler (KL) divergence loss, measuring how much the policy changes between updates. * **X-Axis:** Time/Steps (Markers at 5, 10, 15, 20, 25, 30). * **Y-Axis:** Loss Value (Markers at 0, 0.2, 0.4, 0.6, 0.8, 1, 1.2). ### Data Trends and Points * **Teal Line (Series B):** * **Trend:** Slow initial start, followed by a massive surge/spike in the middle of the training run, then a gradual decline toward a moderate level. * **Key Points:** Remains below 0.2 until step 5. Rises to ~0.5 at step 8. Experiences a major surge starting at step 13, peaking at ~1.05 between steps 18 and 22. Gradually declines to ~0.55 by the end of the run. * **Red Line (Series A):** * **Trend:** Rapid initial rise to a stable plateau with minor fluctuations; significantly more stable than the teal line. * **Key Points:** Sharp rise from 0 to ~0.5 at step 5. For the remainder of the run (steps 5 through 35), it fluctuates narrowly between 0.4 and 0.55. It ends the run at ~0.4. --- ## Summary of Visual Components * **Header:** Each panel contains a title (e.g., `actor/kl_loss`) and a toolbar with four icons: "Expand/Fit", "Pin", "Show Grid/Fullscreen", and "Menu (three dots)". * **Main Chart:** Cartesian coordinate system with light grey grid lines. * **Footer:** X-axis labels and a resize handle (diagonal lines) in the bottom right corner of each panel. * **Language:** All text is in English. </details> Figure 10: Performance on a math reasoning task. Prioritizing tasks from easy to hard (red line) leads to faster and better convergence compared to the default setting (blue line). 3.4.2 Dynamic Experience and Reward Shaping While task curation primes the model before exploration, experience shaping refines the learning signal after each agent-environment interaction. This is critical for RFT algorithms that rely on rich feedback, as standard rewards (e.g., binary pass/fail) are often too sparse to guide learning effectively. We demonstrate how to augment rewards with metrics for quality and diversity, transforming a sparse signal into a dense, multi-faceted one that provides clearer guidance to the trainer. Use Case 1: Quality Reward Augmentation. To encourage the model to generate high-quality responses, we can augment the base reward with a quality score. As illustrated in Figure 11, during each RFT step, we use the data processor to evaluate the quality of each generated rollout. For our experiment, we trained a Qwen2.5-1.5B model and used a more powerful Qwen3-32B as the scorer LLM. Specifically, we invoked the llm_quality_filter from Data-Juicer, which normalized the quality scores to the range [-0.5, 0.5] and added them to the original reward. Crucially, this processing is applied to the experience buffer at each RFT step. This allows the reward signal to adapt dynamically to the policy model’s evolving capabilities, a more responsive approach than one-time static processing. With a sync_interval of 3 over 36 steps on the Math-500 validation set, the results in Figure 12 show that: (1) The model with quality reward augmentation (red line) achieves higher accuracy. (2) The introduced quality reward itself improves over time, confirming it is a learnable signal. (3) We observe a slight increase in response length, which likely reflects an inductive bias from the larger scorer model being implicitly distilled into the smaller policy model. <details> <summary>x9.png Details</summary>  ### Visual Description # Technical Document Extraction: GRPO Workflow Diagram ## 1. Overview This image is a technical flow diagram illustrating the **Group Relative Policy Optimization (GRPO)** training process for a Large Language Model (LLM). It details the pipeline from initial sampling to the multi-component reward calculation used for reinforcement learning. --- ## 2. Component Isolation & Flow Analysis The diagram flows from left to right, segmented into three primary stages: Input/Rollout, Response Generation, and Reward Scoring. ### Region A: Input and Rollout (Left) * **Input Source:** A box labeled **"GSM8K Sample"**. This refers to the Grade School Math 8K dataset, commonly used for evaluating mathematical reasoning. * **Action:** An arrow labeled **"Rollout"** points toward the generation stage. * **Actor (Model):** Beneath the rollout arrow, a purple logo and text identify the model performing the rollout: **"Qwen2.5 1.5B"**. ### Region B: Response Generation (Center) The rollout process generates a group of multiple outputs from the same prompt to facilitate relative comparison. * **Response 1**: Top box. * **Response 2**: Middle box. * **...**: Ellipsis indicating multiple intermediate responses. * **Response n**: Bottom box, representing the $n^{th}$ sample in the group. ### Region C: Reward Scoring Mechanism (Right) A dashed bounding box expands from "Response 2" (serving as a representative example for the group) to show the internal logic of the reward function. The total reward is a summation of three distinct components: 1. **Math Acc Reward**: (Top box) Likely measures the correctness of the mathematical answer. 2. **+ (Addition Operator)** 3. **Format Reward**: (Middle box) Likely measures adherence to specific output formatting (e.g., using chain-of-thought tags). 4. **+ (Addition Operator)** 5. **DJ-Quality Reward**: (Bottom box, highlighted with a **red border**). This indicates a specific or custom quality metric being emphasized in this diagram. * **Scorer Actor:** An arrow points to this specific reward component from a purple logo and text labeled **"Qwen3 32B"**. * **Role Label:** This model is explicitly labeled as the **"LLM Scorer"**. --- ## 3. Textual Transcription | Category | Transcribed Text | | :--- | :--- | | **Main Title/Process** | GRPO | | **Input Data** | GSM8K Sample | | **Process Step** | Rollout | | **Primary Model** | Qwen2.5 1.5B | | **Outputs** | Response 1, Response 2, ..., Response n | | **Reward Component 1** | Math Acc Reward | | **Reward Component 2** | Format Reward | | **Reward Component 3** | DJ-Quality Reward | | **Scoring Model** | Qwen3 32B | | **Scoring Role** | LLM Scorer | --- ## 4. Technical Summary of Logic The diagram describes a reinforcement learning setup where a smaller model (**Qwen2.5 1.5B**) generates multiple reasoning paths ("Rollouts") for a math problem. These responses are then evaluated. While standard rewards check for mathematical accuracy and formatting, a larger, more capable model (**Qwen3 32B**) acts as an "LLM Scorer" to provide a "DJ-Quality Reward," which likely evaluates the nuanced qualitative aspects of the reasoning process that cannot be captured by simple rule-based checks. This multi-faceted reward signal is then used to update the policy via the GRPO algorithm. </details> Figure 11: The enhanced math workflow with quality-reward shaping from data processor, where DJ indicates DataJuicer [2], from which more operators can be utilized to extend this worklow. <details> <summary>x10.png Details</summary>  ### Visual Description # Technical Data Extraction: Performance Metrics Dashboard This document provides a comprehensive extraction of data from three line charts representing machine learning evaluation metrics. The charts are presented in a horizontal layout, sharing a common X-axis representing training steps or epochs (ranging from approximately 5 to 35). --- ## 1. Chart: `eval/math-eval/accuracy/mean` **Description:** This chart tracks the mean accuracy of a mathematical evaluation over time. It contains two data series. ### Metadata * **Title:** `eval/math-eval/accuracy/mean` * **Y-Axis (Accuracy):** Scale from 0.25 to 0.45 with increments of 0.05. * **X-Axis (Steps):** Markers at 5, 10, 15, 20, 25, 30, 35. ### Data Series Analysis * **Red Line (Upper Series):** * **Trend:** Shows a consistent upward trajectory with minor plateaus. * **Key Points:** Starts at ~0.29 (Step 6), rises to ~0.34 (Step 12), plateaus until Step 18, then climbs to a peak of ~0.42 at Step 36. * **Blue Line (Lower Series):** * **Trend:** Shows a steady, linear upward trend, maintaining a gap below the red line. * **Key Points:** Starts at ~0.24 (Step 6) and ends at ~0.36 (Step 36). --- ## 2. Chart: `response_length/mean` **Description:** This chart tracks the average length of the model's responses. It is characterized by high volatility (noise) in both series. ### Metadata * **Title:** `response_length/mean` * **Y-Axis (Length):** Scale from 140 to 220 with increments of 20. * **X-Axis (Steps):** Markers at 5, 10, 15, 20, 25, 30, 35. ### Data Series Analysis * **Red Line (Upper Series):** * **Trend:** Highly oscillatory but maintains a mean between 170 and 190. It experiences an initial spike near Step 4 before stabilizing into a jagged horizontal pattern. * **Key Points:** Peak of ~205 (Step 4), low of ~168 (Step 16), ending at ~195. * **Blue Line (Lower Series):** * **Trend:** Highly volatile with significant downward spikes. It consistently stays below the red line after Step 5. * **Key Points:** Initial peak of ~195 (Step 4), followed by a sharp drop. Frequently fluctuates between 135 and 160. Ends with a sharp spike to ~175 (Step 33) before settling at ~160. --- ## 3. Chart: `eval/math-eval/quality/mean` **Description:** This chart tracks the mean quality score of the mathematical evaluation. It contains a single data series. ### Metadata * **Title:** `eval/math-eval/quality/mean` * **Y-Axis (Quality Score):** Scale from 0.1 to 0.25 with increments of 0.05. * **X-Axis (Steps):** Markers at 5, 10, 15, 20, 25, 30, 35. ### Data Series Analysis * **Red Line:** * **Trend:** Strong upward trend with one notable dip. * **Key Points:** * Starts at ~0.09 (Step 6). * Rises to ~0.15 (Step 12). * Dips slightly to ~0.13 (Step 18). * Sharp increase to ~0.20 (Step 24). * Reaches a plateau/peak of ~0.22 from Step 30 to Step 36. --- ## Summary Table of Final Values (Approximate) | Metric | Series Color | Final Value (Step ~36) | Overall Trend | | :--- | :--- | :--- | :--- | | **Accuracy** | Red | ~0.42 | Increasing | | **Accuracy** | Blue | ~0.36 | Increasing | | **Response Length** | Red | ~195 | Volatile / Stable | | **Response Length** | Blue | ~160 | Volatile / Stable | | **Quality** | Red | ~0.22 | Increasing | **Language Declaration:** The text in this image is entirely in **English**. No other languages were detected. </details> Figure 12: Experimental results for quality-reward shaping. Augmenting the reward with a quality score (red line) improves final accuracy and provides a learnable reward signal. Use Case 2: Diversity Reward Augmentation. A common failure mode in RFT is policy collapse, where the agent repeatedly generates similar, suboptimal responses. To counteract this, we introduce a diversity reward that encourages the explorer to explore different solution paths. As shown in Figure 13, we used the GTE-Qwen2-1.5B model to convert rollouts into semantic embeddings. The diversity reward was calculated based on the cosine similarity of a rollout’s embedding to the mean embedding of its group, with lower similarity (i.e., higher diversity) yielding a higher reward. To prevent exploration from becoming chaotic, we applied a simple decay schedule to the diversity reward’s weight, starting at 0.5 and decaying to 0.3 over the training steps. The experiment, using the same setting as before, yielded compelling results (Figure 14): (1) The diversity-augmented model (red line) shows a significant performance improvement over the baseline. (2) The response length is consistently longer, indicating the reward encourages more elaborate answers. (3) Most importantly, the actor entropy loss remains consistently higher, providing strong evidence that the model is maintaining a healthier, more diverse exploration strategy, which helps it escape local optima. <details> <summary>x11.png Details</summary>  ### Visual Description # Technical Document Extraction: GRPO Reward System Architecture ## 1. Overview This image is a technical flow diagram illustrating a Group Relative Policy Optimization (**GRPO**) reinforcement learning pipeline, specifically focusing on how a "Diversity Reward" is calculated for a mathematical reasoning task (GSM8K). The process involves generating multiple responses, embedding them, and calculating similarity scores to encourage output variety. --- ## 2. Component Isolation & Flow Analysis The diagram flows from left to right, segmented into four primary functional regions: ### Region 1: Input and Rollout (Left) * **Input Node:** A box labeled **"GSM8K Sample"**. * **Process:** An arrow labeled **"Rollout"** points to the next stage. * **Model Attribution:** Below the rollout arrow, there is a logo and blue text identifying the model: **"Qwen2.5 1.5B"**. ### Region 2: Response Generation and Embedding (Center-Left) The system generates $n$ parallel responses from the single input sample. * **Responses (Res):** Three boxes stacked vertically labeled **"Res 1"**, **"Res 2"**, and **"Res n"** (with an ellipsis "..." indicating multiple intermediate responses). * **Embeddings (Ebd):** Each response box points to a corresponding embedding box: **"Ebd 1"**, **"Ebd 2"**, and **"Ebd n"**. * **Embedding Model:** Red text below this section reads **"... Embedding ..."**. A logo and purple text at the bottom identifies the embedding model: **"GTE-Qwen2"**. ### Region 3: Similarity Calculation (Center-Right) * **Aggregation Node:** All embedding boxes converge into a central box labeled **"Ebd Avg"**. * **Metric:** Red text below this box specifies the calculation method: **"Cos Similarity"** (Cosine Similarity). * **Individual Scores:** The "Ebd Avg" node points to three individual score boxes representing the relative similarity/diversity values for each response: * **+0.5** (Red text) * **+0.1** (Green text) * **+0.3** (Yellow text) * An ellipsis "**...**" indicates scores for intermediate responses. ### Region 4: Reward Summation (Right) The final stage is enclosed in a dashed-line box, representing the total reward function. * **Component 1:** **"Format Reward"** * **Operator:** **+** * **Component 2:** **"Math Acc Reward"** (Math Accuracy Reward) * **Operator:** **+** * **Component 3:** **"Diversity Reward"** (Highlighted in a red-bordered box). The individual scores from Region 3 feed directly into this component. --- ## 3. Textual Transcription | Category | Transcribed Text | Context/Notes | | :--- | :--- | :--- | | **Main Title/Label** | `GRPO` | Located at bottom-left; refers to Group Relative Policy Optimization. | | **Input** | `GSM8K Sample` | Dataset used for mathematical word problems. | | **Action** | `Rollout` | The generation of multiple completions. | | **Models** | `Qwen2.5 1.5B`, `GTE-Qwen2` | The LLM and the Embedding model respectively. | | **Data Nodes** | `Res 1`, `Res 2`, `Res n` | Responses 1 through $n$. | | **Data Nodes** | `Ebd 1`, `Ebd 2`, `Ebd n` | Embeddings 1 through $n$. | | **Process Nodes** | `Ebd Avg` | The average embedding of the group. | | **Metrics** | `Cos Similarity` | Mathematical method for comparing embeddings. | | **Values** | `+0.5`, `+0.1`, `+0.3` | Numerical outputs of the similarity/diversity check. | | **Reward Components**| `Format Reward`, `Math Acc Reward`, `Diversity Reward` | The three pillars of the reinforcement learning reward signal. | --- ## 4. Logic and Trend Verification 1. **Group Dynamics:** The diagram confirms the "Group" aspect of GRPO by showing $n$ responses generated from a single sample. 2. **Diversity Logic:** By calculating the "Ebd Avg" and using "Cos Similarity," the system identifies how similar a specific response is to the rest of the group. The resulting values (+0.5, +0.1, +0.3) are then used to populate the **Diversity Reward**. 3. **Total Reward Construction:** The final reward is an additive composite: $$\text{Total Reward} = \text{Format Reward} + \text{Math Acc Reward} + \text{Diversity Reward}$$ 4. **Color Coding:** The use of different colors for the numerical values (+0.5 red, +0.1 green, +0.3 yellow) likely indicates the degree of diversity or the magnitude of the reward assigned to that specific response relative to the group mean. </details> Figure 13: The enhanced math workflow with diversity-reward shaping from data processor <details> <summary>x12.png Details</summary>  ### Visual Description This document provides a technical extraction of data from three performance monitoring charts, likely from a machine learning experiment tracking interface (such as Weights & Biases or TensorBoard). ### Overview The image consists of three distinct line charts arranged horizontally. Each chart tracks two data series (Red and Blue) over a shared X-axis representing training steps or epochs (ranging from approximately 0 to 35). --- ### Chart 1: eval/math-eval/accuracy/mean **Metadata:** * **Title:** `eval/math-eval/accuracy/mean` * **Y-Axis (Accuracy):** Scale from 0.25 to 0.45 (increments of 0.05). * **X-Axis (Steps):** Scale from 5 to 35 (increments of 5). **Data Series Analysis:** 1. **Red Line (Upper Series):** * **Trend:** Shows a general upward trajectory with a significant peak around step 18, followed by a slight dip and a recovery at the end. * **Key Points:** Starts at ~0.33 (step 6), peaks at ~0.40 (step 18), dips to ~0.36 (step 30), and ends at ~0.41 (step 35). 2. **Blue Line (Lower Series):** * **Trend:** Shows a steady, consistent upward slope throughout the duration. * **Key Points:** Starts at ~0.24 (step 6), reaches ~0.30 (step 20), and ends at ~0.36 (step 35). **Summary:** Both models improve in accuracy over time, but the Red series maintains a higher mean accuracy throughout the evaluation. --- ### Chart 2: response_length/mean **Metadata:** * **Title:** `response_length/mean` * **Y-Axis (Length):** Scale from 100 to 400 (increments of 100). * **X-Axis (Steps):** Scale from 0 to 35 (increments of 5). **Data Series Analysis:** 1. **Red Line (Upper Series):** * **Trend:** Initially stable with minor oscillations between 200 and 250, followed by a sharp, volatile increase in the final third of the timeline. * **Key Points:** Starts at ~180. Oscillates around 240 for most of the run. Spikes sharply after step 30, ending at ~380. 2. **Blue Line (Lower Series):** * **Trend:** After an initial drop, the line remains relatively flat with low-amplitude oscillations. * **Key Points:** Starts at ~180, drops to ~140 by step 5, and remains between 130 and 160 for the remainder of the run. **Summary:** The Red series shows a significant increase in response length (potential "verbosity" or "reasoning" expansion) toward the end of training, while the Blue series remains concise. --- ### Chart 3: actor/entropy_loss **Metadata:** * **Title:** `actor/entropy_loss` * **Y-Axis (Loss):** Scale from 0.5 to 1.5 (increments of 0.5). * **X-Axis (Steps):** Scale from 0 to 35 (increments of 5). **Data Series Analysis:** 1. **Red Line (Increasing Series):** * **Trend:** Initially stable/slightly declining, followed by a dramatic and volatile upward trend starting around step 20. * **Key Points:** Starts at ~0.6. Remains below 0.7 until step 20. Spikes aggressively with high variance, peaking near 1.7 and ending at ~1.5. 2. **Blue Line (Decreasing Series):** * **Trend:** Shows an initial small spike, followed by a gradual, steady decline (convergence). * **Key Points:** Starts at ~0.6, peaks briefly at ~0.8 (step 6), then trends downward to end at ~0.25. **Summary:** The Red series experiences a "divergence" or significant increase in entropy loss in the later stages, correlating with the increased response length in Chart 2. The Blue series shows standard loss convergence. --- ### Technical Observations & Correlations * **Correlation:** There is a strong positive correlation between the **Red Line's** increase in `response_length` (Chart 2) and its increase in `entropy_loss` (Chart 3) after step 30. * **Performance:** While the Red series achieves higher accuracy (Chart 1), it does so at the cost of significantly higher entropy and longer response lengths compared to the Blue series. * **Language:** All text in the interface is in **English**. </details> Figure 14: Experimental results for diversity-reward shaping. Rewarding diverse responses (red line) significantly improves task accuracy and maintains higher entropy. 3.5 RFT with Human in the Loop This example demonstrates the human-in-the-loop capability in Trinity-RFT for preference modeling. As illustrated in Listing 6 and Figure 15, our framework integrates Label Studio’s annotation interface with asynchronous data pipelines through four coordinated stages: (1) task generation: auto-creating annotation batches from model rollouts; (2) interactive labeling: providing UI for side-by-side response comparison; (3) quality control: enforcing inter-annotator agreement thresholds; and (4) versioned storage: tracking preference lineage in pre-defined fields like those in DPODataModel. This pipeline reflects Trinity-RFT ’s bi-directional collaboration feature (Section 2.3.4), backed by timeout-aware task polling and support of atomic batch commit. It enables hybrid procedures where initial AI pre-screening can reduce human workload in production deployments. Annotation activities can scale across distributed teams through event-driven task routing. The system’s flexibility benefits rapid adaptation to diverse annotation protocols, allowing developers to implement custom labeling interfaces through XML-based templates or integrate third-party annotation services via unified SDK endpoints. This capability underpins advanced use cases such as safety red-teaming datasets and online instruction tuning scenarios where human judgment remains irreplaceable for quality-critical decisions, particularly in human-centric sociocultural contexts where data quality, difficulty, and reward signals are difficult to verify logically. ⬇ 1 # Human annotation configuration 2 class HumanAnnotationConfig: 3 "" "Preference annotation pipeline configuration" "" 4 5 def __init__ (self): 6 self. process = [ 7 { 8 "human_preference_annotation_mapper": { 9 "wait_for_annotations": True, # Block until annotations complete 10 "timeout": 3600, # Maximum wait time in seconds 11 "prompt_key": "prompt", # Source field for prompts 12 "answer1_key": "answer1", # First candidate response 13 "chosen_key": "chosen" # Selected response key 14 } 15 } 16 ] 17 18 def get_pipeline (self) -> List [Dict]: 19 "" "Get annotation processing pipeline" "" 20 return self. process Listing 6: Configuration for human preference annotation. <details> <summary>x13.png Details</summary>  ### Visual Description # Technical Document Extraction: Label Studio Interface ## 1. Document Overview This image is a high-fidelity screenshot of the **Label Studio** web application interface. It depicts a "Human Preference Annotation" task where a user is asked to evaluate or select between two options based on a provided prompt. --- ## 2. Component Isolation ### Region A: Global Header (Top) * **Logo:** Label Studio (top left). * **Navigation Path:** `Projects / Human_Preference_Annotation_Demo_8a87e7 / Labeling` * **Task Metadata:** * Task ID: `#1` * User: `admin@example.com` * Timestamp: `seconds ago` * Unique Identifier: `#Xiqn9` ### Region B: Sidebar - Task List (Left) This region contains a vertical list of data rows. Each row includes a checkbox, a numerical status (all showing `0`), and a text prompt. | Status | Prompt | | :--- | :--- | | 0 | What is the capital of France? (Currently selected/active) | | 0 | Which planet is known as the Red Planet? | | 0 | What is the chemical symbol for gold? | | 0 | Who wrote 'Romeo and Juliet'? | | 0 | What is the largest ocean on Earth? | | 0 | In which year did World War II end? | | 0 | What is the square root of 64? | | 0 | Who painted the Mona Lisa? | | 0 | What is the main component of the sun? | | 0 | Which programming language was created... (truncated) | ### Region C: Main Annotation Workspace (Center) This is the primary interactive area for the labeling task. It consists of a prompt box and two response boxes. * **Prompt Box (Blue Background):** * Text: "What is the capital of France?" * **Response Options (Dark Grey Backgrounds):** * **Option 1 (Left):** "Paris" * **Option 2 (Right):** "Lyon" ### Region D: Information & History Panel (Right Top) * **Tabs:** `Info`, `History` * **Content:** "Selection Details" (Currently empty/no specific details shown). ### Region E: Regions & Relations Panel (Right Bottom) * **Tabs:** `Regions`, `Relations` * **Controls:** `Manual` (dropdown), `By Time` (sort icon). * **Status Text:** "Regions not added" --- ## 3. Data Structure & Flow The interface represents a **Comparative Choice** or **A/B Testing** data labeling workflow. 1. **Input:** A natural language question (Prompt). 2. **Candidates:** Two potential answers (Paris vs. Lyon). 3. **Action:** The annotator is expected to select the correct or preferred response from the two dark grey blocks. 4. **Context:** The left sidebar indicates this is the first task in a series of general knowledge questions. --- ## 4. Technical Specifications * **Primary Language:** English (No other languages detected). * **UI Elements:** Standard web-based dashboard with a three-pane layout (Navigation/List, Workspace, Metadata). * **Visual Encoding:** * **Blue:** Used for the primary prompt/instruction. * **Dark Grey:** Used for interactive response buttons. * **Light Grey/White:** Used for background and secondary UI elements. </details> Figure 15: An interactive interface for human preference annotation. 3.6 Low-Code Usage and Development with Trinity-Studio <details> <summary>figs/studio-showcase/dashboard.png Details</summary>  ### Visual Description # Technical Document Extraction: RFT Portal Dashboard This document provides a comprehensive extraction of the textual and structural information contained within the provided image of a software interface. ## 1. Component Isolation The interface is organized into three distinct vertical regions: * **Header (Navigation Bar):** A dark blue horizontal bar containing global navigation links. * **Main Title:** A central heading identifying the page. * **Main Content (Service Cards):** A horizontal layout of three interactive cards, each representing a specific tool or service. --- ## 2. Header Information (Navigation Bar) The navigation bar is located at the top of the screen. The active page is highlighted with a subtle background box. | Navigation Item | Status | | :--- | :--- | | **Dashboard** | Active (Selected) | | **pgAdmin** | Inactive | | **Label Studio** | Inactive | | **Training Portal** | Inactive | | **Settings** | Inactive | --- ## 3. Main Title * **Text:** `RFT Portal Dashboard` * **Placement:** Centered below the navigation bar. --- ## 4. Service Cards (Main Content) The dashboard features three primary service cards. Each card follows a consistent structure: an icon with initials, a service title, a brief description, and a call-to-action (CTA) button. ### Card 1: Training Portal * **Icon:** Blue circle with white text **"TP"** * **Title:** Training Portal (Blue text) * **Description:** "Access the training portal to manage your training data and models." * **CTA Button:** Blue button with white text **"Open Training Portal"** ### Card 2: pgAdmin * **Icon:** Green circle with white text **"DB"** * **Title:** pgAdmin (Green text) * **Description:** "Manage your PostgreSQL databases with pgAdmin." * **CTA Button:** Green button with white text **"Open pgAdmin"** ### Card 3: Label Studio * **Icon:** Red/Orange circle with white text **"LS"** * **Title:** Label Studio (Red/Orange text) * **Description:** "Label and annotate your data with Label Studio." * **CTA Button:** Red/Orange button with white text **"Open Label Studio"** --- ## 5. Technical Summary * **Language:** English (100%). No other languages detected. * **Interface Type:** Web-based Management Dashboard. * **Functionality:** The dashboard acts as a centralized hub (portal) for three distinct technical workflows: Machine Learning training (Training Portal), Database Management (pgAdmin), and Data Annotation (Label Studio). * **Visual Logic:** The interface uses color-coding (Blue, Green, Red) to differentiate between the three functional domains, maintaining this color consistency across the icon, title, and button of each respective card. </details> (a) Trinity-Studio dashboard. <details> <summary>figs/studio-showcase/training-portal-click-run.jpg Details</summary>  ### Visual Description # Technical Document Extraction: Training Portal Interface ## 1. Component Isolation The image is a screenshot of a web-based machine learning "Training Portal" interface. It is segmented into the following regions: * **Global Navigation Header:** Dark grey bar at the very top. * **Application Header:** Blue bar with the application title and tools. * **Configuration Input Section:** Area for setting hyperparameters. * **Action & Status Section:** Buttons for file generation and job status notifications. * **Configuration Preview (Partial):** A code block showing the generated YAML/JSON configuration. --- ## 2. Detailed Textual Extraction ### Region: Global Navigation Header * **Dashboard** * **pgAdmin** * **Label Studio** * **Training Portal** (Highlighted/Active state with a grey background box) * **Settings** ### Region: Application Header * **Menu Icon** (Three horizontal lines) * **Training Portal** (Title) * **TOOLS** (Accompanied by a gear icon) ### Region: Configuration Input Section This section contains two numerical input fields with increment/decrement controls. | Label | Current Value | Controls | | :--- | :--- | :--- | | Micro Batch Size Per GPU | 8 | `-` and `+` buttons | | Learning Rate | 1.0e-6 | `-` and `+` buttons (greyed out/disabled) | * **Button:** `[icon] Generate Config` (Centered below the inputs) ### Region: Action & Status Section * **Sub-header:** Generated Config File * **Secondary Navigation:** `Deploy` (Top right of this section), `[vertical ellipsis icon]` * **Action Buttons:** * `[download icon] Save` (Active) * `[terminal icon] Run` (Inactive/Greyed out) * **Status Notification (Green Box):** * `[check mark icon] Job submitted successfully!` * `View progress in the Ray Dashboard: http://127.0.0.1:8265` ### Region: Configuration Preview (Code Block) The text is formatted as a structured configuration file (YAML style). ```yaml mode: both data: total_epochs: 20 batch_size: 96 ``` *(Note: The image cuts off after these lines.)* --- ## 3. Component Analysis and Flow ### System Workflow 1. **Input:** The user defines hyperparameters (Micro Batch Size and Learning Rate). 2. **Generation:** The user clicks "Generate Config" to create the configuration file displayed below. 3. **Execution:** Based on the green status box, the "Run" command (or an automated trigger) has been executed. 4. **Monitoring:** The system provides a local loopback URL (`http://127.0.0.1:8265`) to monitor the training progress via the Ray Dashboard. ### Visual Indicators * **Active State:** The "Training Portal" tab in the top nav is highlighted. * **Success State:** The green background and checkmark icon indicate a successful backend operation (job submission). * **Interactive Elements:** The "Save" button is outlined and appears clickable, while the "Run" button is faded, suggesting the process is already running or the action is currently unavailable. --- ## 4. Language Declaration The entire interface is in **English**. No other languages are present. </details> (b) Start training on the “Training Portal” page. <details> <summary>figs/studio-showcase/pgadmin-select-table.jpg Details</summary>  ### Visual Description # Technical Document Extraction: pgAdmin Database Interface ## 1. Component Isolation The image is a screenshot of a web-based database management interface, specifically a custom implementation of a **pgAdmin** dashboard. The interface is divided into four primary functional regions: * **Global Navigation Header:** Top dark bar for application-level switching. * **Context Header:** Blue bar indicating the current tool and environment selection. * **Sidebar (Navigation Tree):** Left-hand panel for database and table selection. * **Main Content Area:** Central workspace containing the table schema and a query editor. --- ## 2. Global Navigation Header Located at the top of the screen, this bar contains the following navigation links: * Dashboard * **pgAdmin** (Highlighted/Active state with a grey background box) * Label Studio * Training Portal * Settings --- ## 3. Context Header & Sidebar ### Context Header (Blue Bar) * **Title:** pgAdmin (Left-aligned) * **Environment Selector:** A dropdown menu (Right-aligned) currently set to **"Local Development"**. * **Refresh Icon:** A circular arrow icon located to the far right. ### Sidebar (Navigation Tree) * **Database Selector:** A dropdown labeled "Database" with the value **"testdb"** selected. * **Tables List:** A vertical list of tables within the selected database. The table **"experience_buffer"** is currently selected (indicated by a light blue highlight). * xxx * sft_data_buffer * rft_dataset * task_buffer * **experience_buffer** (Selected) * dpo_data_buffer --- ## 4. Main Content Area: Table Structure The primary data display shows the schema for the selected table. ### Header: Table Structure: experience_buffer | Column | Type | Nullable | | :--- | :--- | :--- | | consumed | integer | YES | | priority | double precision | YES | | serialized_exp | bytea | YES | | id | integer | NO | | reward | double precision | YES | | response | character varying | YES | | prompt | character varying | YES | **Data Observations:** * The table contains 7 columns. * The `id` column is the only field marked as **NOT NULL** (Nullable: NO), suggesting it is likely the Primary Key. * The table stores a mix of numeric data (`integer`, `double precision`), binary data (`bytea`), and string data (`character varying`). --- ## 5. Main Content Area: SQL Query Editor Located below the table structure section. * **Section Title:** SQL Query * **Input Field:** A text area with placeholder text: *"Enter SQL query..."* * **Action Button:** A blue button at the bottom left of the section labeled **"▶ EXECUTE QUERY"**. --- ## 6. Technical Summary This interface represents a PostgreSQL management tool focused on a Reinforcement Learning or LLM training pipeline, evidenced by table names like `experience_buffer`, `sft_data_buffer` (Supervised Fine-Tuning), and `dpo_data_buffer` (Direct Preference Optimization). The `experience_buffer` table specifically tracks training data components including rewards, prompts, responses, and serialized experience data. </details> (c) Manage data on the “pgAdmin” page. <details> <summary>figs/studio-showcase/label-studio-enter.jpg Details</summary>  ### Visual Description # Technical Document Extraction: Label Studio Interface This document provides a comprehensive extraction of the textual and structural data from the provided image of the Label Studio web interface. ## 1. Global Navigation Header The top-most section of the interface contains a dark-themed navigation bar with the following links: * **Dashboard** * **pgAdmin** * **Label Studio** (Currently active/selected tab, indicated by a lighter background highlight) * **Training Portal** * **Settings** ## 2. Sub-Header / Brand Bar Located immediately below the global navigation: * **Logo:** Label Studio (accompanied by a square orange icon with white internal segments). * **Menu Icon:** A three-line "hamburger" menu icon. * **Breadcrumb/Location:** "Home" * **User Profile:** A circular avatar on the far right with the initials "AD". ## 3. Main Content Area (Welcome Section) This section serves as the primary landing area for the user. ### Header Text * **Title:** Welcome 👋 * **Subtitle:** Let's get you started. ### Action Buttons Two primary action buttons are positioned horizontally: 1. **Create Project:** Accompanied by a folder icon with a plus sign. 2. **Invite People:** Accompanied by a person icon with a plus sign. ## 4. Recent Projects Section A large container displaying the user's most recent activity. * **Section Title:** Recent Projects * **Action Link:** View All (located at the top right of the container). ### Project List Data The section contains two project entries, both showing 100% completion. | Project Name | Task Progress | Completion Percentage | Visual Indicator | | :--- | :--- | :--- | :--- | | `Human_Preference_Annotation_Demo_acc038` | 10 of 10 Tasks | 100% | Solid teal progress bar (full) | | `Human_Preference_Annotation_Demo_8a87e7` | 10 of 10 Tasks | 100% | Solid teal progress bar (full) | **Trend Verification:** Both progress bars are visually identical, extending to the full width of their designated track, confirming the "100%" textual data. ## 5. Sidebar: Resources Located on the right side of the screen. * **Title:** Resources * **Subtitle:** Learn, explore and get help * **Links:** (Each link is accompanied by an "external link" icon) * Documentation * API Documentation * Release Notes * LabelStud.io Blog * Slack Community ## 6. Footer Information Located at the bottom right of the interface: * **Label Studio Version:** Community (accompanied by the Label Studio logo icon). </details> (d) Process data on the “Label Studio” page. Figure 16: Snapshots of Trinity-Studio. Trinity-Studio provides visual interaction for the core capabilities of Trinity-RFT, designed to bridge the gap between system complexity and user accessibility. As shown in Figure 16(a), its three integrated modules — “Training Portal”, “pgAdmin”, and “Label Studio” — form a cohesive interface that supports low-code usage and development with Trinity-RFT, and make it easy to monitor and track the full RFT pipeline with transparency. - “Training Portal” (Figure 16(b)) implements configuration-to-execution procedures through declarative YAML editing with auto completion and live validation that prevents misconfigurations. Furthermore, the integration of runtime metrics with tools like Wandb/TensorBoard directly helps the active data optimization feature by surfacing signals such as difficulty distribution drifts and diversity metrics mentioned in Section 3.4. This transparency ensures that users can monitor how data curation strategies impact RFT performance in real time. - “pgAdmin” (Figure 16(c)) reflects Trinity-RFT ’s end-to-end data transformation capabilities by providing a visual panel for PostgreSQL-based storage. This design benefits the versioned data lineage requirements of RFT, particularly for scenarios involving asynchronous training (Section 2.3.3). With intuitive SQL query builders, users can easily adjust schema, audit training experiences and human annotation batches with fine-grained precision. This capability is valuable for rapid validation of active learning policies by cross-referencing training outcomes with metadata (e.g., difficulty scores and staleness in asynchronous mode). - “Label Studio” page (Figure 16(d)) operationalizes Trinity-RFT ’s bi-directional human-AI collaboration capability (Section 2.3.4). Utilizing the provided task polling and atomic batch commit mechanisms, users can annotate the data or experiences directly, allowing an asynchronous way to involve human feedback and to dynamically influence data curation. By unifying these capabilities in a single UI, Trinity-Studio reduces the cognitive load of managing complex RFT procedures. For example, a researcher tuning a math reasoning task could use the Training Portal to adjust difficulty scoring parameters, view the resulting distribution shifts in the pgAdmin module, and then validate human annotators’ preferences in the Label Studio page. This end-to-end visibility can be useful for debugging and iterating RFT strategies, and complements the programmatic APIs of Trinity-RFT while maintaining full compatibility with CLI procedures. We implement Trinity-Studio with the Singe-Spa framework [33]. The modular architecture enables custom view development through JavaScript plugins and flexible extensions for general-purpose usage. 4 Conclusion and Next Steps We have presented Trinity-RFT, a general-purpose, flexible, scalable and user-friendly framework for reinforcement fine-tuning of large language models. Trinity-RFT offers a path into “the era of experience” [32], by supporting applications in diverse scenarios with complex agent-environment interaction, and serving as a unified platform for exploring advanced methodologies in each stage of the complete RFT pipeline, at both macroscopic and microscopic levels. Further development of Trinity-RFT includes incorporating more advanced RL algorithms (especially off-policy or asynchronous ones), making the choices of hyperparameters more adaptive and less reliant on manual tuning, augmenting data pipelines with smarter sampling strategies and data processing operations, and conducting more thorough experiments and evaluations with Trinity-RFT. Acknowledgements Trinity-RFT is built upon many excellent open-source projects, including but not limited to: verl [30] and PyTorch’s FSDP [8] for LLM training; vLLM [15] for LLM inference; Data-Juicer [2] for data-related functionalities; AgentScope [11] for agentic workflow; and Ray [20] for distributed runtime. References - [1] ChatLearn. https://github.com/alibaba/ChatLearn. - [2] Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. Data-juicer: A one-stop data processing system for large language models. In International Conference on Management of Data, 2024. - [3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. - [4] Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-rlvr: Training software engineering agents via guidance and environment rewards. arXiv, 2025. - [5] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv, 2025. - [6] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. - [7] Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist. Contrastive policy gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion. In EMNLP, 2024. - [8] Pytorch FSDP. https://pytorch.org/docs/stable/fsdp.html. - [9] Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv, 2025. - [10] Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv, 2025. - [11] Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, Liuyi Yao, Hongyi Peng, Ze Yu Zhang, Lin Zhu, Chen Cheng, Hongzhu Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. Agentscope: A flexible yet robust multi-agent platform. arXiv, 2024. - [12] Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, and Jianping Wu. Asyncflow: An asynchronous streaming rl framework for efficient llm post-training. arXiv, 2025. - [13] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv, 2024. - [14] Huggingface. https://huggingface.co/. - [15] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. - [16] Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, and Wentao Zhang. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. arXiv, 2025. - [17] Model context protocol servers. https://github.com/modelcontextprotocol/servers. - [18] Mistral-AI. Magistral. arXiv, 2025. - [19] Modelscope. https://www.modelscope.cn/home. - [20] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging ai applications. arXiv, 2018. - [21] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In NIPS, 2017. - [22] Nccl. https://github.com/NVIDIA/nccl. - [23] Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. In The Thirteenth International Conference on Learning Representations, 2025. - [24] Long Ouyang, Pamela Mishkin, Jeff Wu, C L Mar, Jacob Hilton, Amanda Askell, and Paul Christiano. Training language models to follow instructions with human feedback. arXiv, 2022. - [25] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. - [26] Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment. arXiv, 2024. - [27] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019. - [28] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017. - [29] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024. - [30] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient rlhf framework. arXiv, 2024. - [31] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. - [32] David Silver and Richard S. Sutton. Welcome to the era of experience. https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf, 2025. - [33] A javascript framework for front-end microservices, 2025. - [34] Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. In The Eleventh International Conference on Learning Representations, 2023. - [35] Kimi Team. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv, 2025. - [36] Qwen Team. Qwen2.5 technical report, 2025. - [37] ROLL Team and Other ROLL Contributors. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library. arXiv, 2025. - [38] TensorBoard. https://www.tensorflow.org/tensorboard. - [39] Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2025. Open source software available from https://github.com/HumanSignal/label-studio. - [40] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020. - [41] Weights & Biases. https://wandb.ai/home. - [42] Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye HAO, Jun Wang, and Kun Shao. DistRL: An asynchronous distributed reinforcement learning framework for on-device control agent. In The Thirteenth International Conference on Learning Representations, 2025. - [43] Bo Wu, Sid Wang, Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, Xiaocheng Tang, Yundi Qian, Beibei Zhu, and Rui Hou. Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale llm training. arXiv, 2025. - [44] LLM-Core-Team Xiaomi. Mimo: Unlocking the reasoning potential of language model – from pretraining to posttraining. arXiv, 2025. - [45] Tianbing Xu. Training large language models to reason via EM policy gradient. arXiv, 2025. - [46] Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv, 2025. - [47] Chaorui Yao, Yanxi Chen, Yuchang Sun, Yushuo Chen, Wenhao Zhang, Xuchen Pan, Yaliang Li, and Bolin Ding. Group-relative reinforce is secretly an off-policy algorithm: Demystifying some myths about grpo and its friends. arXiv, 2025. - [48] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. - [49] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022. - [50] Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, and Daxin Jiang. Streamrl: Scalable, heterogeneous, and elastic rl for llms with disaggregated stream generation. arXiv, 2025.