2509.24203v1

# Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends ## Abstract Off-policy reinforcement learning (RL) for large language models (LLMs) is attracting growing interest, driven by practical constraints in real-world applications, the complexity of LLM-RL infrastructure, and the need for further innovations of RL methodologies. While classic REINFORCE and its modern variants like Group Relative Policy Optimization (GRPO) are typically regarded as on-policy algorithms with limited tolerance of off-policyness, we present in this work a first-principles derivation for group-relative REINFORCE without assuming a specific training data distribution, showing that it admits a native off-policy interpretation. This perspective yields two general principles for adapting REINFORCE to off-policy settings: regularizing policy updates, and actively shaping the data distribution. Our analysis demystifies some myths about the roles of importance sampling and clipping in GRPO, unifies and reinterprets two recent algorithms — Online Policy Mirror Descent (OPMD) and Asymmetric REINFORCE (AsymRE) — as regularized forms of the REINFORCE loss, and offers theoretical justification for seemingly heuristic data-weighting strategies. Our findings lead to actionable insights that are validated with extensive empirical studies, and open up new opportunities for principled algorithm design in off-policy RL for LLMs. Source code for this work is available at https://github.com/modelscope/Trinity-RFT/tree/main/examples/rec_gsm8k. footnotetext: Equal contribution. Contact: chaorui@ucla.edu, chenyanxi.cyx@alibaba-inc.com footnotetext: UCLA. Work done during an internship at Alibaba Group. footnotetext: Alibaba Group. ## 1 Introduction The past few years have witnessed rapid progress in reinforcement learning (RL) for large language models (LLMs). This began with reinforcement learning from human feedback (RLHF) (Bai et al., 2022; Ouyang et al., 2022) that aligns pre-trained LLMs with human preferences, followed by reasoning-oriented RL that enables LLMs to produce long chains of thought (OpenAI, 2024; DeepSeek-AI, 2025; Kimi-Team, 2025b; Zhang et al., 2025b). More recently, agentic RL (Kimi-Team, 2025a; Gao et al., 2025; Zhang et al., 2025a) aims to train LLMs for agentic capabilities such as tool use, long-horizon planning, and multi-step task execution in dynamic environments. Alongside these developments, off-policy RL has been attracting growing interest. In the “era of experience” (Silver and Sutton, 2025), LLM-powered agents need to be continually updated through interaction with the environment. Practical constraints in real-world deployment and the complexity of LLM-RL infrastructure often render on-policy training impractical (Noukhovitch et al., 2025): rollout generation and model training can proceed at mismatched speeds, data might be collected from different policies, reward feedback might be irregular or delayed, and the environment may be too costly or unstable to query for fresh trajectories. Moreover, in pursuit of higher sample efficiency and model performance, it is desirable to go beyond the standard paradigm of independent rollout sampling, e.g., via replaying past experiences (Schaul et al., 2016; Rolnick et al., 2019; An et al., 2025), synthesizing higher-quality experiences based on auxiliary information (Da et al., 2025; Liang et al., 2025; Guo et al., 2025), or incorporating expert demonstrations into online RL (Yan et al., 2025; Zhang et al., 2025c) — all of which incur off-policyness. However, the prominent algorithms in LLM-RL — Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024) — are essentially on-policy methods: as modern variants of REINFORCE (Williams, 1992), their fundamental rationale is to produce unbiased estimates of the policy gradient, which requires fresh data sampled from the current policy. PPO and GRPO can handle a limited degree of off-policyness via importance sampling, but require that the current policy remains sufficiently close to the behavior policy. Truly off-policy LLM-RL often demands ad-hoc analysis and algorithm design; worse still, as existing RL infrastructure (Sheng et al., 2024; Hu et al., 2024; von Werra et al., 2020; Wang et al., 2025; Pan et al., 2025; Fu et al., 2025a) is typically optimized for REINFORCE-style algorithms, their support for specialized off-policy RL algorithms could be limited. All these have motivated our investigation into principled and infrastructure-friendly algorithm design for off-policy RL. #### Core finding: a native off-policy interpretation for group-relative REINFORCE. Consider a one-step RL setting and a group-relative variant of REINFORCE that, like in GRPO, assumes access to multiple responses $\{y_{1},\dots,y_{K}\}$ for the same prompt $x$ and use the group mean reward $\overline{r}$ as the baseline in advantage calculation. Each response is a sequence of tokens $y_{i}=(y^{1}_{i},y^{2}_{i},\dots)$ , and receives a response-level reward $r_{i}=r(x,y_{i})$ . Let $\pi_{\bm{\theta}}(\cdot|x)$ denote an autoregressive policy parameterized by $\bm{\theta}$ . The update rule for each iteration of group-relative REINFORCE is $\bm{\theta}^{\prime}=\bm{\theta}+\eta\bm{g}$ , where $\eta$ is the learning rate, and $\bm{g}$ is the sum of updates from multiple prompts and their corresponding responses. For a specific prompt $x$ , the update would be For notational simplicity and consistency, we use the same normalization factor $1/K$ for both response-wise and token-wise formulas in Eq. (1a) and (1b). For practical implementation, the gradient is calculated with samples from a mini-batch, and typically normalized by the total number of response tokens. This mismatch does not affect our theoretical studies in this work. Interestingly, our analysis of REINFORCE in this work provides certain justifications for calculating the token-mean loss within a mini-batch, instead of first taking the token-mean loss within each sequence and then taking the average across sequences (Shao et al., 2024); our perspective is complementary to the rationales explained in prior works like DAPO (Yu et al., 2025), although a deeper understanding of this aspect is beyond our current focus. $$ \displaystyle\bm{g}\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big) \displaystyle=\frac{1}{K}\sum_{1\leq i\leq K}(r_{i}-\overline{r})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}\,|\,x)\quad\qquad\qquad\text{(response-wise)} \displaystyle=\frac{1}{K}\sum_{1\leq i\leq K}\sum_{1\leq t\leq|y_{i}|}(r_{i}-\overline{r})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})\quad\text{(token-wise)}. $$ Here, the response-wise and token-wise formulas are linked by the elementary decomposition $\log\pi_{\bm{\theta}}(y_{i}\,|\,x)=\sum_{t}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})$ , where $y^{<t}_{i}$ denotes the first $t-1$ tokens of $y_{i}$ . A major finding of this work is that group-relative REINFORCE admits a native off-policy interpretation. We establish this in Section 2 via a novel, first-principles derivation that makes no explicit assumption about the sampling distribution of the responses $\{y_{i}\}$ , in contrast to the standard policy gradient theory. Our derivation provides a new perspective for understanding how REINFORCE makes its way towards the optimal policy by constructing a series of surrogate objectives and taking gradient steps for the corresponding surrogate losses. Such analysis can be extended to multi-step RL settings as well, with details deferred to Appendix A. #### Implications: principles and concrete methods for augmenting REINFORCE. While the proposed off-policy interpretation does not imply that vanilla REINFORCE should converge to the optimal policy when given arbitrary training data (which is too good to be true), our analysis in Section 3 identifies two general principles for augmenting REINFORCE in off-policy settings: (1) regularize the policy update step to stabilize learning, and (2) actively shape the training data distribution to steer the policy update direction. As we will see in Section 4, this unified framework demystifies common myths about the rationales behind many recent RL algorithms: (1) It reveals that in GRPO, clipping (as a form of regularization) plays a much more essential role than importance sampling, and it is often viable to enlarge the clipping range far beyond conventional choices for accelerated convergence without sacrificing stability. (2) Two recent algorithms — Kimi’s Online Policy Mirror Descent (OPMD) (Kimi-Team, 2025b) and Meta’s Asymmetric REINFORCE (AsymRE) (Arnal et al., 2025) — can be reinterpreted as adding a regularization loss to the standard REINFORCE loss, which differs substantially from the rationales explained in their original papers. (3) Our framework justifies heuristic data-weighting strategies like discarding certain low-reward samples or up-weighting high-reward ones, even though they violate assumptions in policy gradient theory and often require ad-hoc analysis in prior works. Extensive empirical studies in Section 4 and Appendix B validate these insights and demonstrate the efficacy and/or limitations of various algorithms under investigation. By revealing the off-policy nature of group-relative REINFORCE, our work opens up new opportunities for principled, infrastructure-friendly algorithm design in off-policy LLM-RL with solid theoretical foundation. ## 2 Two interpretations for REINFORCE Consider the standard reward-maximization objective in reinforcement learning: $$ \max_{\bm{\theta}}\quad J(\bm{\theta})\coloneqq{\mathbb{E}}_{x\sim D}\;J(\bm{\theta};x),\quad\text{where}\quad J(\bm{\theta};x)\coloneqq{\mathbb{E}}_{y\sim\pi_{\bm{\theta}}(\cdot|x)}\;r(x,y), \tag{2} $$ where $D$ is a distribution over the prompts $x$ . We first recall the standard on-policy interpretation of REINFORCE in Section 2.1, and then present our proposed off-policy interpretation in Section 2.2. ### 2.1 Recap: on-policy interpretation via policy gradient theory In the classical on-policy view, REINFORCE updates policy parameters $\bm{\theta}$ using samples that are drawn directly from $\pi_{\bm{\theta}}$ . The policy gradient theorem (Sutton et al., 1998) tells us that $$ \nabla_{\bm{\theta}}J(\bm{\theta};x)=\nabla_{\bm{\theta}}\;{\mathbb{E}}_{y\sim\pi_{\bm{\theta}}(\cdot|x)}\;r(x,y)={\mathbb{E}}_{y\sim\pi_{\bm{\theta}}(\cdot|x)}\!\Big[\big(r(x,y)-b(x)\big)\,\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y|x)\Big], $$ where $b(x)$ is a baseline for reducing variance when $\nabla_{\bm{\theta}}J(\bm{\theta};x)$ is estimated with finite samples. If samples are drawn from a different behavior policy $\pi_{\textsf{b}}$ instead, the gradient can be rewritten as | | $\displaystyle\nabla_{\bm{\theta}}J(\bm{\theta};x)$ | $\displaystyle={\mathbb{E}}_{y\sim\pi_{\textsf{b}}(\cdot|x)}\!\bigg[\big(r(x,y)-b(x)\big)\,\frac{\pi_{\bm{\theta}}(y\mid x)}{\pi_{\textsf{b}}(y\mid x)}\,\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y\mid x)\bigg].$ | | | --- | --- | --- | --- | While the raw importance-sampling weight ${\pi_{\bm{\theta}}(y|x)}/{\pi_{b}(y|x)}$ facilitates unbiased policy gradient estimate, it may be unstable when $\pi_{\bm{\theta}}$ and $\pi_{\textsf{b}}$ diverge. Modern variants of REINFORCE address this by modifying the probability ratios (e.g., via clipping or normalization), which achieves better bias-variance trade-off in the policy gradient estimate and leads to a stable learning process. In the LLM context, we have $\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y\,|\,x)=\sum_{t}\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y^{t}\,|\,x,y^{<t})$ , but the response-wise probability ratio $\pi_{\bm{\theta}}(y|x)/\pi_{\textsf{b}}(y|x)$ can blow up or shrink exponentially with the sequence length. Practical implementations typically adopt token-wise probability ratio instead: | | $\displaystyle\widetilde{g}(\bm{\theta};x)$ | $\displaystyle={\mathbb{E}}_{y\sim\pi_{\textsf{b}}(\cdot|x)}\!\bigg[\big(r(x,y)-b(x)\big)\,\sum_{1\leq t\leq|y|}\frac{\pi_{\bm{\theta}}(y^{t}\,|\,x,y^{<t})}{\pi_{\textsf{b}}(y^{t}\,|\,x,y^{<t})}\,\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y^{t}\,|\,x,y^{<t})\bigg].$ | | | --- | --- | --- | --- | Although this becomes a biased approximation of $\nabla_{\bm{\theta}}J(\bm{\theta};x)$ , classical RL theory still offers policy improvement guarantees if $\pi_{\bm{\theta}}$ is sufficiently close to $\pi_{\textsf{b}}$ (Kakade and Langford, 2002; Fragkiadaki, 2018; Schulman et al., 2015, 2017; Achiam et al., 2017). ### 2.2 A new interpretation: REINFORCE is inherently off-policy We now provide an alternative off-policy interpretation for group-relative REINFORCE. Let us think of policy optimization as an iterative process $\bm{\theta}_{1},\bm{\theta}_{2},\dots$ , and focus on the $t$ -th iteration that updates the policy model parameters from $\bm{\theta}_{t}$ to $\bm{\theta}_{t+1}$ . Our derivation consists of three steps: (1) define a KL-regularized surrogate objective, and show that its optimal solution must satisfy certain consistency conditions; (2) define a surrogate loss (with finite samples) that enforces such consistency conditions; and (3) take one gradient step of the surrogate loss, which turns out to be equivalently the group-relative REINFORCE method. #### Step 1: surrogate objective and consistency condition. Consider the following KL-regularized surrogate objective that incentivizes the policy to make a stable improvement over $\pi_{\bm{\theta}_{t}}$ : $$ \max_{\bm{\theta}}\quad J(\bm{\theta};\pi_{\bm{\theta}_{t}})\coloneqq{\mathbb{E}}_{x\sim D}\Big[{\mathbb{E}}_{y\sim\pi_{\bm{\theta}}(\cdot|x)}[r(x,y)]-\tau\cdot D_{\textsf{KL}}\big(\pi_{\bm{\theta}}(\cdot|x)\,\|\,\pi_{\bm{\theta}_{t}}(\cdot|x)\big)\Big], \tag{3} $$ where $\tau$ is a regularization coefficient. It is a well-known fact that the optimal policy $\pi$ for this surrogate objective satisfies the following (Nachum et al., 2017; Korbak et al., 2022; Rafailov et al., 2023; Richemond et al., 2024; Kimi-Team, 2025b): for any prompt $x$ and response $y$ , $$ \displaystyle\pi(y|x)=\frac{\pi_{\bm{\theta}_{t}}(y|x)e^{r(x,y)/\tau}}{Z(x,\pi_{\bm{\theta}_{t}})},\,\,\text{where}\,\,Z(x,\pi_{\bm{\theta}_{t}})\coloneqq\int\pi_{\bm{\theta}_{t}}(y^{\prime}|x)e^{r(x,y^{\prime})/\tau}\mathop{}\!\mathrm{d}y^{\prime}. \tag{4} $$ Note that Eq. (4) is equivalent to the following: for any pair of responses $y_{1}$ and $y_{2}$ , | | $\displaystyle\frac{\pi(y_{1}|x)}{\pi(y_{2}|x)}=\frac{\pi_{\bm{\theta}_{t}}(y_{1}|x)}{\pi_{\bm{\theta}_{t}}(y_{2}|x)}\exp\bigg(\frac{r(x,y_{1})-r(x,y_{2})}{\tau}\bigg).$ | | | --- | --- | --- | Taking logarithm of both sides, we have this pairwise consistency condition: $$ \displaystyle r_{1}-\tau\cdot\big(\log\pi(y_{1}|x)-\log\pi_{\bm{\theta}_{t}}(y_{1}|x)\big)=r_{2}-\tau\cdot\big(\log\pi(y_{2}|x)-\log\pi_{\bm{\theta}_{t}}(y_{2}|x)\big). \tag{5} $$ #### Step 2: surrogate loss with finite samples. Given a prompt $x$ and $K$ responses $y_{1},\dots,y_{K}$ , we define the following mean-squared surrogate loss that enforces the consistency condition: $$ \widehat{L}({\bm{\theta}};x,\pi_{\bm{\theta}_{t}})\coloneqq\frac{1}{K^{2}}\sum_{1\leq i<j\leq K}\frac{(a_{i}-a_{j})^{2}}{(1+\tau)^{2}},\,\,\text{where}\,\,a_{i}\coloneqq r_{i}-\tau\Big(\log\pi_{\bm{\theta}}(y_{i}|x)-\log\pi_{\bm{\theta}_{t}}(y_{i}|x)\Big). \tag{6} $$ Here, we normalize $a_{i}-a_{j}$ by $1+\tau$ to account for the loss scale. In theory, if this surrogate loss is defined by infinite samples with sufficient coverage of the action space, then its minimizer is the same as the optimal policy for the surrogate objective in Eq. (3). #### Step 3: one gradient step of the surrogate loss. Let us conduct further analysis for $(a_{i}-a_{j})^{2}$ . The trick here is that, if we take only one gradient step of this loss at $\bm{\theta}=\bm{\theta}_{t}$ , then the values of $\log\pi_{\bm{\theta}}(y_{i}|x)-\log\pi_{\bm{\theta}_{t}}(y_{i}|x)$ and $\log\pi_{\bm{\theta}}(y_{j}|x)-\log\pi_{\bm{\theta}_{t}}(y_{j}|x)$ are simply zero. As a result, | | $\displaystyle\nabla_{\bm{\theta}}{(a_{i}-a_{j})^{2}}\big|_{\bm{\theta}_{t}}={-2\tau}\,(r_{i}-r_{j})\Big(\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}|x)\big|_{\bm{\theta}_{t}}-\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{j}|x)\big|_{\bm{\theta}_{t}}\Big)\quad\Rightarrow$ | | | --- | --- | --- | Putting these back to the surrogate loss defined in Eq. (6), we end up with this policy update step: $$ \bm{g}\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big)=\frac{2\tau}{(1+\tau)^{2}}\cdot\frac{1}{K}\sum_{1\leq i\leq K}\big(r_{i}-\overline{r}\big)\,\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}\,|\,x). \tag{7} $$ That’s it! We have just derived the group-relative REINFORCE method, but without any on-policy assumption about the distribution of training data $\{x,\{y_{i},r_{i}\}_{1\leq i\leq K}\}$ . The regularization coefficient $\tau>0$ controls the update step size; a larger $\tau$ effectively corresponds to a smaller learning rate. #### Summary and remarks. Figure 1 visualizes the proposed interpretation of what REINFORCE is actually doing. The curve going through $\bm{\theta}_{t}\rightarrow\bm{\theta}_{t+1}\rightarrow\widetilde{\bm{\theta}}_{t+1}\rightarrow\bm{\theta}^{\star}$ stands for the ideal optimization trajectory from $\bm{\theta}_{t}$ to the optimal policy model parametrized by $\bm{\theta}^{\star}$ , if the algorithm solves each intermediate surrogate objective $J(\bm{\theta};\pi_{\bm{\theta}_{t}})$ / surrogate loss $\widehat{L}(\bm{\theta};\pi_{\bm{\theta}_{t}})$ exactly at each iteration $t$ . In comparison, REINFORCE is effectively taking a single gradient step of the surrogate loss and immediately moving on to the next iteration $\bm{\theta}_{t+1}$ with a new surrogate objective. Two remarks are in place. (1) Our derivation of group-relative REINFORCE can be generalized to multi-step RL settings, by replacing a response $y$ in the previous analysis with a full trajectory consisting of multiple turns of agent-environment interaction. For example, regarding the surrogate objective in Eq. (3), we need to replace the response-level reward and KL divergence with their trajectory-level counterparts. Interested readers might refer to Appendix A for the full analysis. (2) The above analysis suggests that we might interpret group-relative REINFORCE from a pointwise or pairwise perspective. While the policy update in Eq. (7) is stated in a pointwise manner, we have also seen that, at each iteration, REINFORCE is implicitly enforcing the pairwise consistency condition in Eq. (5) among multiple responses. This allows us the flexibility to choose whichever perspective that offers more intuition for our analysis later in this work. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Gradient-Based Optimization Process ### Overview The image depicts a mathematical optimization process involving parameter updates and trajectory visualization. It combines equations describing iterative updates with a graphical representation of the optimization landscape and parameter evolution. ### Components/Axes 1. **Equations**: - Left side: - `θ_{t+1} = θ_t - η_t∇_θL(θ; π_θ_t) |_{θ=θ_t}` (Gradient descent update rule) - `θ̃_{t+1} = arg max_θ J(θ; π_θ_t) = arg min_θ L̃(θ; π_θ_t)` (Optimization objective equivalence) - Right side: - Graph showing parameter space with points θ_t, θ̃_{t+1}, and θ* 2. **Graph Elements**: - **Points**: - θ_t (black dot, starting position) - θ̃_{t+1} (purple dot, next parameter estimate) - θ* (black dot, optimal parameter) - **Arrows**: - Blue arrow: Gradient vector ∇_θL(θ; π_θ_t) pointing from θ_t to θ̃_{t+1} - Purple dashed line: Optimization trajectory from θ_t to θ* - **Axes**: Implicit parameter space axes (no explicit labels) 3. **Color Coding**: - Blue: Gradient direction (∇_θL) - Purple: Optimization trajectory (J(θ) maximization/L̃ minimization) ### Detailed Analysis - **Gradient Descent Update**: - θ_{t+1} is calculated by subtracting the gradient of the loss function L scaled by learning rate η_t from the current parameter θ_t. - The gradient ∇_θL(θ; π_θ_t) is evaluated at the current parameter θ_t. - **Optimization Objective**: - θ̃_{t+1} is determined by maximizing the objective function J(θ; π_θ_t), which is equivalent to minimizing the loss function L̃(θ; π_θ_t). - This represents a dual perspective of the same optimization problem. - **Trajectory Visualization**: - The purple dashed line shows the path from the current parameter θ_t toward the optimal parameter θ*. - The blue gradient vector indicates the direction of steepest ascent for the objective function J. ### Key Observations 1. The gradient vector (blue) points directly toward θ̃_{t+1}, confirming it as the next parameter estimate. 2. The optimization trajectory (purple) curves toward θ*, suggesting convergence to the optimal solution. 3. θ̃_{t+1} lies between θ_t and θ* on the trajectory, indicating iterative progress. 4. The equivalence between maximizing J and minimizing L̃ is visually represented through the shared trajectory. ### Interpretation This diagram illustrates a gradient-based optimization algorithm, likely in machine learning or statistical inference. The process involves: 1. **Parameter Update**: Adjusting θ_t using gradient descent to compute θ̃_{t+1}. 2. **Objective Optimization**: Simultaneously framing the problem as maximizing J or minimizing L̃. 3. **Convergence**: The trajectory (purple line) demonstrates movement toward the optimal parameter θ*. The visualization emphasizes the relationship between gradient direction and optimization trajectory, showing how iterative updates navigate the parameter space toward the optimal solution. The equivalence between the maximization and minimization formulations highlights the duality in optimization problems, where different objective formulations can lead to the same parameter updates. </details> Figure 1: A visualization of our off-policy interpretation for group-relative REINFORCE. Here $\widehat{L}(\bm{\theta};\pi_{\bm{\theta}_{t}})={\mathbb{E}}_{x\sim\widehat{D}}[\widehat{L}(\bm{\theta};x,\pi_{\bm{\theta}_{t}})]$ , where $\widehat{D}$ is the sampling distribution for prompts, and $\widehat{L}(\bm{\theta};x,\pi_{\bm{\theta}_{t}})$ is the surrogate loss defined in Eq. (6) for a specific prompt $x$ . ## 3 Pitfalls and augmentations Although we have provided a native off-policy interpretation for REINFORCE, it certainly does not guarantee convergence to the optimal policy when given arbitrary training data. This section identifies pitfalls that could undermine vanilla REINFORCE, which motivate two principles for augmentations in off-policy settings. #### Pitfalls of vanilla REINFORCE. In Figure 1, we might expect that ideally, (1) $\widetilde{\bm{\theta}}_{t+1}-\bm{\theta}_{t}$ aligns with the direction of $\bm{\theta}^{\star}-\bm{\theta}_{t}$ ; and (2) $\bm{\theta}_{t+1}-\bm{\theta}_{t}$ aligns with the direction of $\widetilde{\bm{\theta}}_{t+1}-\bm{\theta}_{t}$ . One pitfall, however, is that even if both conditions hold, they do not necessarily imply that $\bm{\theta}_{t+1}-\bm{\theta}_{t}$ should align well with $\bm{\theta}^{\star}-\bm{\theta}_{t}$ . That is, $\langle\widetilde{\bm{\theta}}_{t+1}-\bm{\theta}_{t},\bm{\theta}^{\star}-\bm{\theta}_{t}\rangle>0$ and $\langle\bm{\theta}_{t+1}-\bm{\theta}_{t},\widetilde{\bm{\theta}}_{t+1}-\bm{\theta}_{t}\rangle>0$ do not imply $\langle\bm{\theta}_{t+1}-\bm{\theta}_{t},\bm{\theta}^{\star}-\bm{\theta}_{t}\rangle>0$ . Moreover, it is possible that $\bm{\theta}_{t+1}-\bm{\theta}_{t}$ might not align well with $\widetilde{\bm{\theta}}_{t+1}-\bm{\theta}_{t}$ . Recall from Eq. (7) that, from $\bm{\theta}_{t}$ to $\bm{\theta}_{t+1}$ , we take one gradient step for a surrogate loss that enforces the pairwise consistency condition among a finite number of samples. Given the enormous action space of an LLM, some implicit assumptions about the training data (e.g., balancedness and coverage) would be needed to ensure that the gradient aligns well with the direction towards the optimum of the surrogate objective, namely $\widetilde{\bm{\theta}}_{t+1}-\bm{\theta}_{t}$ . In fact, without a mechanism that ensures boundedness of policy update under a sub-optimal data distribution, vanilla REINFORCE could eventually converge to a sub-optimal policy. Let us show this with a minimal example in a didactic 3-arm bandit setting. Suppose that there are three actions $\{a_{j}\}_{1\leq j\leq 3}$ with rewards $\{r(a_{j})\}$ . Consider $K$ training samples $\{y_{i}\}_{1\leq i\leq K}$ , where $y_{i}\in\{a_{j}\}_{1\leq j\leq 3}$ is sampled from some behavior policy $\pi_{\textsf{b}}$ . Denote by $\mu_{r}\coloneqq\sum_{1\leq j\leq 3}\pi_{\textsf{b}}(a_{j})r(a_{j})$ the expected reward under $\pi_{\textsf{b}}$ , and $\overline{r}\coloneqq\sum_{i}r(y_{i})/K$ the average reward of training samples. We consider the softmax parameterization, i.e., $\pi_{\bm{\theta}}(a_{j})=e^{\theta_{j}}/\sum_{\ell}e^{\theta_{\ell}}$ for a policy parameterized by $\bm{\theta}\in{\mathbb{R}}^{3}$ . A standard fact is that $\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(a_{j})=\bm{e}_{j}-\pi_{\bm{\theta}}$ , where $\bm{e}_{j}\in{\mathbb{R}}^{3}$ is a one-hot vector with value 1 at entry $j$ . Now we examine the policy update direction of REINFORCE, as $K\rightarrow\infty$ : | | $\displaystyle\bm{g}$ | $\displaystyle=\frac{1}{K}\sum_{1\leq i\leq K}(r(y_{i})-\overline{r})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i})\rightarrow\sum_{1\leq j\leq 3}\pi_{\textsf{b}}(a_{j})(r(a_{j})-\mu_{r})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(a_{j})$ | | | --- | --- | --- | --- | For example, if $\bm{r}=[r(a_{j})]_{1\leq j\leq 3}=[0,0.8,1]$ and $\pi_{\textsf{b}}=[0.3,0.6,0.1]$ , then basic calculation says $\mu_{r}=0.58$ , $\bm{r}-\mu_{r}=[-0.58,0.22,0.42]$ , and finally $g_{2}=\pi_{\textsf{b}}(a_{2})(r(a_{2})-\mu_{r})>\pi_{\textsf{b}}(a_{3})(r(a_{3})-\mu_{r})=g_{3}$ , which implies that the policy will converge to the sub-optimal action $a_{2}$ . #### Two principles for augmenting REINFORCE. The identified pitfalls of vanilla REINFORCE suggest two general principles for augmenting REINFORCE in off-policy scenarios: - One is to regularize the policy update step, ensuring that the optimization trajectory remains bounded and reasonably stable when given training data from a sub-optimal distribution; - The other is to steer the policy update direction, by actively weighting the training samples rather than naively using them as is. These two principles are not mutually exclusive, and might be integrated within a single algorithm. We will see in the next section that many RL algorithms can be viewed as instantiations of them. ## 4 Rethinking the rationales behind recent RL algorithms This section revisits various RL algorithms through a unified lens — the native off-policy interpretation of group-relative REINFORCE and its augmentations — and demystifies some common myths about their working mechanisms. Our main findings are summarized as follows: | ID | Finding | Analysis & Experiments | | --- | --- | --- | | F1 | GRPO’s effectiveness in off-policy settings stems from clipping as regularization rather than importance sampling. A wider clipping range than usual often accelerates training without harming stability. | Section 4.1, Figures 3, 4, 6, 9, 10 | | F2 | Kimi’s OPMD and Meta’s AsymRE can be interpreted as REINFORCE loss + regularization loss, a perspective that is complementary to the rationales in their original papers. | Section 4.2, Figure 11 | | F3 | Data-oriented heuristics — such as dropping excess negatives or up-weighting high-reward rollouts — fit naturally into our off-policy view and show strong empirical performance. | Section 4.3, Figures 5, 6, 7 | #### Experimental setup. We primarily consider two off-policy settings in our experiments, specified by the sync_interval and sync_offset parameters in the Trinity-RFT framework (Pan et al., 2025). sync_interval specifies the number of generated rollout batches (each corresponding to one gradient step) between two model synchronization operations, while sync_offset specifies the relative lag between the generation and consumption of each batch. These parameters can be deliberately set to large values in practice, for improving training efficiency via pipeline parallelism and reduced frequency of model synchronization. In addition, $\texttt{sync\_offset}>1$ serves to simulate realistic scenarios where environmental feedback could be delayed. We also consider a stress-test setting that only allows access to offline data generated by the initial policy model. See Figure 2 for an illustration of these off-policy settings, and Appendix B.2 for further details. We conduct experiments on math reasoning tasks like GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021) and Guru (math subset) (Cheng et al., 2025), as well as tool-use tasks like ToolACE (Liu et al., 2025a). We consider models of different families and scales, including Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct (Qwen-Team, 2025), Llama-3.1-8B-Instruct, and Llama-3.2-3B-Instruct (Dubey et al., 2024). Additional experiment details can be found in Appendix B. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: Synchronization Mode Visualization ### Overview The image contains two side-by-side diagrams comparing synchronization modes in a system. Each diagram illustrates three horizontal sections labeled **Rollout**, **Buffer**, and **Training**, with vertical "sync" markers indicating synchronization points. The diagrams differ in their `sync_interval` and `sync_offset` parameters, affecting the alignment of sections. ### Components/Axes - **Sections**: - **Rollout**: Purple blocks with sequential numbering (e.g., 0, 1, 2, 3). - **Buffer**: Yellow blocks spanning the full width of each diagram. - **Training**: Blue blocks with sequential numbering (e.g., 0, 1, 2, 3). - **Vertical Markers**: Labeled "sync" to denote synchronization points. - **Parameters**: - **Left Diagram**: `sync_interval = 4`, `sync_offset = 0`. - **Right Diagram**: `sync_interval = 1`, `sync_offset = 4`. ### Detailed Analysis #### Left Diagram (`sync_interval = 4`, `sync_offset = 0`) - **Rollout**: Blocks 0–3 (purple) align with **Training** blocks 0–3 (blue) at the first sync point (x=4). - **Buffer**: Spans the entire width, acting as a continuous layer. - **Training**: Blocks 0–3 (blue) align with Rollout at x=4. Subsequent sync points occur every 4 units (x=8, x=12, etc.). #### Right Diagram (`sync_interval = 1`, `sync_offset = 4`) - **Rollout**: Blocks 0–4 (purple) extend beyond the first sync point (x=1). - **Training**: Blocks 0–3 (blue) align with Rollout at x=4 (offset by 4 units). - **Sync Frequency**: Sync markers appear every 1 unit, but the offset causes misalignment between Rollout and Training sections. ### Key Observations 1. **Alignment**: - Left diagram shows perfect alignment between Rollout and Training at sync points due to `sync_offset = 0`. - Right diagram’s `sync_offset = 4` creates a phase shift, delaying alignment between sections. 2. **Sync Interval Impact**: - Larger `sync_interval` (4) reduces sync frequency, creating longer uninterrupted sections. - Smaller `sync_interval` (1) increases sync frequency but introduces misalignment due to offset. 3. **Buffer Role**: The Buffer section remains uninterrupted in both diagrams, suggesting it acts as a stabilizing layer. ### Interpretation The diagrams demonstrate how synchronization parameters (`sync_interval` and `sync_offset`) influence system behavior: - **Left Diagram**: Optimal alignment for coordinated operations (e.g., training and rollout phases). - **Right Diagram**: Frequent syncs with offset may lead to inefficiencies or conflicts between sections. - **Buffer Function**: Likely serves as a decoupling layer to absorb variability between Rollout and Training phases. The visual trends confirm that synchronization strategy directly impacts system stability and resource utilization. </details> Figure 2: A visualization of the rollout-training scheduling in sync_interval = 4 (left) or sync_offset = 4 (right) modes. Each block denotes one batch of samples for one gradient step, and the number in it denotes the corresponding batch id. Training blocks are color-coded by data freshness, with lighter color indicating increasing off-policyness. ### 4.1 Demystifying myths about GRPO Recall that in GRPO, the advantage for each response $y_{i}$ is defined as $A_{i}={(r_{i}-\overline{r})}/{\sigma_{r}}$ , where $\overline{r}$ and $\sigma_{r}$ denote the within-group mean and standard deviation of the rewards $\{r_{i}\}_{1\leq i\leq K}$ respectively. We consider the practical implementation of GRPO with token-wise importance-sampling (IS) weighting and clipping, whose loss function for a specific prompt $x$ and responses $\{y_{i}\}$ is In our experiments with GRPO, we neglect KL regularization with respect to an extra reference model, or entropy regularization that encourages output diversity. Recent works (Yu et al., 2025; Liu et al., 2025b) have shown that these practical techniques are often unnecessary. | | $\displaystyle\widehat{L}=\frac{1}{K}\sum_{1\leq i\leq K}\sum_{1\leq t\leq|y_{i}|}\min\bigg\{\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}A_{i},\,\operatorname{clip}\Big(\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})},1-\epsilon_{\textsf{low}},1+\epsilon_{\textsf{high}}\Big)A_{i}\bigg\},$ | | | --- | --- | --- | where $\pi_{\textsf{old}}$ denotes the older policy version that generated this group of rollout data. The gradient of this loss can be written as (Schulman et al., 2017) $$ \bm{g}\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big)=\frac{1}{K}\sum_{1\leq i\leq K}\sum_{1\leq t\leq|y_{i}|}\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})\cdot A_{i}\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}M_{i}^{t}, $$ where $M_{i}^{t}$ denotes a one-side clipping mask: $$ M_{i}^{t}=\mathbbm{1}\bigg(A_{i}>0,\;\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\leq 1+\epsilon_{\textsf{high}}\bigg)+\mathbbm{1}\bigg(A_{i}<0,\;\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\geq 1-\epsilon_{\textsf{low}}\bigg). \tag{8} $$ #### Ablation study with the REC series. To isolate the roles of importance sampling and clipping, we consider a series of RE INFORCE-with- C lipping (REC) algorithms. Due to space limitation, we defer our studies of more clipping mechanisms to Appendix B.3, and focus on REC with one-side clipping in this section. More specifically, REC-OneSide-IS removes advantage normalization in GRPO (to reduce variability), and REC-OneSide-NoIS further removes IS weighting: | | $\displaystyle\text{{REC-OneSide-IS}:}\;\;\bm{g}$ | $\displaystyle=\frac{1}{K}\sum_{1\leq i\leq K}\sum_{1\leq t\leq|y_{i}|}\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})\cdot(r_{i}-\overline{r})\,\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\,M_{i}^{t},$ | | | --- | --- | --- | --- | <details> <summary>x3.png Details</summary>  ### Visual Description ## Line Graphs: Algorithm Performance Comparison Across Training Steps ### Overview The image contains six line graphs comparing the performance of reinforcement learning algorithms (REINFORCE, REC-OneSide-NoIS, REC-OneSide-IS, GRPO) across three scenarios: `sync_interval=20`, `sync_offset=10`, and `offline`. Each graph tracks **Evaluation Accuracy** (top row) and **Training Reward** (bottom row) over 150 training steps. Key trends include algorithm-specific performance variations under different synchronization settings. --- ### Components/Axes 1. **X-Axis**: Training Steps (0–150, increments of 50) 2. **Y-Axes**: - Top Row: Evaluation Accuracy (0.0–0.8) - Bottom Row: Training Reward (0.0–1.0) 3. **Legends**: - **Blue**: REINFORCE - **Purple**: REC-OneSide-NoIS (0.2) - **Green**: GRPO - **Dotted Purple**: REC-OneSide-IS (0.2) - **Solid Purple**: REC-OneSide-NoIS (0.6, 2.0) 4. **Graph Labels**: - Top Row: `sync_interval=20`, `sync_offset=10`, `offline` - Bottom Row: Corresponding Training Reward graphs --- ### Detailed Analysis #### Top Row (Evaluation Accuracy) 1. **`sync_interval=20`**: - **REINFORCE (Blue)**: Starts at ~0.3, dips to ~0.1 at 50 steps, then recovers to ~0.5 by 150 steps. - **REC-OneSide-NoIS (0.2) (Purple)**: Steady increase from ~0.3 to ~0.7. - **GRPO (Green)**: Gradual rise from ~0.3 to ~0.6. - **REC-OneSide-IS (0.2) (Dotted Purple)**: Consistent growth from ~0.3 to ~0.7. 2. **`sync_offset=10`**: - **REINFORCE (Blue)**: Sharp drop to ~0.2 at 50 steps, recovers to ~0.5 by 150 steps. - **REC-OneSide-NoIS (0.2) (Purple)**: Stable increase from ~0.3 to ~0.6. - **GRPO (Green)**: Flat line at ~0.5. - **REC-OneSide-IS (0.2) (Dotted Purple)**: Slight upward trend from ~0.4 to ~0.6. 3. **`offline`**: - **REINFORCE (Blue)**: Drops to ~0.1 at 50 steps, remains flat. - **REC-OneSide-NoIS (0.2) (Purple)**: Stable at ~0.5. - **GRPO (Green)**: Flat at ~0.4. - **REC-OneSide-IS (0.2) (Dotted Purple)**: Slight decline from ~0.5 to ~0.4. #### Bottom Row (Training Reward) 1. **`sync_interval=20`**: - **REINFORCE (Blue)**: Peaks at ~0.7, crashes to ~0.0 at 50 steps, then recovers to ~0.5. - **REC-OneSide-NoIS (0.2) (Purple)**: Stable ~0.5–0.6. - **GRPO (Green)**: Gradual rise from ~0.4 to ~0.6. - **REC-OneSide-IS (0.2) (Dotted Purple)**: Stable ~0.5–0.6. 2. **`sync_offset=10`**: - **REINFORCE (Blue)**: Peaks at ~0.7, drops to ~0.2 at 50 steps, recovers to ~0.5. - **REC-OneSide-NoIS (0.2) (Purple)**: Stable ~0.5–0.6. - **GRPO (Green)**: Flat at ~0.5. - **REC-OneSide-IS (0.2) (Dotted Purple)**: Slight increase from ~0.5 to ~0.6. 3. **`offline`**: - **REINFORCE (Blue)**: Peaks at ~0.7, drops to ~0.1 at 50 steps, remains flat. - **REC-OneSide-NoIS (0.2) (Purple)**: Stable ~0.4–0.5. - **GRPO (Green)**: Flat at ~0.4. - **REC-OneSide-IS (0.2) (Dotted Purple)**: Stable ~0.4–0.5. --- ### Key Observations 1. **REINFORCE Instability**: - Consistently underperforms in evaluation accuracy and training reward under synchronization settings (`sync_interval=20`, `sync_offset=10`). - Training reward crashes to near-zero at 50 steps in all scenarios, suggesting sensitivity to hyperparameters or environment settings. 2. **REC-OneSide-NoIS (0.2) Robustness**: - Maintains stable evaluation accuracy (~0.5–0.7) and training reward (~0.5–0.6) across all scenarios. 3. **GRPO Performance**: - Evaluation accuracy plateaus at ~0.5–0.6 in synchronization scenarios but underperforms in `offline` (~0.4). 4. **REC-OneSide-IS (0.2) Consistency**: - Shows steady improvement in evaluation accuracy and stable training reward, outperforming GRPO in synchronization settings. 5. **Offline Scenario**: - All algorithms exhibit reduced performance compared to synchronization settings, with REINFORCE being the most affected. --- ### Interpretation The data suggests that **REINFORCE** struggles with synchronization-dependent tasks, likely due to its high variance in gradient estimation. In contrast, **REC-OneSide-NoIS (0.2)** and **REC-OneSide-IS (0.2)** demonstrate robustness, maintaining performance across scenarios. The `offline` setting degrades performance for all algorithms, highlighting the importance of synchronization in training stability. GRPO’s flat performance in `offline` may indicate limitations in handling non-synchronized environments. The training reward crashes for REINFORCE suggest it requires careful tuning to avoid instability, while REC variants balance exploration and exploitation more effectively. </details> Figure 3: Empirical results for REC algorithms on GSM8k with Qwen2.5-1.5B-Instruct. Training reward curves are smoothed with a running-average window of size 3. Numbers in the legend denote clipping parameters $\epsilon_{\textsf{low}},\epsilon_{\textsf{high}}$ . <details> <summary>x4.png Details</summary>  ### Visual Description ## Line Graphs: Training Reward and Clipping Fraction vs Training Steps ### Overview The image contains two side-by-side line graphs. The left graph plots **Training Reward** against **Training Steps**, while the right graph plots **Clipping Fraction** against **Training Steps**. Both graphs share the same x-axis (Training Steps: 0–400) and y-axis scales (logarithmic for Clipping Fraction). Three data series are compared across both graphs, differentiated by color and line style. ### Components/Axes - **Left Graph (Training Reward)**: - **Y-axis**: "Training Reward" (linear scale: 0.25–1.00). - **X-axis**: "Training Steps" (0–400, increments of 100). - **Legend**: - **Solid Purple**: REC-OneSide-NoIS (0.2, 0.25). - **Dotted Purple**: REC-OneSide-IS (0.2, 0.25). - **Orange**: REC-Ring-NoIS (0.2, 0.25) & (0.6, 2.0). - **Right Graph (Clipping Fraction)**: - **Y-axis**: "Clipping Fraction" (logarithmic scale: 10⁻⁴–10⁻¹). - **X-axis**: "Training Steps" (0–400, increments of 100). - **Legend**: - **Solid Purple**: REC-OneSide-NoIS (0.6, 2.0). - **Orange**: REC-TwoSide-NoIS (0.2, 0.25). ### Detailed Analysis #### Left Graph (Training Reward): - **Solid Purple (REC-OneSide-NoIS (0.2, 0.25))**: - Starts at ~0.25, rises sharply to ~0.75 by 100 steps, then plateaus near 1.00. - Slight fluctuations but overall upward trend. - **Dotted Purple (REC-OneSide-IS (0.2, 0.25))**: - Similar trajectory to solid purple but with minor oscillations. - Ends slightly below 1.00 (~0.95). - **Orange (REC-Ring-NoIS (0.2, 0.25) & (0.6, 2.0))**: - Starts at ~0.25, rises to ~0.85 by 100 steps, then plateaus near 0.95. - Smoother curve with fewer fluctuations. #### Right Graph (Clipping Fraction): - **Solid Purple (REC-OneSide-NoIS (0.6, 2.0))**: - Starts at ~10⁻³, decreases to ~10⁻⁴ by 100 steps, then stabilizes. - Sharp initial drop, then minor oscillations. - **Orange (REC-TwoSide-NoIS (0.2, 0.25))**: - Starts at ~10⁻³, decreases to ~10⁻⁴ by 100 steps, then stabilizes. - Slightly higher initial values than solid purple. ### Key Observations 1. **Training Reward**: - All models improve over time, with REC-Ring-NoIS (orange) achieving the highest reward (~0.95) by 400 steps. - REC-OneSide-NoIS (solid purple) and REC-OneSide-IS (dotted purple) perform similarly but slightly lower. 2. **Clipping Fraction**: - Both models show a logarithmic decline, with REC-OneSide-NoIS (solid purple) achieving the lowest clipping fraction (~10⁻⁴) by 400 steps. - REC-TwoSide-NoIS (orange) has slightly higher clipping fractions initially but converges to similar values. 3. **Parameter Impact**: - Higher parameters (e.g., 0.6, 2.0) in REC-OneSide-NoIS correlate with better performance in both metrics. ### Interpretation The data suggests that increasing the parameters (e.g., 0.6, 2.0) in the REC-OneSide-NoIS model improves both training reward and reduces clipping fraction, indicating more stable and efficient learning. The REC-Ring-NoIS model (orange) balances performance across both metrics, while the REC-OneSide-IS model (dotted purple) shows minor oscillations but comparable results to REC-OneSide-NoIS. The logarithmic scale for clipping fraction highlights the exponential decay in errors, emphasizing the importance of minimizing clipping for robust training. </details> Figure 4: Empirical results for REC on ToolACE with Llama-3.2-3B-Instruct. Training reward curves are smoothed with a running-average window of size 3. Details about REC-TwoSide and REC-Ring are provided in Appendix B.3. #### Experiments. Figure 3 presents GSM8k results with Qwen2.5-1.5B-Instruct in various off-policy settings. REC-OneSide-IS / NoIS and GRPO (with the same $\epsilon_{\textsf{low}}=\epsilon_{\textsf{high}}=0.2$ ) have nearly identical performance, indicating that importance sampling is non-essential, whereas the collapse of REINFORCE highlights the critical role of clipping. Radically enlarging $(\epsilon_{\textsf{low}},\epsilon_{\textsf{high}})$ to $(0.6,2.0)$ accelerates REC-OneSide-NoIS without compromising stability in both sync_interval = 20 and sync_offset = 10 settings. Similar patterns also appear in Figure 4 (ToolAce with Llama-3.2-3B-Instruct) and other results in Appendix B. As for the stress-test (“offline”) setting, Figure 3 reveals an intrinsic trade-off between the speed and stability of policy improvement, motivating future work toward better algorithms that achieve both. ### 4.2 Understanding Kimi’s OPMD and Meta’s AsymRE Besides clipping, another natural method is to add a regularization loss $R(\cdot)$ to vanilla REINFORCE: | | $\displaystyle\widehat{L}\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big)$ | $\displaystyle=-\frac{1}{K}\sum_{i\in[K]}(r_{i}-\overline{r})\log\pi_{\bm{\theta}}(y_{i}\,|\,x)+\tau\cdot R\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big),$ | | | --- | --- | --- | --- | and take $\bm{g}=-\nabla_{\bm{\theta}}\widehat{L}$ . We show below that Kimi’s OPMD and Meta’s AsymRE are indeed special cases of this unified formula, with empirical validation of their efficacy deferred to Appendix B.5. #### Kimi’s OPMD. Kimi-Team (2025b) derives an OPMD variant by taking logarithm of both sides of Eq. (4), which leads to a consistency condition and further motivates the following surrogate loss: $$ \widetilde{L}=\frac{1}{K}\sum_{1\leq i\leq K}\bigg(r_{i}-\tau\log Z(x,\pi_{\bm{\theta}_{t}})-\tau\,\Big(\log\pi_{\bm{\theta}}(y_{i}\,|\,x)-\log\pi_{\bm{\theta}_{t}}(y_{i}|x)\Big)\bigg)^{2}. $$ With $K$ responses generated by $\pi_{\textsf{old}}=\pi_{\bm{\theta}_{t}}$ , the term $\tau\log Z(x,\pi_{\bm{\theta}_{t}})$ can be approximated by a finite-sample estimate $\tau\log(\sum_{i}e^{r_{i}/\tau}/K)$ , which can be further approximated by the mean reward $\overline{r}=\sum_{i}r_{i}/K$ if $\tau$ is large. With these approximations, the gradient of $\widetilde{L}$ becomes equivalent to that of the following loss (which is the final version of Kimi’s OPMD): $$ \widehat{L}=-\frac{1}{K}\sum_{1\leq i\leq K}(r_{i}-\overline{r})\log\pi_{\bm{\theta}}(y_{i}\,|\,x)+\frac{\tau}{2K}\sum_{1\leq i\leq K}\Big(\log\pi_{\bm{\theta}}(y_{i}\,|\,x)-\log\pi_{\textsf{old}}(y_{i}\,|\,x)\Big)^{2}. $$ In comparison, our analysis in Sections 2 and 3 suggests that this is in itself a principled loss function for off-policy RL, adding a mean-squared regularization loss to the vanilla REINFORCE loss. #### Meta’s AsymRE. AsymRE (Arnal et al., 2025) modifies REINFORCE by tuning down the baseline (from $\overline{r}$ to $\overline{r}-\tau$ ) in advantage calculation, which was motivated by the intuition of prioritizing learning from positive samples and justified by multi-arm bandit analysis in the original paper. We offer an alternative interpretation for AsymRE by rewriting its loss function: | | $\displaystyle\widehat{L}$ | $\displaystyle=-\frac{1}{K}\sum_{i}\Big(r_{i}-(\overline{r}-\tau)\Big)\log\pi_{\bm{\theta}}(y_{i}\,|\,x)=-\frac{1}{K}\sum_{i}(r_{i}-\overline{r})\log\pi_{\bm{\theta}}(y_{i}\,|\,x)-\frac{\tau}{K}\sum_{i}\log\pi_{\bm{\theta}}(y_{i}\,|\,x).$ | | | --- | --- | --- | --- | Note that the first term on the right-hand side is the REINFORCE loss, and the second term serves as regularization, enforcing imitation of responses from an older version of the policy model. For the latter, we may also add a term that is independent of $\bm{\theta}$ to it and take the limit $K\rightarrow\infty$ : | | $\displaystyle-\frac{1}{K}\sum_{1\leq i\leq K}\log\pi_{\bm{\theta}}(y_{i}\,|\,x)+\frac{1}{K}\sum_{1\leq i\leq K}\log\pi_{\textsf{old}}(y_{i}\,|\,x)=\frac{1}{K}\sum_{1\leq i\leq K}\log\frac{\pi_{\textsf{old}}(y_{i}\,|\,x)}{\pi_{\bm{\theta}}(y_{i}\,|\,x)}$ | | | --- | --- | --- | which turns out to be a finite-sample approximation of KL regularization. ### 4.3 Understanding data-weighting methods We now shift our attention to the second principle for augmenting REINFORCE, i.e., actively shaping the training data distribution. #### Pairwise weighting. Recall from Section 2 that we define the surrogate loss in Eq. (6) as an unweighted sum of pairwise mean-squared losses. However, if we have certain knowledge about which pairs are more informative for RL training, we may assign higher weights to them. This motivates generalizing $\sum_{i<j}(a_{i}-a_{j})^{2}$ to $\sum_{i<j}w_{i,j}(a_{i}-a_{j})^{2}$ , where $\{w_{i,j}\}$ are non-negative weights. Assuming that $w_{i,j}=w_{j,i}$ and following the steps in Section 2, we end up with $$ \bm{g}\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big)=\frac{1}{K}\sum_{1\leq i\leq K}\Big(\sum_{1\leq j\leq K}w_{i,j}\Big)\bigg(r_{i}-\frac{\sum_{j}w_{i,j}r_{j}}{\sum_{j}w_{i,j}}\bigg)\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}\,|\,x). $$ In the special case where $w_{i,j}=w_{i}w_{j}$ , this becomes $$ \bm{g}=\Big(\sum_{j}w_{j}\Big)\;\frac{1}{K}\sum_{1\leq i\leq K}w_{i}\big(r_{i}-\overline{r}_{w}\big)\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}\,|\,x),\,\,\text{where}\,\,\overline{r}_{w}\coloneqq\frac{\sum_{j}w_{j}r_{j}}{\sum_{j}w_{j}}. \tag{9} $$ Based on this, we investigate two RE INFORCE-with- d ata-weighting (RED) methods. #### RED-Drop : sample dropping. The idea is to use a filtered subset $\mathcal{S}\subseteq[K]$ of responses for training; for example, the Kimi-Researcher technical blog (Kimi-Team, 2025a) proposes to “discard some negative samples strategically”, as negative gradients increase the risk of entropy collapse. This is indeed a special case of Eq. (9), by setting $w_{i}=\sqrt{K}/|\mathcal{S}|$ for $i\in\mathcal{S}$ and $0$ otherwise: $$ \bm{g}\big(\bm{\theta};x,\{y_{i},r_{i}\}_{1\leq i\leq K}\big)=\frac{1}{|\mathcal{S}|}\sum_{i\in\mathcal{S}}(r_{i}-\overline{r}_{\mathcal{S}})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}\,|\,x),\,\,\text{where}\,\,\overline{r}_{\mathcal{S}}=\frac{1}{|\mathcal{S}|}\sum_{i\in\mathcal{S}}r_{i}. \tag{10} $$ While this is no longer an unbiased estimate of policy gradient even if all responses are sampled from the current policy, it is still well justified by our off-policy interpretation of REINFORCE. #### RED-Weight : pointwise loss weighting. Another approach for prioritizing high-reward responses is to directly up-weight their gradient terms in Eq. (1a). To better understand the working mechanism of this seemingly heuristic method, we rewrite its policy update: | | $\displaystyle\bm{g}$ | $\displaystyle=\sum_{1\leq i\leq K}w_{i}(r_{i}-\overline{r})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}|x)=\sum_{1\leq i\leq K}w_{i}(r_{i}-\overline{r}_{w}+\overline{r}_{w}-\overline{r})\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}|x)$ | | | --- | --- | --- | --- | This is the pairwise-weighted REINFORCE gradient in Eq. (9), plus a regularization term (weighted by $\overline{r}_{w}-\overline{r}>0$ ) that resembles the one in AsymRE but prioritizes imitating higher-reward responses, echoing the finding from offline RL literature (Hong et al., 2023a, b) that regularizing against high-reward trajectories can be more effective than conservatively imitating all trajectories in the dataset. #### Implementation details. Below are the concrete instantiations adopted in our empirical studies: - RED-Drop: When the number of negative samples in a group exceeds the number of positive ones, we randomly drop the excess negatives so that positives and negatives are balanced. After this subsampling step, we recompute the advantages using the remaining samples, which are then fed into the loss. - RED-Weight: Each sample $i$ is weighted by $w_{i}=\exp({A_{i}}/{\tau})$ , where $A_{i}$ denotes its advantage estimate and $\tau>0$ is a temperature parameter controlling the sharpness of weighting. This scheme amplifies high-advantage samples while down-weighting low-advantage ones. We fix $\tau=1$ for all experiments. <details> <summary>x5.png Details</summary>  ### Visual Description ## Line Graphs: Algorithm Performance Comparison ### Overview The image contains four line graphs arranged in a 2x2 grid, comparing the performance of four reinforcement learning algorithms (REINFORCE, RED-Drop, RED-Weight, REC-OneSide-NoIS) across different training metrics. The graphs show evaluation accuracy and training reward trends over training steps, with distinct performance patterns emerging between the algorithms. ### Components/Axes 1. **Top-Left Graph** - **Title**: "on-policy" - **Y-Axis**: Evaluation Accuracy (0.0 to 0.8) - **X-Axis**: Training Steps (0 to 150) - **Legend**: - Blue: REINFORCE - Orange: RED-Drop - Purple: REC-OneSide-NoIS (0.6, 2.0) - Red: RED-Weight 2. **Top-Right Graph** - **Title**: "sync_interval = 20" - **Y-Axis**: Evaluation Accuracy (0.0 to 0.8) - **X-Axis**: Training Steps (0 to 150) - **Legend**: Same as top-left graph 3. **Bottom-Left Graph** - **Title**: "Training Reward" - **Y-Axis**: Training Reward (0.0 to 1.0) - **X-Axis**: Training Steps (0 to 150) - **Legend**: Same as top-left graph 4. **Bottom-Right Graph** - **Title**: "Training Reward" - **Y-Axis**: Training Reward (0.0 to 1.0) - **X-Axis**: Training Steps (0 to 150) - **Legend**: Same as top-left graph ### Detailed Analysis #### Top-Left Graph ("on-policy") - **Trend**: All algorithms show upward trajectories, plateauing near 0.75–0.8 evaluation accuracy. - **Data Points**: - REINFORCE (blue): Starts at ~0.35, peaks at ~0.78 by 150 steps. - RED-Drop (orange): Starts at ~0.38, peaks at ~0.77. - REC-OneSide-NoIS (purple): Starts at ~0.32, peaks at ~0.79. - RED-Weight (red): Starts at ~0.36, peaks at ~0.78. #### Top-Right Graph ("sync_interval = 20") - **Trend**: REINFORCE (blue) drops sharply after 50 steps, while others improve. - **Data Points**: - REINFORCE: Starts at ~0.45, drops to ~0.15 by 150 steps. - RED-Drop: Starts at ~0.42, peaks at ~0.72. - REC-OneSide-NoIS: Starts at ~0.40, peaks at ~0.76. - RED-Weight: Starts at ~0.44, peaks at ~0.74. #### Bottom-Left Graph ("Training Reward") - **Trend**: All algorithms show gradual improvement with minor fluctuations. - **Data Points**: - REINFORCE: Starts at ~0.5, peaks at ~0.95. - RED-Drop: Starts at ~0.52, peaks at ~0.98. - REC-OneSide-NoIS: Starts at ~0.55, peaks at ~0.97. - RED-Weight: Starts at ~0.53, peaks at ~0.96. #### Bottom-Right Graph ("Training Reward") - **Trend**: REINFORCE (blue) exhibits erratic drops, while others stabilize. - **Data Points**: - REINFORCE: Starts at ~0.5, drops to ~0.0 by 150 steps. - RED-Drop: Starts at ~0.52, stabilizes at ~0.95. - REC-OneSide-NoIS: Starts at ~0.55, stabilizes at ~0.94. - RED-Weight: Starts at ~0.53, stabilizes at ~0.93. ### Key Observations 1. **On-Policy Performance**: All algorithms achieve high evaluation accuracy (~0.75–0.8) under on-policy training, with REC-OneSide-NoIS slightly outperforming others. 2. **Sync Interval Impact**: REINFORCE’s evaluation accuracy collapses under sync_interval = 20, while other algorithms maintain performance. 3. **Training Reward Variance**: REINFORCE shows unstable training rewards in sync_interval scenarios, whereas RED-Drop and REC-OneSide-NoIS maintain consistent rewards. ### Interpretation - **Algorithm Robustness**: RED-Drop and REC-OneSide-NoIS demonstrate superior stability across training scenarios, suggesting better generalization. - **REINFORCE Limitations**: REINFORCE’s performance degrades significantly under sync_interval constraints, indicating sensitivity to hyperparameters or training dynamics. - **Training Reward Correlation**: Higher training rewards align with better evaluation accuracy, except for REINFORCE in sync_interval settings, where reward and accuracy decouple. - **Practical Implications**: Algorithms with adaptive mechanisms (e.g., RED-Drop, REC-OneSide-NoIS) may be preferable for real-world applications requiring stable training under varying conditions. *Note: All values are approximate, derived from visual inspection of the graphs.* </details> Figure 5: Empirical performance of RED-Drop and RED-Weight on GSM8k with Qwen2.5-1.5B-Instruct, in both on-policy and off-policy settings. Training reward curves are smoothed with a running-average window of size 3. #### Experiments. Figure 5 presents GSM8k results with Qwen2.5-1.5B-Instruct, which confirm the efficacy of RED-Drop and RED-Weight in on/off-policy settings, comparable to REC-OneSide-NoIS with enlarged $(\epsilon_{\textsf{low}},\epsilon_{\textsf{high}})$ . Figure 6 reports larger-scale experiments on Guru-Math with Qwen2.5-7B-Instruct, where RED-Weight achieves higher rewards than GRPO, with similar KL distance to the initial policy. Figure 7 further validates the efficacy of RED-Weight on MATH with Llama-3.1-8B-Instruct; compared to GRPO and REC-OneSide-NoIS, RED-Weight achieves higher rewards with lower KL divergence, while maintaining more stable entropy and response lengths. <details> <summary>x6.png Details</summary>  ### Visual Description ## Line Graphs: Training Reward and KL Divergence vs Training Steps (sync_interval = 20) ### Overview Two line graphs are presented side-by-side, both labeled with `sync_interval = 20`. The left graph tracks **Training Reward** (y-axis) against **Training Steps** (x-axis), while the right graph tracks **KL Divergence** (y-axis) against the same x-axis. Three methods are compared: **GRPO** (green), **REC-OneSide-NoIS (0.2)** (purple), and **RED-Weight** (orange). Both graphs show trends over 1500 training steps. --- ### Components/Axes - **Left Graph (Training Reward)**: - **X-axis**: Training Steps (0 to 1500, linear scale). - **Y-axis**: Training Reward (0 to 0.25, linear scale). - **Legend**: Located at the bottom, mapping colors to methods: - Green: GRPO - Purple: REC-OneSide-NoIS (0.2) - Orange: RED-Weight - **Right Graph (KL Divergence)**: - **X-axis**: Training Steps (0 to 1500, linear scale). - **Y-axis**: KL Divergence (10⁻² to 10⁰, logarithmic scale). - **Legend**: Same as the left graph. --- ### Detailed Analysis #### Left Graph (Training Reward) - **GRPO (green)**: Starts near 0.12, increases steadily to ~0.24 by 1500 steps. Smooth upward trend with minor fluctuations. - **REC-OneSide-NoIS (0.2) (purple)**: Similar trajectory to GRPO, peaking at ~0.23. Slightly more volatile but closely aligned with GRPO. - **RED-Weight (orange)**: Outperforms others, reaching ~0.26 by 1500 steps. Consistent upward slope with minor noise. #### Right Graph (KL Divergence) - **GRPO (green)**: Begins at ~10⁻², rises to ~10⁻¹ by 1500 steps. Gradual increase with occasional spikes. - **REC-OneSide-NoIS (0.2) (purple)**: Mirrors GRPO’s trend, ending near ~10⁻¹. Slightly smoother than GRPO. - **RED-Weight (orange)**: Starts at ~10⁻², peaks at ~10⁻¹ by 1500 steps. More pronounced fluctuations, including sharp spikes (e.g., ~10⁻¹⁵ at ~1000 steps). --- ### Key Observations 1. **Training Reward**: - All methods improve over time, but **RED-Weight** achieves the highest reward (~0.26 vs. ~0.24 for others). - GRPO and REC-OneSide-NoIS perform similarly, with GRPO slightly edging ahead in later steps. 2. **KL Divergence**: - All methods show increasing divergence, indicating growing deviation from a target distribution. - **RED-Weight** exhibits the highest divergence (~10⁻¹) and most instability (spikes), suggesting potential overfitting or optimization challenges. - GRPO and REC-OneSide-NoIS demonstrate more stable divergence patterns. 3. **Sync Interval**: - Both graphs share `sync_interval = 20`, implying synchronized updates every 20 steps. This may influence the observed trends in reward and divergence. --- ### Interpretation - **Performance Trade-off**: RED-Weight achieves the highest training reward but at the cost of higher KL divergence, suggesting it may prioritize reward maximization over distributional stability. - **Stability vs. Reward**: GRPO and REC-OneSide-NoIS balance reward and divergence better, with smoother KL curves. Their lower divergence might indicate more robust generalization. - **Spike Analysis**: The sharp KL divergence spikes in RED-Weight (e.g., ~10⁻¹⁵ at ~1000 steps) could reflect transient instability during training, possibly due to aggressive optimization or parameter updates. - **Sync Interval Impact**: The fixed `sync_interval = 20` might introduce periodic synchronization effects, visible in the periodic fluctuations of all lines. --- ### Spatial Grounding & Verification - **Legend Placement**: Bottom-center, clearly aligned with both graphs. Colors match line colors exactly (green, purple, orange). - **Axis Consistency**: Both graphs share identical x-axis labels and scales, ensuring direct comparability. - **Trend Verification**: Visual inspection confirms RED-Weight’s higher reward and divergence align with its orange line’s position in both graphs. --- ### Content Details - **Training Reward Values**: - GRPO: ~0.12 → 0.24 - REC-OneSide-NoIS: ~0.12 → 0.23 - RED-Weight: ~0.12 → 0.26 - **KL Divergence Values**: - GRPO: ~10⁻² → 10⁻¹ - REC-OneSide-NoIS: ~10⁻² → 10⁻¹ - RED-Weight: ~10⁻² → 10⁻¹ (with spikes to ~10⁻¹⁵). --- ### Final Notes The graphs highlight a trade-off between reward maximization and distributional stability. RED-Weight’s superior reward comes with higher divergence, while GRPO and REC-OneSide-NoIS offer more balanced performance. The sync interval’s role in these dynamics warrants further investigation, particularly its impact on spike patterns in KL divergence. </details> Figure 6: Empirical results on Guru-Math with Qwen2.5-7B-Instruct. Training reward curves are smoothed with a running-average window of size 3. <details> <summary>x7.png Details</summary>  ### Visual Description ## Line Graphs: Model Performance Metrics Over Training Steps ### Overview The image contains four line graphs arranged in a 2x2 grid, each tracking different performance metrics (Training Reward, KL Divergence, Entropy, Response Length) for three models (GRPO, REC-OneSide-NoIS, RED-Weight) over 400 training steps. All graphs share a synchronization interval of 20. The y-axes vary by metric, with x-axes uniformly labeled "Training Steps". ### Components/Axes 1. **Top-Left Graph**: - **Title**: Training Reward - **Y-Axis**: Training Reward (0.0–0.8) - **X-Axis**: Training Steps (0–400) - **Legend**: GRPO (green), REC-OneSide-NoIS (purple), RED-Weight (orange) 2. **Top-Right Graph**: - **Title**: KL Divergence - **Y-Axis**: KL Divergence (0.0–0.06) - **X-Axis**: Training Steps (0–400) - **Legend**: Same as above 3. **Bottom-Left Graph**: - **Title**: Entropy - **Y-Axis**: Entropy (0–8) - **X-Axis**: Training Steps (0–400) - **Legend**: Same as above 4. **Bottom-Right Graph**: - **Title**: Response Length - **Y-Axis**: Response Length (0–2000) - **X-Axis**: Training Steps (0–400) - **Legend**: Same as above ### Detailed Analysis #### Training Reward (Top-Left) - **GRPO (green)**: Starts at ~0.3, rises steadily to ~0.75 by 400 steps. - **REC-OneSide-NoIS (purple)**: Similar trajectory to GRPO, peaking at ~0.72. - **RED-Weight (orange)**: Slightly lower than the other two, reaching ~0.70. - **Trend**: All models show consistent improvement, with GRPO and REC-OneSide-NoIS outperforming RED-Weight. #### KL Divergence (Top-Right) - **GRPO (green)**: Begins near 0.0, rises to ~0.04 by 400 steps. - **REC-OneSide-NoIS (purple)**: Peaks at ~0.035 around 200 steps, then declines. - **RED-Weight (orange)**: Remains below 0.02 throughout. - **Trend**: RED-Weight maintains the lowest divergence, suggesting better stability. #### Entropy (Bottom-Left) - **GRPO (green)**: Drops from ~7 to ~4 by 400 steps. - **REC-OneSide-NoIS (purple)**: Steep decline from ~7 to ~2. - **RED-Weight (orange)**: Gradual decrease to ~5. - **Trend**: REC-OneSide-NoIS achieves the lowest entropy, indicating more efficient decision-making. #### Response Length (Bottom-Right) - **GRPO (green)**: Decreases from ~2000 to ~1000. - **REC-OneSide-NoIS (purple)**: Sharp drop to ~500. - **RED-Weight (orange)**: Reduces to ~1500. - **Trend**: REC-OneSide-NoIS has the shortest response length, suggesting computational efficiency. ### Key Observations 1. **Performance Trade-offs**: - REC-OneSide-NoIS excels in entropy and response length but lags slightly in training reward. - RED-Weight shows the most stable KL divergence but underperforms in other metrics. - GRPO balances reward and divergence but has moderate entropy and response length. 2. **Anomalies**: - REC-OneSide-NoIS’s entropy and response length drop sharply, suggesting a phase transition in training efficiency. - KL divergence for REC-OneSide-NoIS peaks early, then stabilizes. ### Interpretation The data suggests that **REC-OneSide-NoIS** prioritizes efficiency (low entropy, short response length) at the cost of slightly lower training reward. **GRPO** offers a balanced performance, while **RED-Weight** sacrifices reward for stability (low KL divergence). The synchronization interval of 20 likely mitigates catastrophic forgetting, as all models show consistent improvement. Notably, REC-OneSide-NoIS’s rapid entropy reduction implies a focus on refining decision boundaries early in training. </details> <details> <summary>x8.png Details</summary>  ### Visual Description ## Line Chart: Algorithm Performance on MATH500 Accuracy (sync_interval = 20) ### Overview The chart compares the training performance of three algorithms (GRPO, REC-OneSide-NoIS (0.2), and RED-Weight) on the MATH500 accuracy metric over 400 training steps. The y-axis represents accuracy (0.40–0.50), and the x-axis represents training steps (0–400). All lines converge near 0.50 accuracy by the final step, but exhibit distinct trajectories. ### Components/Axes - **X-axis (Training Steps)**: Labeled "Training Steps," with markers at 0, 200, and 400. - **Y-axis (MATH500 Accuracy)**: Labeled "MATH500 Accuracy," scaled from 0.40 to 0.50 in 0.05 increments. - **Legend**: Located in the bottom-right corner, with three entries: - **GRPO**: Green line. - **REC-OneSide-NoIS (0.2)**: Purple line. - **RED-Weight**: Orange line. - **Title**: "sync_interval = 20" is displayed at the top. ### Detailed Analysis 1. **GRPO (Green Line)**: - Starts at ~0.40 accuracy at 0 steps. - Sharp upward trend to ~0.45 accuracy by 200 steps. - Plateaus at ~0.45 accuracy for the remainder of training (200–400 steps). 2. **REC-OneSide-NoIS (0.2) (Purple Line)**: - Begins at ~0.40 accuracy at 0 steps. - Gradual upward trend, reaching ~0.45 accuracy at 200 steps. - Continues rising to ~0.50 accuracy by 400 steps. 3. **RED-Weight (Orange Line)**: - Starts at ~0.45 accuracy at 0 steps. - Dips to ~0.43 accuracy at 200 steps. - Recovers to ~0.50 accuracy by 400 steps, with minor fluctuations. ### Key Observations - **GRPO** shows the fastest initial improvement but plateaus earlier than the other algorithms. - **REC-OneSide-NoIS (0.2)** demonstrates steady, consistent growth, achieving the highest final accuracy (~0.50). - **RED-Weight** exhibits volatility, with a notable dip at 200 steps before recovering to match the final accuracy of REC-OneSide-NoIS. - All algorithms converge near 0.50 accuracy by 400 steps, but REC-OneSide-NoIS (0.2) maintains the most stable upward trajectory. ### Interpretation The chart suggests that **REC-OneSide-NoIS (0.2)** is the most effective algorithm for this task, as it achieves the highest final accuracy with minimal volatility. **GRPO** performs well initially but stagnates, while **RED-Weight**'s fluctuations indicate potential instability in its training process. The "sync_interval = 20" parameter may influence these dynamics, though its exact role is not explained in the chart. The convergence at ~0.50 accuracy implies that all algorithms reach a similar ceiling, but REC-OneSide-NoIS (0.2) does so more efficiently. </details> Figure 7: Comparison of RED-Weight, REC-OneSide-NoIS, and GRPO on MATH with Llama-3.1-8B-Instruct. Reported metrics for training include reward, KL distance to the initial model, entropy, and response length. We also report evaluation accuracy on the MATH500 subset. ## 5 Related works Off-policy RL for LLMs has been studied from various perspectives. Importance sampling has long been considered one foundational mechanism for off-policy RL; besides PPO and GRPO, recent extensions include GSPO (Zheng et al., 2025) and GMPO (Zhao et al., 2025) that work with sequence-wise probability ratios, CISPO (Chen et al., 2025) that clips probability ratios rather than token updates, decoupled PPO (Fu et al., 2025a) that adapts PPO to asynchronous RL, among others. AsymRE (Arnal et al., 2025) offers an alternative baseline-shift approach (with ad-hoc analysis for discrete bandit settings), while OPMD (Kimi-Team, 2025b) partly overlaps with our analysis up to Eq. (4) before diverging, as discussed earlier in Section 4.2. Contrastive Policy Gradient (Flet-Berliac et al., 2024) overlaps with our analysis up to Eq. (6), but it requires paired responses within the same micro-batch (in order to optimize the pairwise surrogate loss), rendering it less infra-friendly than REINFORCE variants. Other perspectives include learning dynamics of DPO and SFT (Ren and Sutherland, 2025), training offline loss functions with negative gradients on on-policy data (Tajwar et al., 2024), or improving generalization of SFT via probability-aware rescaling (Wu et al., 2025). Another line of research integrates expert data into online RL (Yan et al., 2025; Zhang et al., 2025c; Fu et al., 2025b). Our work contributes complementary perspectives to this growing toolkit for off-policy LLM-RL. ## 6 Limitations and future work While our work offers a new off-policy interpretation for group-relative REINFORCE and shows its broad implications for LLM-RL, several limitations remain. (1) Our current analysis covers single/multi-step RL with response/trajectory-level rewards, and assumes access to multiple rollouts per query. Future work may expand its scope and applicability, e.g., generalizing to settings with step-level rewards or only one rollout per query. (2) Our analysis lacks formal guarantees for policy improvement or convergence. Future work may identify distributional assumptions that yield provable guarantees for REINFORCE variants in off-policy settings. (3) Our experiments focus on settings where training data is generated by older policy versions. Extensions to broader off-policy settings (e.g., advanced experience synthesis or incorporation of expert data) may reveal new insights. Addressing these limitations will further solidify the theoretical foundation and advance principled algorithm design for off-policy LLM-RL. ## References - Achiam et al. [2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 22–31. PMLR, 2017. - An et al. [2025] Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. POLARIS: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https://hkunlp.github.io/blog/2025/Polaris. - Arnal et al. [2025] Charles Arnal, GaĂŤtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and negative rewards. arXiv Preprint arXiv:2506.20520, 2025. - Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. - Chen et al. [2025] Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. MiniMax-M1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. - Cheng et al. [2025] Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Revisiting reinforcement learning for LLM reasoning from a cross-domain perspective. arXiv preprint arXiv:2506.14965, 2025. - Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv, 2021. - Da et al. [2025] Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-RLVR: Training software engineering agents via guidance and environment rewards. arXiv, 2025. - DeepSeek-AI [2025] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv, 2025. - Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv, 2024. - Flet-Berliac et al. [2024] Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, and Matthieu Geist. Contrastive policy gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21353–21370, 2024. - Fragkiadaki [2018] Katerina Fragkiadaki. Natural policy gradients, TRPO, PPO. https://www.andrew.cmu.edu/course/10-703/slides/Lecture_NaturalPolicyGradientsTRPOPPO.pdf, 2018. - Fu et al. [2025a] Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. AReaL: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv, 2025a. - Fu et al. [2025b] Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. SRFT: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv, 2025b. - Gao et al. [2025] Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv, 2025. - Guo et al. [2025] Yongxin Guo, Wenbo Deng, Zhenglin Cheng, and Xiaoying Tang. G 2 RPO-A: Guided group relative policy optimization with adaptive guidance. arXiv, 2025. - Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurIPS, 2021. - Hong et al. [2023a] Zhang-Wei Hong, Pulkit Agrawal, Remi Tachet des Combes, and Romain Laroche. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. In The Eleventh International Conference on Learning Representations, 2023a. - Hong et al. [2023b] Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. - Hu et al. [2024] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405.11143, 2024. - Kakade and Langford [2002] Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002. - Kimi-Team [2025a] Kimi-Team. Kimi-Researcher. https://moonshotai.github.io/Kimi-Researcher, 2025a. - Kimi-Team [2025b] Kimi-Team. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025b. - Korbak et al. [2022] Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed as bayesian inference. In Conference on Empirical Methods in Natural Language Processing, 2022. - Liang et al. [2025] Xiao Liang, Zhong-Zhi Li, Yeyun Gong, Yang Wang, Hengyuan Zhang, Yelong Shen, Ying Nian Wu, and Weizhu Chen. SwS: Self-aware weakness-driven problem synthesis in reinforcement learning for LLM reasoning. arXiv Preprint arXiv:2506.08989, 2025. - Liu et al. [2025a] Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Wang Xinzhi, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. ToolACE: Winning the points of LLM function calling. In The Thirteenth International Conference on Learning Representations, 2025a. - Liu et al. [2025b] Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, Siran Yang, Jiamang Wang, Wenbo Su, and Bo Zheng. Part I: Tricks or traps? a deep dive into RL for LLM reasoning. arXiv preprint arXiv:2508.08221, 2025b. - Nachum et al. [2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In NIPS, 2017. - Noukhovitch et al. [2025] Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous RLHF: Faster and more efficient off-policy RL for language models. In The Thirteenth International Conference on Learning Representations, 2025. - OpenAI [2024] OpenAI. OpenAI o1 system card. arXiv Preprint arXiv:2412.16720, 2024. - Ouyang et al. [2022] Long Ouyang, Pamela Mishkin, Jeff Wu, C L Mar, Jacob Hilton, Amanda Askell, and Paul Christiano. Training language models to follow instructions with human feedback. arXiv, 2022. - Pan et al. [2025] Xuchen Pan, Yanxi Chen, Yushuo Chen, Yuchang Sun, Daoyuan Chen, Wenhao Zhang, Yuexiang Xie, Yilun Huang, Yilei Zhang, Dawei Gao, Weijie Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. Trinity-RFT: A general-purpose and unified framework for reinforcement fine-tuning of large language models. arXiv Preprint arXiv:2505.17826, 2025. - Qwen-Team [2025] Qwen-Team. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. - Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. - Ren and Sutherland [2025] Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations, 2025. - Richemond et al. [2024] Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment. arXiv Preprint arXiv:2405.19107, 2024. - Rolnick et al. [2019] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019. - Schaul et al. [2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv Preprint arXiv:1511.05952, 2016. - Schulman et al. [2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015. - Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. - Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Junxiao Song Runxin Xu, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024. - Sheng et al. [2024] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv, 2024. - Silver and Sutton [2025] David Silver and Richard S. Sutton. Welcome to the era of experience. https://storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf, 2025. - Sutton et al. [1998] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998. - Tajwar et al. [2024] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Forty-first International Conference on Machine Learning, 2024. - Vaswani et al. [2022] Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C. Machado, Pablo Samuel Castro, and Nicolas Le Roux. A functional mirror ascent view of policy gradient methods with function approximation. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 151, Valencia, Spain, 2022. PMLR. - von Werra et al. [2020] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020. - Wang et al. [2025] Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library. arXiv preprint arXiv:2506.06122, 2025. - Williams [1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. - Wu et al. [2025] Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of SFT: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629, 2025. - Yan et al. [2025] Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv Preprint arXiv:2504.14945, 2025. - Yao et al. [2025] Feng Yao, Liyuan Liu an Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient RL framework secretly brings you off-policy RL training. https://fengyao.notion.site/off-policy-rl, 2025. - Yu et al. [2025] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. - Zhang et al. [2025a] Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for LLMs: A survey, 2025a. - Zhang et al. [2025b] Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv:2509.08827, 2025b. - Zhang et al. [2025c] Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy RL meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11408, 2025c. - Zhao et al. [2025] Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, Fang Wan, and Furu Wei. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673, 2025. - Zheng et al. [2025] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. ## Appendix A Extending Section 2.2 to multi-step RL This section extends the off-policy interpretation proposed in Section 2.2 to multi-step RL settings. Let us start by introducing some notations. In multi-step RL, the initial prompt $x$ is also regarded as the initial state $s^{1}=x$ . A rollout trajectory consisting of multiple turns of agent-environment interaction is denoted by $$ \mathcal{T}=(s^{1},a^{1},s^{2},a^{2},\dots)=(s^{\ell},a^{\ell})_{1\leq\ell\leq|\mathcal{T}|}, $$ where $s^{\ell}$ is the state and $a^{\ell}$ is the action, i.e., an LLM response (akin to $y$ in Section 2.2). Let $c^{\ell}$ denote the context up to step $\ell$ , so that $a^{\ell}\sim\pi(\cdot|c^{\ell})$ for some policy $\pi$ . Throughout this section, we consider trajectory-level rewards $r(x,\mathcal{T})$ . Let $\rho_{\bm{\theta}}(\cdot|x)$ denote the trajectory distribution induced by policy $\pi_{\bm{\theta}}$ at initial state $s^{1}=x$ . The following analysis focuses on the $t$ -th iteration, updating the policy model from $\bm{\theta}_{t}$ to $\bm{\theta}_{t+1}$ . #### Step 1: surrogate objective and consistency condition. For the $t$ -th iteration of policy optimization, consider the following KL-regularized objective: $$ \max_{\bm{\theta}}\quad J(\bm{\theta};\pi_{\bm{\theta}_{t}})\coloneqq{\mathbb{E}}_{x\sim D}\bigg[{\mathbb{E}}_{\mathcal{T}\sim\rho_{\bm{\theta}}(\cdot|x)}[r(x,\mathcal{T})]-\tau\cdot D_{\textsf{KL}}\big(\rho_{\bm{\theta}}(\cdot|x)\,\|\,\rho_{\bm{\theta}_{t}}(\cdot|x)\big)\bigg]. \tag{11} $$ The optimal policy $\pi$ and the induced trajectory distribution $\rho$ satisfies the following: for any trajectory $\mathcal{T}$ , $$ \displaystyle\rho(\mathcal{T}|x) \displaystyle=\frac{\rho_{\bm{\theta}_{t}}(\mathcal{T}|x)e^{r(x,\mathcal{T})/\tau}}{Z(x,\rho_{\bm{\theta}_{t}})},\quad\text{where} \displaystyle Z(x,\rho_{\bm{\theta}_{t}}) \displaystyle\coloneqq\int\rho_{\bm{\theta}_{t}}(\mathcal{T}^{\prime}|x)e^{r(x,\mathcal{T}^{\prime})/\tau}\mathop{}\!\mathrm{d}\mathcal{T}^{\prime}={\mathbb{E}}_{\mathcal{T}^{\prime}\sim\rho_{\bm{\theta}_{t}}(\cdot|x)}[e^{r(x,\mathcal{T}^{\prime})/\tau}]. \tag{12} $$ This is equivalent to the following: for any pair of trajectories $\mathcal{T}_{1}$ and $\mathcal{T}_{2}$ , | | $\displaystyle\frac{\rho(\mathcal{T}_{1}|x)}{\rho(\mathcal{T}_{2}|x)}=\frac{\rho_{\bm{\theta}_{t}}(\mathcal{T}_{1}|x)}{\pi_{\bm{\theta}_{t}}(\mathcal{T}_{2}|x)}e^{\big(r(x,\mathcal{T}_{1})-r(x,\mathcal{T}_{2})\big)/\tau}.$ | | | --- | --- | --- | Taking logarithm of both sides and doing some rearrangement, we have equivalently $$ \displaystyle r(x,\mathcal{T}_{1})-\tau\cdot\big(\log\rho(\mathcal{T}_{1}|x)-\log\rho_{\bm{\theta}_{t}}(\mathcal{T}_{1}|x)\big)=r(x,\mathcal{T}_{2})-\tau\cdot\big(\log\rho(\mathcal{T}_{2}|x)-\log\rho_{\bm{\theta}_{t}}(\mathcal{T}_{2}|x)\big). \tag{14} $$ Note that for a trajectory $\mathcal{T}$ , we have $$ \log\rho(\mathcal{T}|x)-\log\rho_{\bm{\theta}_{t}}(\mathcal{T}|x)=\sum_{\ell}\log\pi(a^{\ell}|c^{\ell})-\sum_{\ell}\log\pi_{\bm{\theta}_{t}}(a^{\ell}|c^{\ell}) $$ since the state-transition probability terms in $\log\rho(\mathcal{T}|x)$ and $\log\rho_{\bm{\theta}_{t}}(\mathcal{T}|x)$ cancel out. #### Step 2: surrogate loss with finite samples. Given $K$ trajectories from the same initial state $s_{1}=x$ , we define the following mean-squared surrogate loss that enforces the consistency condition: $$ \displaystyle\widehat{L}({\bm{\theta}};x,\pi_{\bm{\theta}_{t}})\coloneqq\frac{1}{K^{2}}\sum_{1\leq i<j\leq K}\frac{(a_{i}-a_{j})^{2}}{(1+\tau)^{2}}, \displaystyle\text{where}\quad a_{i}\coloneqq r(x,\mathcal{T}_{i})-\tau\Big(\sum_{\ell}\log\pi_{\bm{\theta}}(a^{\ell}_{i}|c^{\ell}_{i})-\sum_{\ell}\log\pi_{\bm{\theta}_{t}}(a^{\ell}_{i}|c^{\ell}_{i})\Big). \tag{15} $$ With infinite samples and sufficient coverage of the action space, the optimum of this surrogate loss would be the same as the optimal policy for the surrogate objective in Eq. (11). #### Step 3: one gradient step of the surrogate loss. By the same trick as in Section 2.2, we have | | $\displaystyle\nabla_{\bm{\theta}}{(a_{i}-a_{j})^{2}}\big|_{\bm{\theta}_{t}}={-2\tau}\,\Big(r(x,\mathcal{T}_{i})-r(x,\mathcal{T}_{j})\Big)\Big(\nabla_{\bm{\theta}}\sum_{\ell}\log\pi_{\bm{\theta}}(a^{\ell}_{i}|c^{\ell}_{i})\big|_{\bm{\theta}_{t}}-\nabla_{\bm{\theta}}\sum_{\ell}\log\pi_{\bm{\theta}}(a^{\ell}_{j}|c^{\ell}_{j})\big|_{\bm{\theta}_{t}}\Big),$ | | | --- | --- | --- | and | | $\displaystyle\nabla_{\bm{\theta}}\sum_{1\leq i<j\leq K}\frac{(a_{i}-a_{j})^{2}}{(1+\tau)^{2}}\Big|_{\bm{\theta}_{t}}=\frac{-2\tau K}{(1+\tau)^{2}}\sum_{1\leq i\leq K}\big(r(x,\mathcal{T}_{i})-\overline{r}(x)\big)\nabla_{\bm{\theta}}\sum_{\ell}\log\pi_{\bm{\theta}}(a^{\ell}_{i}|c^{\ell}_{i})\Big|_{\bm{\theta}_{t}},$ | | | --- | --- | --- | where $\overline{r}(x)\coloneqq\sum_{1\leq j\leq K}r(x,\mathcal{T}_{j})/K$ denotes the group mean reward in the last line. In sum, the gradient of the surrogate loss in Eq. (16) becomes: $$ \nabla_{\bm{\theta}}\widehat{L}(\bm{\theta};x,\pi_{\bm{\theta}_{t}})\big|_{\bm{\theta}_{t}}=\frac{-2\tau}{(1+\tau)^{2}}\cdot\frac{1}{K}\sum_{1\leq i\leq K}\big(r(x,\mathcal{T}_{i})-\overline{r}(x)\big)\,\nabla_{\bm{\theta}}\sum_{\ell}\log\pi_{\bm{\theta}}(a^{\ell}_{i}|c^{\ell}_{i})\Big|_{\bm{\theta}_{t}}. $$ This motivates the following policy update step: $$ g\big(\bm{\theta};x,\{\mathcal{T}_{i},r_{i}\}_{1\leq i\leq K}\big)=\frac{2\tau}{(1+\tau)^{2}}\cdot\frac{1}{K}\sum_{1\leq i\leq K}\big(r(x,\mathcal{T}_{i})-\overline{r}(x)\big)\,\nabla_{\bm{\theta}}\sum_{1\leq\ell\leq|\mathcal{T}_{i}|}\log\pi_{\bm{\theta}}(a^{\ell}_{i}|c^{\ell}_{i}), \tag{17} $$ which concludes our derivation of group-relative REINFORCE in multi-step RL settings. ## Appendix B Implementation details and additional experiments We implement all algorithms with the Trinity-RFT framework [Pan et al., 2025], and run experiments on NVIDIA L20, H20, and A800 GPUs. See Tables 1 and 2 for detailed configurations of our experiments. ### B.1 Dataset details We provide additional descriptions of the datasets used in our experiments: - GSM8k [Cobbe et al., 2021] is a widely used benchmark with 8.5k grade-school math word problems, designed to test arithmetic reasoning and step-by-step problem solving. - MATH [Hendrycks et al., 2021] covers algebra, geometry, probability, and number theory, containing 12.5k examples in total (7.5k for training and 5k for testing); it demands advanced symbolic reasoning beyond GSM8k. - Guru [Cheng et al., 2025] is a multi-domain reasoning dataset with 91.9k examples spanning math, code, science, logic, simulation, and tabular tasks; we use its math subset (around 54k samples), which introduces diverse problem formats for evaluating transfer of reasoning strategies. - ToolACE [Liu et al., 2025a] is a multilingual benchmark with around 11k synthetic samples designed to evaluate LLMs’ ability to solve tasks by selecting and invoking external tools via strict JSON-formatted function calls; we use a 5k single-turn subset in our experiments. ### B.2 Understanding the synchronization parameters We parameterize rollout-training scheduling by two configuration parameters in Trinity-RFT: the synchronization interval (sync_interval) and synchronization offset (sync_offset). Their meanings are visualized in Figure 2 and explained in the following. The parameter sync_interval specifies the number of generated rollout batches (which equals the number of gradient steps for training the policy model) between two consecutive executions of model weight synchronization. When sync_interval $=1$ , the rollout and policy models synchronize after each gradient step with one batch of samples, yielding a strictly on-policy process (if we ignore the issue of precision mismatch between rollout and training engines [Yao et al., 2025]). When sync_interval $>1$ , sync_interval rollout batches are generated with stale model weights before synchronization, which accelerates the overall RL process through pipeline parallelism but incurs off-policyness. The parameter sync_offset specifies the lag between the generation and consumption of each batch. More specifically, sync_offset batches are generated and saved to the buffer before training is launched, which is also useful for reducing pipeline bubbles and improving hardware utilization [Noukhovitch et al., 2025]. In some of our experiments, we deliberately set sync_offset to a large value, in order to simulate a scenario where reward signals from the environment are lagged. In general, with $(\texttt{sync\_interval},\texttt{sync\_offset})=(m,n)$ , the off-policyness of a consumed batch with zero-index id $l$ corresponds to its temporal distance from the most recent synchronized policy is $(l\bmod m)+n$ . For example, $(4,0)$ yields off-policyness ${0,1,2,3}$ within each interval, while $(1,4)$ yields a constant off-policyness of $4$ . Table 1: Default hyperparameters. Deviations from defaults are noted in figure captions. | | GSM8K (Qwen2.5-1.5B) | ToolACE (Llama-3.2-3B) | Guru (Qwen2.5-7B) | MATH (Llama-3.1-8B) | | --- | --- | --- | --- | --- | | Learning rate | $1\!\times\!10^{-6}$ | $1\!\times\!10^{-6}$ | $1\!\times\!10^{-6}$ | $5\!\times\!10^{-7}$ | | Batch size | 96 | 96 | 64 | 64 | | $K$ | 8 | 8 | 16 | 16 | | Weight decay | 0.01 | 0.01 | 0.1 | 0.1 | | Warmup steps | 0 | 0 | 80 | 40 | | Eval temperature | 1.0 | N/A | N/A | 0.6 | | Eval top-p | 1.0 | N/A | N/A | 1.0 | | Figures | 3, 5, 9, 10, 11 | 4 | 6 | 7 | Table 2: Other shared hyperparameters across all experiments. | Parameter | Value | | --- | --- | | Optimizer | AdamW | | $(\beta_{1},\beta_{2})$ | (0.9, 0.999) | | Gradient clipping | 1.0 | | Warmup style | constant | | Weight-decay increment style | constant | | Auxiliary LR decay style | exponential | | Training inference temperature | 1.0 | | Training inference top-p | 1.0 | ### B.3 REC with different clipping mechanisms In addition to one-side clipping investigated in Section 4, here we compare additional clipping mechanisms for the REC series, to understand how the geometry of clipping — asymmetric vs. symmetric bounds and the presence of a zero-gradient band — affects the learning process. #### REC-TwoSide-IS / NoIS . We replace the mask $M_{i}^{t}$ in REC-OneSide-IS / NoIS in Eq. (8) with a two-side mask It turns out that REC-TwoSide-NoIS resembles the sPPO algorithm proposed by Vaswani et al. [2022], though derived with different rationales. : $$ \widetilde{M}_{i}^{t}=\mathbbm{1}\Big(1-\epsilon_{\textsf{low}}\leq\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\leq 1+\epsilon_{\textsf{high}}\Big). \tag{18} $$ Two-side clipping imposes weaker regularization than one-side clipping does with the same clipping parameter $(\epsilon_{\textsf{low}},\epsilon_{\textsf{high}})$ . This can potentially improve training efficiency, but might also be risky when the probability ratio $\pi_{\bm{\theta}}/\pi_{\textsf{old}}$ goes far off. To compensate for this, we design REC-Ring. #### REC-Ring . In addition to the inner band $(1-\epsilon_{\textsf{low}},1+\epsilon_{\textsf{high}})$ as in Eq. (18), we further specify outer safety margins $\epsilon_{\textsf{low}}^{\prime}\!\geq\!\epsilon_{\textsf{low}}$ and $\epsilon_{\textsf{high}}^{\prime}\!\geq\!\epsilon_{\textsf{high}}$ . The REC-Ring mask is: $$ \displaystyle\widehat{M}_{i}^{t} \displaystyle=\mathbbm{1}\Big(1-\epsilon_{\textsf{low}}\leq\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\leq 1+\epsilon_{\textsf{high}}\Big) \displaystyle\qquad+\mathbbm{1}\Big(A_{i}>0\text{ and }\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\leq 1-\epsilon_{\textsf{low}}^{\prime}\Big) \displaystyle\qquad+\mathbbm{1}\Big(A_{i}<0\text{ and }\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}\geq 1+\epsilon_{\textsf{high}}^{\prime}\Big). \tag{19} $$ A comparison of the clipping mechanisms are visualized in Figure 8. Note that REC-OneSide and REC-TwoSide can be regarded as special cases of REC-Ring. <details> <summary>x9.png Details</summary>  ### Visual Description ## Diagram Type: Control System Behavior Schematics ### Overview The image contains three technical diagrams illustrating control system behavior under different conditions. Each diagram uses directional arrows labeled with "A > 0" (positive effect) and "A < 0" (negative effect) to represent system responses across normalized parameter ranges. The diagrams are labeled GRPO/REC-OneSide, REC-TwoSide, and REC-Ring, with distinct configurations of dashed boundary lines and arrow placements. ### Components/Axes 1. **X-Axis**: - Labeled as `πθ/πold` (normalized parameter ratio). - GRPO/REC-OneSide: Ranges from `1 - εlow` to `1 + εhigh`. - REC-TwoSide: Ranges from `1 - εlow` to `1 + εhigh`. - REC-Ring: Ranges from `1 - ε'_low` to `1 + ε'_high`, with intermediate markers at `1 - εlow` and `1 + εhigh`. 2. **Y-Axis**: - Not explicitly labeled; diagrams appear to represent 1D parameter space. 3. **Legends**: - Purple arrows indicate: - `A > 0` (rightward arrows, positive effect). - `A < 0` (leftward arrows, negative effect). 4. **Dashed Lines**: - Vertical boundaries marking critical thresholds (e.g., `1 ± εlow`, `1 ± εhigh`). ### Detailed Analysis #### GRPO/REC-OneSide - **Structure**: Two vertical dashed lines at `1 - εlow` and `1 + εhigh`. - **Arrows**: - Left of `1 - εlow`: `A < 0` (leftward). - Between `1 - εlow` and `1 + εhigh`: `A > 0` (rightward). - Right of `1 + εhigh`: `A < 0` (leftward). #### REC-TwoSide - **Structure**: Single vertical dashed line at `1 - εlow`. - **Arrows**: - Left of `1 - εlow`: `A < 0` (leftward). - Right of `1 - εlow`: `A > 0` (rightward). #### REC-Ring - **Structure**: Two vertical dashed lines at `1 - εlow` and `1 + εhigh`, with additional markers at `1 - ε'_low` and `1 + ε'_high`. - **Arrows**: - Left of `1 - ε'_low`: `A > 0` (rightward). - Between `1 - εlow` and `1 + εhigh`: `A > 0` (rightward) and `A < 0` (leftward). - Right of `1 + ε'_high`: `A < 0` (leftward). ### Key Observations 1. **Boundary Sensitivity**: - GRPO/REC-OneSide shows bidirectional instability outside `1 ± εlow/εhigh`. - REC-TwoSide simplifies to a single threshold (`1 - εlow`), with uniform behavior on either side. - REC-Ring introduces asymmetric thresholds (`ε'_low`/`ε'_high`), creating a "ring" of mixed effects between `1 - εlow` and `1 + εhigh`. 2. **Arrow Consistency**: - All diagrams use purple arrows matching the legend. - Directionality aligns with parameter magnitude (e.g., `A > 0` dominates in central regions). 3. **Threshold Variability**: - REC-Ring distinguishes between primed (`ε'_low/ε'_high`) and unprimed (`εlow/εhigh`) thresholds, suggesting adaptive or hierarchical control logic. ### Interpretation These diagrams likely model system stability or performance in control theory, where: - `A` represents a control parameter (e.g., gain, error). - `πθ/πold` normalizes the parameter relative to a baseline (`πold`). - `ε` terms define allowable deviation ranges for stability. - **GRPO/REC-OneSide**: Represents a system where deviations beyond `1 ± εlow/εhigh` destabilize the parameter (`A < 0`), while central deviations stabilize it (`A > 0`). - **REC-TwoSide**: Simplifies to a single critical threshold (`1 - εlow`), with opposing effects on either side. - **REC-Ring**: Introduces a "safe zone" between `1 - εlow` and `1 + εhigh` where bidirectional effects (`A > 0` and `A < 0`) coexist, possibly indicating nonlinear interactions. The primed thresholds (`ε'_low/ε'_high`) may reflect adaptive adjustments in dynamic environments. The schematics emphasize how parameter normalization and threshold design influence system behavior, with REC-Ring suggesting a more complex, context-dependent control strategy. </details> Figure 8: A visualization of activated gradient for various REC algorithms. Here, $A$ represents the advantage of a specific token, and an arrow pointing to the right and annotated with “ $A>0$ ” means there is activated gradient that incentivizes increasing $\pi_{\bm{\theta}}$ when the token advantage is positive and the probability ratio $\pi_{\bm{\theta}}/\pi_{\textsf{old}}$ lies in the corresponding interval. #### Experiments. We compare the following algorithms: REINFORCE, GRPO, REC-TwoSide-IS, REC-TwoSide-NoIS, and REC-Ring-NoIS. Clipping parameters are set to $(\epsilon_{\text{low}},\epsilon_{\text{high}})=(0.2,0.2)$ , and for REC-Ring we additionally set $(\epsilon_{\textsf{low}}^{\prime},\epsilon_{\textsf{high}}^{\prime})=(0.6,2.0)$ . <details> <summary>x10.png Details</summary>  ### Visual Description ## Line Graphs: Performance Metrics Across Training Steps ### Overview The image contains three columns of line graphs comparing performance metrics across three training methods (REINFORCE, GRPO, REC-TwoSides, REC-RingNoIS) under different synchronization settings ("sync interval = 20", "sync offset = 10", "offline"). Each column contains four subplots: Evaluation Accuracy, Training Reward, KL Divergence, and Clipping Fraction. The x-axis represents training steps (0–150), while y-axes vary by subplot. --- ### Components/Axes - **Columns**: - Left: "sync interval = 20" (blue dashed line) - Middle: "sync offset = 10" (orange dashed line) - Right: "offline" (no dashed line) - **Subplots**: 1. **Evaluation Accuracy**: Y-axis = 0.0–1.0 2. **Training Reward**: Y-axis = 0.0–1.0 3. **KL Divergence**: Y-axis = 10–100 4. **Clipping Fraction**: Y-axis = 10–100 - **Legends**: Positioned at the bottom of each column, with colors: - Blue: REINFORCE - Green: GRPO - Orange: REC-TwoSides - Purple: REC-RingNoIS - Dashed Orange: REC-TwoSides (0.2) & (0.6, 2.0) --- ### Detailed Analysis #### **Column 1: sync interval = 20** 1. **Evaluation Accuracy**: - Blue (REINFORCE): Starts at ~0.6, dips to ~0.4 at 50 steps, then stabilizes. - Green (GRPO): Flat at ~0.6. - Orange (REC-TwoSides): Peaks at ~0.8 at 100 steps, then declines. - Purple (REC-RingNoIS): Smooth increase from ~0.4 to ~0.8. - Dashed Orange: Peaks at ~0.7 at 100 steps. 2. **Training Reward**: - Blue: Drops sharply to ~0.2 at 50 steps, then stabilizes. - Green: Flat at ~0.6. - Orange: Peaks at ~0.8 at 100 steps. - Purple: Stable at ~0.7. - Dashed Orange: Peaks at ~0.7 at 100 steps. 3. **KL Divergence**: - Blue: Fluctuates between 10–20. - Green: Stable at ~10. - Orange: Spikes to ~30 at 100 steps. - Purple: Stable at ~10. - Dashed Orange: Spikes to ~30 at 100 steps. 4. **Clipping Fraction**: - Blue: Fluctuates between 10–20. - Green: Stable at ~10. - Orange: Increases to ~20 at 100 steps. - Purple: Stable at ~10. - Dashed Orange: Increases to ~20 at 100 steps. #### **Column 2: sync offset = 10** 1. **Evaluation Accuracy**: - Blue: Starts at ~0.5, dips to ~0.3 at 50 steps, then stabilizes. - Green: Flat at ~0.6. - Orange: Peaks at ~0.7 at 100 steps. - Purple: Smooth increase from ~0.5 to ~0.8. - Dashed Orange: Peaks at ~0.6 at 100 steps. 2. **Training Reward**: - Blue: Drops to ~0.3 at 50 steps, then stabilizes. - Green: Flat at ~0.6. - Orange: Peaks at ~0.7 at 100 steps. - Purple: Stable at ~0.7. - Dashed Orange: Peaks at ~0.6 at 100 steps. 3. **KL Divergence**: - Blue: Fluctuates between 10–20. - Green: Stable at ~10. - Orange: Spikes to ~25 at 100 steps. - Purple: Stable at ~10. - Dashed Orange: Spikes to ~25 at 100 steps. 4. **Clipping Fraction**: - Blue: Fluctuates between 10–20. - Green: Stable at ~10. - Orange: Increases to ~15 at 100 steps. - Purple: Stable at ~10. - Dashed Orange: Increases to ~15 at 100 steps. #### **Column 3: offline** 1. **Evaluation Accuracy**: - Blue: Starts at ~0.4, dips to ~0.2 at 50 steps, then stabilizes. - Green: Flat at ~0.6. - Orange: Peaks at ~0.7 at 100 steps. - Purple: Smooth increase from ~0.5 to ~0.8. - Dashed Orange: Peaks at ~0.6 at 100 steps. 2. **Training Reward**: - Blue: Drops to ~0.2 at 50 steps, then stabilizes. - Green: Flat at ~0.6. - Orange: Peaks at ~0.7 at 100 steps. - Purple: Stable at ~0.7. - Dashed Orange: Peaks at ~0.6 at 100 steps. 3. **KL Divergence**: - Blue: Fluctuates between 10–20. - Green: Stable at ~10. - Orange: Spikes to ~20 at 100 steps. - Purple: Stable at ~10. - Dashed Orange: Spikes to ~20 at 100 steps. 4. **Clipping Fraction**: - Blue: Fluctuates between 10–20. - Green: Stable at ~10. - Orange: Increases to ~10 at 100 steps. - Purple: Stable at ~10. - Dashed Orange: Increases to ~10 at 100 steps. --- ### Key Observations 1. **REC-RingNoIS (Purple)** consistently shows the highest Evaluation Accuracy and Training Reward across all settings, with smooth trends. 2. **REC-TwoSides (Orange)** exhibits the highest KL Divergence and Clipping Fraction, indicating greater exploration but potential instability. 3. **REINFORCE (Blue)** performs poorly in Training Reward and Evaluation Accuracy, with erratic trends. 4. **GRPO (Green)** maintains stable metrics across all settings, suggesting robustness. 5. The dashed orange line (REC-TwoSides variants) shows intermediate performance, combining traits of REC-TwoSides and REC-RingNoIS. --- ### Interpretation The data suggests that synchronization settings significantly impact performance: - **Sync Interval = 20** and **Sync Offset = 10** allow REC-RingNoIS to outperform others, likely due to balanced exploration/exploitation. - **Offline** settings degrade REINFORCE's performance, highlighting its reliance on synchronization. - REC-TwoSides variants (dashed orange) show trade-offs: higher KL Divergence/Clipping Fraction (exploration) but lower Evaluation Accuracy/Training Reward (stability). - GRPO's stability across all settings implies it is less sensitive to synchronization parameters, making it a reliable baseline. Notably, REC-RingNoIS achieves the best balance of high Evaluation Accuracy and low KL Divergence, suggesting it optimizes policy updates effectively. The dashed orange line's intermediate performance indicates that combining REC-TwoSides variants may mitigate some instability while retaining exploration benefits. </details> Figure 9: Comparison of REC variants on GSM8K with Qwen2.5-1.5B-Instruct under different off-policy settings. Evaluation accuracy, training reward, KL divergence (with respect to the initial model) and clipping fraction are reported. Training reward curves are smoothed with a running-average window of size 3. Figure 9 presents the empirical results. We observe that for REC-TwoSide, importance sampling is non-essential in all three settings, akin to the case of REC-OneSide. In addition, REC-TwoSide methods demonstrate fast policy improvement at the beginning but tend to collapse later on, whereas REC-Ring achieves a better balance of convergence speed and stability. ### B.4 Ablation: the impact of learning rates Recall that in Section 4.1, we have demonstrated empirically the advantages of enlarging the clipping parameters $\epsilon_{\textsf{low}},\epsilon_{\textsf{high}}$ for REC-OneSide-NoIS. One might wonder if the relatively weak performance of GRPO or REC-OneSide with conventional $\epsilon_{\textsf{low}}=\epsilon_{\textsf{high}}=0.2$ is genuinely rooted in the clipping mechanism itself, or simply due to the choice of a small learning rate. To answer this, we enhance the experiment of Figure 3 by sweeping learning rates over $\{1\!\times\!10^{-5},2\!\times\!10^{-6},5\!\times\!10^{-6}\}$ . The results are illustrated in Figure 10, which confirm that simply increasing the learning rate cannot bridge the performance gap between GRPO with $\epsilon_{\textsf{low}}=\epsilon_{\textsf{high}}=0.2$ and REC-OneSide-NoIS with $\epsilon_{\textsf{low}}=0.6,\epsilon_{\textsf{high}}=2.0$ . This shows that relaxing the clipping range acts as a genuine improvement of regularization, rather than merely mimicking a larger learning rate. <details> <summary>x11.png Details</summary>  ### Visual Description ## Line Graphs: Evaluation Accuracy and Training Reward with sync_interval = 20 ### Overview Two line graphs are presented side-by-side, comparing the performance of different reinforcement learning algorithms over training steps. The left graph shows **Evaluation Accuracy**, and the right graph shows **Training Reward**. Both graphs use a sync interval of 20 and include multiple data series with distinct line styles and colors. --- ### Components/Axes #### Left Graph (Evaluation Accuracy) - **X-axis**: Training Steps (0 to 160, increments of 20) - **Y-axis**: Evaluation Accuracy (0.0 to 0.8, increments of 0.2) - **Legend**: Located at the bottom, mapping colors to algorithms: - **Green (solid)**: GRPO (0.2) (lr = 1e-5) - **Green (dashed)**: GRPO (0.2) (lr = 2e-6) - **Green (dotted)**: GRPO (0.2) (lr = 5e-6) - **Purple (solid)**: REC-OneSide-NoIS (0.2) (lr = 1e-5) - **Purple (dashed)**: REC-OneSide-NoIS (0.2) (lr = 2e-6) - **Purple (dotted)**: REC-OneSide-NoIS (0.2) (lr = 5e-6) - **Blue (dashed)**: REINFORCE (lr = 2e-6) #### Right Graph (Training Reward) - **X-axis**: Training Steps (0 to 160, increments of 20) - **Y-axis**: Training Reward (0.0 to 1.0, increments of 0.2) - **Legend**: Same as the left graph, with identical color-to-algorithm mappings. --- ### Detailed Analysis #### Left Graph (Evaluation Accuracy) 1. **REC-OneSide-NoIS (solid purple, lr = 1e-5)**: - Starts at ~0.35, steadily increases to ~0.75 by step 160. - Smooth upward trend with minimal fluctuation. 2. **GRPO (solid green, lr = 1e-5)**: - Begins at ~0.3, rises to ~0.65 by step 160. - Slightly less steep than REC-OneSide-NoIS. 3. **REINFORCE (blue dashed, lr = 2e-6)**: - Starts at ~0.3, drops sharply to ~0.0 at step 40. - Recovers to ~0.2 by step 160 but remains unstable. 4. **GRPO (dashed green, lr = 2e-6)**: - Starts at ~0.35, plateaus at ~0.55 by step 160. - Less volatile than REINFORCE but slower growth. 5. **REC-OneSide-NoIS (dashed purple, lr = 2e-6)**: - Similar to solid purple but slightly lower (~0.7 vs. 0.75). 6. **GRPO (dotted green, lr = 5e-6)**: - Starts at ~0.3, peaks at ~0.6, then drops to ~0.4 by step 160. - Most volatile among GRPO variants. #### Right Graph (Training Reward) 1. **REC-OneSide-NoIS (solid purple, lr = 1e-6)**: - Starts at ~0.4, rises to ~0.95 by step 160. - Consistent upward trend with minor fluctuations. 2. **REC-OneSide-NoIS (dashed purple, lr = 2e-6)**: - Similar to solid purple but slightly lower (~0.9 vs. 0.95). 3. **REINFORCE (blue dashed, lr = 2e-6)**: - Starts at ~0.4, drops to ~0.0 at step 40. - Recovers to ~0.6 by step 160 but with sharp dips. 4. **GRPO (solid green, lr = 1e-5)**: - Starts at ~0.4, rises to ~0.85 by step 160. - Smooth growth with minor oscillations. 5. **GRPO (dashed green, lr = 2e-6)**: - Starts at ~0.45, plateaus at ~0.75 by step 160. 6. **GRPO (dotted green, lr = 5e-6)**: - Starts at ~0.4, peaks at ~0.8, then drops to ~0.6 by step 160. --- ### Key Observations 1. **REC-OneSide-NoIS (lr = 1e-6)** consistently outperforms other algorithms in both metrics, achieving the highest evaluation accuracy (~0.75) and training reward (~0.95). 2. **REINFORCE** exhibits significant instability, with sharp drops in both graphs (e.g., to ~0.0 at step 40 in Evaluation Accuracy). 3. **GRPO** variants show mixed performance: - Lower learning rates (1e-5) yield better results than higher rates (5e-6). - Dotted GRPO (lr = 5e-6) underperforms in Evaluation Accuracy but matches REC-OneSide-NoIS in Training Reward. 4. **Sync Interval = 20** appears to stabilize training for REC-OneSide-NoIS but does not mitigate REINFORCE's volatility. --- ### Interpretation The data suggests that **REC-OneSide-NoIS with a learning rate of 1e-6** is the most robust algorithm for this task, balancing high evaluation accuracy and stable training rewards. **REINFORCE** struggles with convergence, likely due to its high learning rate (2e-6), which causes overshooting during training. **GRPO** performs well with lower learning rates but becomes unstable at higher rates. The sync interval of 20 may help synchronize updates for REC-OneSide-NoIS, reducing noise in its training process. These results highlight the importance of tuning learning rates and sync intervals for algorithm stability and performance. </details> Figure 10: Comparison of GRPO and REC-OneSide-NoIS on GSM8K with Qwen2.5-1.5B-Instruct. Evaluation accuracy (left) and training reward (right) are reported for varying learning rates. ### B.5 Experiments for OPMD and AsymRE Figure 11 presents empirical results for OPMD and AsymRE in various off-policy settings. It is worth noting that, while the analysis and experiments in their original papers [Kimi-Team, 2025b, Arnal et al., 2025] focus on a setting that is effectively the same as our $\texttt{sync\_interval}>1$ setting, our analysis and experiments have also validated their efficacy in $\texttt{sync\_offset}>1$ scenarios. <details> <summary>x12.png Details</summary>  ### Visual Description ## Line Graphs: Evaluation Accuracy and Training Reward Across Training Steps ### Overview The image contains six line graphs arranged in two rows (three graphs per row) comparing the performance of four reinforcement learning methods: **REINFORCE**, **REC-OneSide-NoIS (0.6, 2.0)**, **OPMD**, and **AsymRE (-0.1)**. The graphs are categorized by three scenarios: 1. **sync_interval = 20** (top-left) 2. **sync_offset = 10** (top-right) 3. **offline** (bottom row, shared across all methods) Each graph tracks two metrics: - **Evaluation Accuracy** (top subplot) - **Training Reward** (bottom subplot) ### Components/Axes - **X-axis**: Training Steps (0 to 150, increments of 50) - **Y-axis (Top)**: Evaluation Accuracy (0 to 1.0) - **Y-axis (Bottom)**: Training Reward (0 to 1.0) - **Legend**: Located at the bottom of all graphs, with colors: - **Blue**: REINFORCE - **Purple**: REC-OneSide-NoIS (0.6, 2.0) - **Green**: OPMD - **Light Green**: AsymRE (-0.1) ### Detailed Analysis #### **sync_interval = 20** (Top-Left) - **Evaluation Accuracy**: - **REINFORCE (Blue)**: Starts at ~0.3, dips to ~0.0 at 50 steps, then rises to ~0.6 by 150 steps. - **REC-OneSide-NoIS (Purple)**: Steady increase from ~0.4 to ~0.8. - **OPMD (Green)**: Gradual rise from ~0.3 to ~0.6. - **AsymRE (Light Green)**: Fluctuates between ~0.4 and ~0.7, ending at ~0.7. - **Training Reward**: - **REINFORCE (Blue)**: Peaks at ~0.8 at 50 steps, then drops to ~0.0 by 150 steps. - **REC-OneSide-NoIS (Purple)**: Stable increase from ~0.5 to ~0.9. - **OPMD (Green)**: Gradual rise from ~0.4 to ~0.7. - **AsymRE (Light Green)**: Stable ~0.6–0.7. #### **sync_offset = 10** (Top-Right) - **Evaluation Accuracy**: - **REINFORCE (Blue)**: Starts at ~0.3, dips to ~0.1 at 50 steps, then rises to ~0.6. - **REC-OneSide-NoIS (Purple)**: Steady increase from ~0.4 to ~0.8. - **OPMD (Green)**: Gradual rise from ~0.3 to ~0.6. - **AsymRE (Light Green)**: Fluctuates between ~0.4 and ~0.7, ending at ~0.7. - **Training Reward**: - **REINFORCE (Blue)**: Peaks at ~0.8 at 50 steps, then drops to ~0.2 by 150 steps. - **REC-OneSide-NoIS (Purple)**: Stable increase from ~0.5 to ~0.9. - **OPMD (Green)**: Gradual rise from ~0.4 to ~0.7. - **AsymRE (Light Green)**: Stable ~0.6–0.7. #### **offline** (Bottom Row, Shared Across All Methods) - **Evaluation Accuracy**: - **REINFORCE (Blue)**: Starts at ~0.3, dips to ~0.1 at 50 steps, then rises to ~0.4. - **REC-OneSide-NoIS (Purple)**: Stable ~0.5–0.6. - **OPMD (Green)**: Stable ~0.4–0.5. - **AsymRE (Light Green)**: Stable ~0.5–0.6. - **Training Reward**: - **REINFORCE (Blue)**: Peaks at ~0.8 at 50 steps, then drops to ~0.2 by 150 steps. - **REC-OneSide-NoIS (Purple)**: Stable ~0.6–0.7. - **OPMD (Green)**: Stable ~0.5–0.6. - **AsymRE (Light Green)**: Stable ~0.6–0.7. ### Key Observations 1. **REC-OneSide-NoIS (Purple)** consistently achieves the highest **Evaluation Accuracy** across all scenarios, with minimal fluctuations. 2. **REINFORCE (Blue)** shows significant volatility, with sharp drops in **Training Reward** (e.g., ~0.8 → 0.0 in sync_interval=20). 3. **AsymRE (-0.1)** (Light Green) maintains moderate performance but exhibits instability in **Training Reward** (e.g., dips to ~0.3 in sync_interval=20). 4. **OPMD (Green)** demonstrates steady but suboptimal performance compared to REC-OneSide-NoIS. 5. In the **offline** scenario, all methods underperform compared to synchronized settings, with **REINFORCE** showing the largest drop in **Training Reward**. ### Interpretation - **REC-OneSide-NoIS (0.6, 2.0)** emerges as the most robust method, maintaining high **Evaluation Accuracy** and **Training Reward** across all scenarios. Its stability suggests superior adaptability to synchronization parameters. - **REINFORCE**'s volatility indicates sensitivity to hyperparameters or environment dynamics, particularly in synchronized settings. The sharp drop in **Training Reward** at 50 steps (e.g., sync_interval=20) may reflect catastrophic forgetting or optimization instability. - **AsymRE (-0.1)**'s fluctuating performance suggests potential trade-offs between exploration and exploitation, with inconsistent rewards despite moderate accuracy. - The **offline** scenario highlights the importance of synchronization: all methods underperform, but **REINFORCE** is disproportionately affected, possibly due to reliance on external feedback loops. This analysis underscores the critical role of synchronization strategies in reinforcement learning, with REC-OneSide-NoIS offering the most reliable performance. </details> Figure 11: Empirical results for OPMD and AsymRE (cf. Section 4.2) on GSM8K with Qwen2.5-1.5B-Instruct under various off-policy settings. The regularization coefficient for OPMD and the baseline shift for AsymRE are both $0.1$ . Training reward curves are smoothed with a running-average window of size 3. ## Appendix C Summary: a unified view of various algorithms For convenient reference, Table 3 summarizes the algorithms investigated in Section 4. Table 3: A summary of algorithms investigated in Section 4. | Augmentation | Algorithm | Gradient / Loss | | --- | --- | --- | | Regularize by clipping | GRPO | $\bm{g}=\frac{1}{K}\sum_{i}\sum_{t}\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})\cdot A_{i}\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}M_{i}^{t}$ | | REC-OneSide-IS | $\bm{g}=\frac{1}{K}\sum_{i}\sum_{t}\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})\cdot(r_{i}-\overline{r})\frac{\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}{\pi_{\textsf{old}}(y_{i}^{t}\,|\,x,y_{i}^{<t})}M_{i}^{t}$ | | | REC-OneSide-NoIS | $\bm{g}=\frac{1}{K}\sum_{i}\sum_{t}\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}^{t}\,|\,x,y_{i}^{<t})\cdot(r_{i}-\overline{r})M_{i}^{t}$ | | | Add regularization loss | OPMD | $\widehat{L}=-\tfrac{1}{K}\!\sum_{i}(r_{i}-\overline{r})\log\pi_{\bm{\theta}}(y_{i}|x)+\tfrac{\tau}{2K}\!\sum_{i}(\log\pi_{\bm{\theta}}(y_{i}|x)-\log\pi_{\textsf{old}}(y_{i}|x))^{2}$ | | AsymRE | $\widehat{L}=-\tfrac{1}{K}\!\sum_{i}(r_{i}-\overline{r})\log\pi_{\bm{\theta}}(y_{i}|x)-\tfrac{\tau}{K}\!\sum_{i}\log\pi_{\bm{\theta}}(y_{i}|x)$ | | | Reweight data | Pairwise-weighted REINFORCE | $\bm{g}=\tfrac{1}{K}\!\sum_{i}\Big(\sum_{j}w_{i,j}\Big)\Big(r_{i}-\tfrac{\sum_{j}w_{i,j}r_{j}}{\sum_{j}w_{i,j}}\Big)\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}|x)$ | | RED-Drop | $\bm{g}=\tfrac{1}{|S|}\!\sum_{i\in S}(r_{i}-\overline{r}_{S})\,\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}|x)$ | | | RED-Weight | $\bm{g}=\sum_{i}w_{i}(r_{i}-\overline{r})\,\nabla_{\bm{\theta}}\log\pi_{\bm{\theta}}(y_{i}|x),\;\;w_{i}=\exp(A_{i}/\tau)$ | |