# GRPO is Secretly a Process Reward Model
**Authors**:
- Michael Sullivan (Department of Language Science and Technology)
- Saarland Informatics Campus (Saarland University, Saarbrücken, Germany)
## Abstract
We prove theoretically that the GRPO RL algorithm induces a non-trivial process reward model (PRM), under certain assumptions regarding within-group overlap of token sequences across completions. We then show empirically that these assumptions are met under real-world conditions: GRPO does in fact induce a non-trivial PRM. Leveraging the framework of GRPO-as-a-PRM, we identify a flaw in the GRPO objective: non-uniformly distributed process steps hinder both exploration and exploitation (under different conditions). We propose a simple modification to the algorithm to mitigate this defect ( $λ$ -GRPO), and show that LLMs trained with $λ$ -GRPO achieve higher validation accuracy and performance on downstream reasoning tasks—and reach peak performance more rapidly—than LLMs trained with standard GRPO. Our results call into question the advantage of costly, explicitly-defined PRMs for GRPO: we show that it is possible to instead leverage the hidden, built-in PRM structure within the vanilla GRPO algorithm to boost model performance with a negligible impact on training time and cost.
## 1 Introduction
Process reward models (PRMs)—models that assign reward to intermediate steps (see Section 2.2)—allow for finer-grained reward assignment than outcome-level signals, thereby yielding improved multi-step reasoning performance (Lightman et al., 2024). PRMs are therefore particularly applicable to RL training for step-by-step processes such as mathematical reasoning.
However, training neural PRMs requires costly, step-level human annotation (Zhang et al., 2025), and such models are particularly susceptible to reward hacking (Cui et al., 2025). These shortcomings have resulted in the limited adoption of learned PRMs for RL training Setlur et al. (2025), leading to the development of Monte-Carlo-based and other heuristic, non-neural PRMs (e.g. Wang et al., 2024; Kazemnejad et al., 2025; Hou et al., 2025).
PRMs are typically employed with RL algorithms such as Proximal Policy Optimization (PPO; Schulman et al., 2017) that employ a critic model and/or generalized advantage estimation (GAE). Although Group Relative Policy Optimization (GRPO; Shao et al., 2024) greatly simplifies and reduces the memory consumption of RL training, leading to its adoption for a wide range of applications—e.g. tool use (Qian et al., 2025; Sullivan et al., 2025), RLHF (Yang et al., 2025b), and, in particular, mathematical reasoning (Shao et al., 2024; DeepSeek-AI, 2025) —it does so by eliminating the critic model and GAE of PPO (see Section 2.1). GRPO has therefore not been widely used with PRMs: to the best of our knowledge, Shao et al. (2024), Yang et al. (2025a), and Feng et al. (2025) are the only instances in which GRPO is employed with step-level rewards—and these approaches necessitate the modification of the algorithm to accommodate finer-grained reward signals.
In this paper, we show that—under certain mild assumptions—GRPO induces a Monte-Carlo-based PRM (Section 3). Specifically, we prove theoretically that GRPO assigns rewards (and advantages) derived from outcome-level rewards and Monte-Carlo-sampled completions to sub-trajectories, whenever subsets of trajectories within each group share identical prefixes. We then show empirically that this identical-prefix condition is almost always met under real-world conditions, yielding rich step-level process reward structures. These two findings definitively demonstrate that the GRPO objective covertly assigns and optimizes for complex, structured step-level rewards and advantages.
An investigation of the properties of GRPO’s hidden PRM reveals a defect in the objective function that hinders both exploration and exploitation (under different conditions) during RL training (Section 4): namely, a vulnerability to non-uniformly-distributed process steps within a group. To mitigate this shortcoming, we propose including a PRM-aware normalization factor into the GRPO loss function ( $λ$ -GRPO).
We show that $λ$ -GRPO results in higher validation accuracy and a $∼$ 2x training speedup over standard GRPO (Section 5). On downstream reasoning benchmarks, $λ$ -GRPO consistently improves over standard GRPO, demonstrating the superiority of our method.
Our findings call into question the utility of dedicated PRMs for GRPO, and suggest that future work may benefit instead from exploiting the step-level reward signal already available to the GRPO algorithm.
## 2 Background
### 2.1 GRPO
GRPO is a variant of PPO that discards the critic model and GAE of the latter. Instead, for each query $q$ in the training set, GRPO nondeterministically samples a group ${G}$ of $k$ trajectories (completions to $q$ ) and computes the advantage ${a}_i$ for the completion $g^(i)∈{G}$ relative to the mean (outcome) reward of ${G}$ , as in Equation 1 (where ${r}_i$ is the reward for $g^(i)$ ).
$$
{a}_i=\frac{{r}_i-r_{mean}({G})}{r_{std}({G})} \tag{1}
$$
In our theoretical analysis in Section 3, we make two key assumptions: first, we assume the use of the DAPO token-level policy gradient objective (Yu et al., 2025), rather than sample-level loss. Although it differs from the original GRPO formulation laid out in Shao et al. (2024), Yu et al. (2025) show that this objective leads to more stable training, and it is the standard GRPO loss function employed in commonly-used RL packages (e.g. the TRL GRPO trainer https://huggingface.co/docs/trl/main/en/grpo_trainer).
Second, we assume that the number of update iterations per batch ( $μ$ ) is set to $μ=1$ . Under this assumption, the ratio ${P}_i,t$ is fixed at $1.0$ (see Equation 2b), allowing us to ignore the clipping factor of the GRPO loss function in our theoretical analysis.
Under these two assumptions, the per-group GRPO loss $L_GRPO({G})$ reduces to that in Equation 2.
$$
\displaystyle L_GRPO({G})=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}∑_g^(i)∈{G}∑_t=0^\textit{len(g^(i))-1}({P}_i,t·{a}_i)-{D}_i,t
$$
$$
{P}_i,t=\frac{π_θ(g^(i)_t\hskip 2.84526pt|\hskip 2.84526ptq,g^(i)_:t)}{π_θ_{\textit{old}}(g^(i)_t\hskip 2.84526pt|\hskip 2.84526ptq,g^(i)_:t)}
$$
$$
{D}_i,t=β·≤ft(\frac{π_θ_{\textit{ref}}(g^(i)_t\hskip 2.84526pt|\hskip 2.84526ptq,g^(i)_:t)}{π_θ(g^(i)_t\hskip 2.84526pt|\hskip 2.84526ptq,g^(i)_:t)}-\textit{ln}\frac{π_θ_{\textit{ref}}(g^(i)_t\hskip 2.84526pt|\hskip 2.84526ptq,g^(i)_:t)}{π_θ(g^(i)_t\hskip 2.84526pt|\hskip 2.84526ptq,g^(i)_:t)}-1\right)
$$
### 2.2 Process Reward Models (PRMs)
Given an alphabet $Σ$ (i.e. set of tokens), we formally define a PRM as a function $f_φ\colonΣ^*→(Σ^*×ℝ)^*$ parameterized by $φ$ that maps a trajectory $y∈Σ^*$ to the sequence $f_φ(y)=((y_:i_1,r^(y)_0),(y_i_{1:i_2},r^(y)_1),\dots,(y_i_{n-1:},r^(y)_n-1))$ of pairs of process steps (sub-trajectories) $y_i_{k:i_k+1}$ and step-level rewards $r^(y)_k$ .
While PRMs are typically contrasted with outcome reward models (ORMs)—which assign a single reward to the entire trajectory—under the above definition, an ORM $f_φ^\prime^\prime$ is simply a trivial PRM: i.e. $f_φ^\prime^\prime(y)=((y,r^(y)_0))$ .
Both the division of the trajectory $y$ into steps and the assignment of rewards to those steps are dependent upon the PRM in question. When trajectories are clearly delineated into individual steps—e.g. via ReACT-style prompting Yao et al. (2023) or instructing the model to divide its reasoning into demarcated steps—the PRM can simply be directed to assign a reward to each pre-defined step (e.g. Li & Li, 2025). In other cases, trajectories are heuristically split into steps—for example, at high-entropy tokens (e.g. Hou et al., 2025).
Although the assignment of step-level reward can be performed by a model with learned parameters $φ$ (e.g. Uesato et al., 2022), Kazemnejad et al. (2025) and Hou et al. (2025) combine Monte Carlo estimation with outcome-level rewards to yield heuristic PRMs that do not require the labor-intensive annotation of—and are less susceptible to reward-hacking than—their learned counterparts. In cases such as these in which the PRM $f_φ$ is not learned, we simply consider $φ$ to be fixed/trivial.
## 3 The PRM in the Machine
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Hierarchical Clustering of Data Arrays
### Overview
The image presents a visual representation of a hierarchical clustering structure. On the left, there are six distinct data arrays, labeled $g^{(1)}$ through $g^{(6)}$, each composed of color-coded segments containing numerical indices. On the right, a tree diagram illustrates the hierarchical grouping of these arrays, where the color of the text in the tree nodes corresponds to the color of the specific segments within the arrays on the left.
### Components/Axes
* **Left Panel (Data Arrays):** Six horizontal bars representing arrays $g^{(1)}$ through $g^{(6)}$. Each bar is divided into numbered cells starting from 0.
* **Right Panel (Tree Diagram):** A hierarchical tree structure starting from a root node $G$.
* **Color Coding:** The colors used in the array segments on the left are mapped directly to the text colors of the corresponding leaf nodes in the tree diagram on the right.
### Detailed Analysis
#### Left Panel: Data Arrays
The arrays are listed vertically. Each array is segmented by color, and each segment contains sequential integers.
* **$g^{(1)}$**: Length 6 (indices 0-5).
* Segment 1: Blue (indices 0-1)
* Segment 2: Light Pink (indices 2-5)
* **$g^{(2)}$**: Length 9 (indices 0-8).
* Segment 1: Blue (indices 0-1)
* Segment 2: Orange (indices 2-8)
* **$g^{(3)}$**: Length 7 (indices 0-6).
* Segment 1: Red (indices 0-3)
* Segment 2: Green (indices 4-6)
* **$g^{(4)}$**: Length 9 (indices 0-8).
* Segment 1: Red (indices 0-3)
* Segment 2: Purple (indices 4-5)
* Segment 3: Magenta (indices 6-8)
* **$g^{(5)}$**: Length 8 (indices 0-7).
* Segment 1: Red (indices 0-3)
* Segment 2: Purple (indices 4-5)
* Segment 3: Gold/Tan (indices 6-7)
* **$g^{(6)}$**: Length 5 (indices 0-4).
* Segment 1: Teal (indices 0-4)
#### Right Panel: Tree Diagram
The tree branches downward from the root $G = \{g^{(1)}, g^{(2)}, g^{(3)}, g^{(4)}, g^{(5)}, g^{(6)}\}$.
* **Level 1 Branches:**
* **Left Branch (Blue text):** $\{g^{(1)}, g^{(2)}\}$
* **Middle Branch (Red text):** $\{g^{(3)}, g^{(4)}, g^{(5)}\}$
* **Right Branch (Teal text):** $\{g^{(6)}\}$
* **Level 2 Branches:**
* From $\{g^{(1)}, g^{(2)}\}$:
* Left: $\{g^{(1)}\}$ (Light Pink text)
* Right: $\{g^{(2)}\}$ (Orange text)
* From $\{g^{(3)}, g^{(4)}, g^{(5)}\}$:
* Left: $\{g^{(3)}\}$ (Green text)
* Right: $\{g^{(4)}, g^{(5)}\}$ (Purple text)
* **Level 3 Branches:**
* From $\{g^{(4)}, g^{(5)}\}$:
* Left: $\{g^{(4)}\}$ (Magenta text)
* Right: $\{g^{(5)}\}$ (Gold/Tan text)
### Key Observations
* **Shared Prefixes:** The tree structure is determined by shared segments. For example, $g^{(1)}$ and $g^{(2)}$ both start with the "Blue" segment (0-1), which is why they are grouped together in the first branch.
* **Recursive Partitioning:** The "Red" group ($g^{(3)}, g^{(4)}, g^{(5)}$) shares the "Red" segment (0-3). Within this group, $g^{(4)}$ and $g^{(5)}$ are further grouped together because they share the "Purple" segment (4-5).
* **Leaf Node Correspondence:** The final leaf nodes in the tree (e.g., $\{g^{(1)}\}$ in light pink) correspond exactly to the unique, non-shared terminal segment of the respective array on the left.
### Interpretation
This diagram illustrates a **hierarchical clustering process** (likely a decision tree or a prefix-based classification). The data arrays represent sequences or feature vectors. The algorithm partitions these sequences based on shared prefixes (the colored segments).
1. **Primary Split:** The data is split into three main clusters based on the first segment color (Blue, Red, Teal).
2. **Sub-clustering:** Within the Blue and Red clusters, the algorithm continues to split the data based on the next available shared segment.
3. **Termination:** The process terminates when a sequence has a unique terminal segment that distinguishes it from others in its immediate cluster (e.g., $g^{(4)}$ vs $g^{(5)}$).
This is a visual representation of how a system might classify complex data by iteratively identifying commonalities and separating unique features.
</details>
Figure 1: Toy example of a group ${G}=\{g^(1),\dots,g^(6)\}$ (left) and its corresponding ${B}({G})$ tree (right). Tokens (boxes) are numbered for readability—subscripted numbers within boxes only indicate position. Each process set (node in the ${B}({G})$ tree) is a set of trajectories that share a common prefix, and corresponds to a process step (subtrajectory) spanning those shared tokens: in this figure, colored nodes in ${B}({G})$ correspond to those subsequences in ${G}$ that span tokens/boxes of the same color Same-colored boxes in ${G}$ indicate identical sequences of tokens across trajectories only: for example, $g^(3)_:4=g^(4)_:4$ and $g^(4)_4:6=g^(5)_4:6$ , but it is not necessarily the case that e.g. $g^(3)_0=g^(3)_1$ .. GRPO implicitly assigns a step-level reward and advantage to the tokens of each process step, which are computed as functions of the mean outcome-level reward of each trajectory in the corresponding process set.
In Section 3.1, we prove that GRPO theoretically induces a PRM (given the assumptions of Section 2.1) as defined in Section 2.2. However, this PRM is only non-trivial—i.e. not equivalent to an ORM—if subsets of trajectories within each group share identical initial sub-trajectories.
In Section 3.2, we empirically demonstrate that such rich, overlapping prefix structures arise very frequently under real-world conditions: this shows that GRPO is “secretly” a non-trivial PRM.
### 3.1 Theoretical Analysis
Let ${B}({G})=\{λ⊆{G}\hskip 2.84526pt|\hskip 2.84526pt∃ n≥ 0∀ g^(i),g^(k)∈λ\colon g^(i)_:n=g^(k)_:n\}$ be the set of all process sets: sets $λ⊆{G}$ of completions such that all $g^(i)∈λ$ are identical up to the $n^th$ token, for some $n≥ 0$ (see Figure 2). Note that there is a natural tree structure on ${B}({G})$ , which is induced by the $⊇$ relation.
Each $λ∈{B}({G})$ defines a process step within each $g^(i)∈λ$ , spanning the subsequence $g^(i)_s(λ):e(λ)$ from the $s(λ)^th$ to the $e(λ)^th$ tokens of $g^(i)$ . The endpoint $e(λ)$ is defined as the largest $n$ such that $g^(i)_:n=g^(k)_:n$ for all $g^(i),g^(k)∈λ$ , and $s(λ)$ is defined as the endpoint of the immediate parent $Pa_{B({G})}(λ)$ of $λ$ in the tree structure induced on ${B}({G})$ .
$$
\begin{split}&e(λ)=\textit{max}\{n≥ 0\hskip 2.84526pt|\hskip 2.84526pt∀ g^(i),g^(k)∈λ:g^(i)_:n=g^(k)_:n\}\\
&s(λ)=\begin{cases}0&if λ=\textit{root}(B(G))\\
e(Pa_{B({G})}(λ))&otherwise\end{cases}\end{split}
$$
For example, $s(\{g^(4),g^(5)\})=e(\{g^(3),g^(4),g^(5)\})=4$ in Figure 2, and $e(\{g^(4),g^(5)\})=6$ : the process step corresponding to $\{g^(4),g^(5)\}$ spans $g^(4)_4:6$ and $g^(5)_4:6$ .
For each $g^(i)∈{G}$ and each $0≤ t<\textit{len}(g^(i))$ , let $λ^(i,t)∈{B}({G})$ denote the unique process set such that $g^(i)∈λ^(i,t)$ and $s(λ^(i,t))≤ t<e(λ^(i,t))$ . In other words, $λ^(i,t)$ is the process step to which the token $g^(i)_t$ belongs. In Figure 2, $λ^(i,t)$ corresponds to the set whose color matches that of $g^(i)_t$ : $λ^(1,0)=\{g^(1),g^(2)\}$ , $λ^(1,3)=\{g^(1)\}$ , $λ^(5,5)=\{g^(4),g^(5)\}$ , etc.
Now, for each process step defined by some $λ∈{B}({G})$ , we define the step-level reward $\hat{R}(λ)$ via Monte Carlo estimation (Equation 3): $\hat{R}(λ)$ is the mean outcome-level reward of each trajectory in $λ$ . In other words, $\hat{R}(λ)$ is the mean reward of each leaf node dominated by $λ$ in the tree structure induced on ${B}({G})$ —i.e. of each sampled completion to the process step defined by $λ$ .
$$
\hat{R}(λ)=\frac{∑_g^(i)∈λ{r}_i}{|λ|} \tag{3}
$$
For each trajectory $g^(i)∈{G}$ and each $0≤ t<\textit{len}(g^(i))$ define the reward ${R}_i,t$ for the token $g^(i)_t$ as the reward of the process step to which $g^(i)_t$ belongs: ${R}_i,t=\hat{R}(λ^(i,t))$ . For example, in Figure 2, the step-level reward for the sub-trajectories $g^(3)_:4$ , $g^(4)_:4$ , $g^(5)_:4$ is the mean of the outcome-level rewards for $g^(3)$ , $g^(4)$ , and $g^(5)$ : ${R}_3,0=\dots={R}_5,3=\textit{mean}(\{{r}_3,{r}_4,{r}_5\})$ .
By the definition given in Section 2.2, ${R}_i,t$ and ${B}({G})$ clearly define a PRM: each $g^(i)∈{G}$ is mapped to the sequence $(g^(i)_:s(λ_{1)},{R}_i,0),(g^(i)_s(λ_{1):s(λ_2)},{R}_i,s(λ_{1)}),\dots,(g^(i)_s(λ_{n):},{R}_i,s(λ_{n)})$ , where $(λ_0={G})→\dots→(λ_n=\{g^(i)\})$ is the unique path in the tree structure induced on ${B}({G})$ from the root ${G}$ to the node $\{g^(i)\}$ .
Now, define the step-level advantage ${A}_i,t$ for the token $g^(i)_t$ in an analogous manner to the original GRPO definition in Equation 1 —i.e. as the normalized difference between the step-level reward ${R}_i,t$ for $g^(i)_t$ and the mean reward of ${G}$ : ${A}_i,t=({R}_i,t-r_\textit{mean}({G}))/r_\textit{std}({G})$ .
Replacing the term ${a}_i$ with ${A}_i,t$ in Equation 2a yields a PRM-aware RL objective (Equation 4).
$$
L_PRM({G})=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}∑_g^(i)∈{G}∑_t=0^\textit{len(g^(i))-1}({P}_i,t·\hbox{\pagecolor{yellow}$\displaystyle{A}_i,t$})-{D}_i,t \tag{4}
$$
We now show that the standard GRPO objective defined in Equation 2a with outcome-level rewards ( $L_GRPO$ ) is equivalent to the PRM defined in Equations 3 - 4 ( $L_PRM$ ).
**Theorem 1**
*For any query $q$ , policy $π_θ$ , and group ${G}∼π_θ(\textendash\hskip 2.84526pt|\hskip 2.84526ptq)$ with outcome-level rewards $\{r_i\}_g^(i)∈{G}$ : $L_GRPO({G})=L_PRM({G})$ .*
* Proof*
Let $λ$ be any process set in ${B}({G})$ , and let $t$ be any integer such that $s(λ)≤ t<e(λ)$ . We first prove that the sum of the PRM loss terms ${P}_i,t·{A}_i,t-{D}_i,t$ for each trajectory $g^(i)∈λ$ at the token $t$ is equivalent to the sum of the standard GRPO loss terms ${P}_i,t·{a}_i-{D}_i,t$ (Equation 5). Recall that for any $g^(i),g^(k)∈λ$ , $g^(i)_:t+1=g^(k)_:t+1$ by definition: therefore, ${P}_i,t={P}_k,t$ and ${D}_i,t={D}_k,t$ (Equations 2b - 2c). Again by definition, $g^(i),g^(k)∈λ$ implies that ${R}_i,t={R}_k,t=\hat{R}(λ)$ (Equation 3), and so ${A}_i,t={A}_k,t$ . As such, we may define $\hat{P}_t(λ)={P}_k,t$ , $\hat{D}_t(λ)={D}_k,t$ , and $\hat{A}(λ)={A}_k,t$ in Equation 5, choosing any arbitrary $g^(k)∈λ$ . $$
\begin{split}&∑_g^(i)∈λ({P}_i,t·{A}_i,t)-{D}_i,t=|λ|·((\hat{P}_t(λ)·\hat{A}(λ))-\hat{D}_t(λ))\\
&=|λ|·≤ft(≤ft(\hat{P}_t(λ)·\frac{\hat{R}(λ)-r_\textit{mean}({G})}{r_\textit{std}({G})}\right)-\hat{D}_t(λ)\right)=|λ|·≤ft(≤ft(\hat{P}_t(λ)·\frac{∑_g^(i)∈λ\frac{{r}_i}{|λ|}-r_\textit{mean}({G})}{r_\textit{std}({G})}\right)-\hat{D}_t(λ)\right)\\
&=≤ft(\hat{P}_t(λ)\frac{|λ|(∑_g^(i)∈λ\frac{{r}_i}{|λ|}-r_\textit{mean}({G}))}{r_\textit{std}({G})}\right)-|λ|\hat{D}_t(λ)=≤ft(\hat{P}_t(λ)\frac{∑_g^(i)∈λ{r}_i-∑_g^(i)∈λr_\textit{mean}({G})}{r_\textit{std}({G})}\right)-|λ|\hat{D}_t(λ)\\
&=≤ft(∑_g^(i)∈λ{P}_i,t\frac{{r}_i-r_\textit{mean}({G})}{r_\textit{std}({G})}\right)-∑_g^(i)∈λ{D}_i,t=∑_g^(i)∈λ({P}_i,t·{a}_i)-{D}_i,t\end{split} \tag{5}
$$ Now, letting $t_\textit{max}=\textit{max}_g^(i)∈{G}\hskip 2.84526pt\textit{len}(g^(i))$ , for each $0≤ t<t_\textit{max}$ we can define a partition ${X}_t⊆{B}({G})$ of $\{g^(i)∈{G}\hskip 2.84526pt|\hskip 2.84526pt\textit{len}(g^(i))≤ t\}$ such that ${X}_t=\{λ∈{B}({G})\hskip 2.84526pt|\hskip 2.84526pts(λ)≤ t<e(λ)\}$ is the set of all process sets corresponding to a token span containing the index $t$ . The GRPO loss term $L_GRPO({G})$ (Equation 2a) can be equivalently expressed as in Equation 6 (and analogously for $L_PRM({G})$ of Equation 4). $$
L_GRPO({G})=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_t=0^t_\textit{max-1}∑_λ∈{X_t}∑_g_{i∈λ}({P}_i,t·{a}_i)-{D}_i,t \tag{6}
$$ We then have the following equalities by Equations 5 and 6: $$
\begin{split}L_GRPO({G})&=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_g^(i)∈{G}∑_t=0^\textit{len(g^(i))-1}({P}_i,t·{a}_i)-{D}_i,t=\\
\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_t=0^t_\textit{max-1}&∑_λ∈{X_t}∑_g^(i)∈λ({P}_i,t·{a}_i)-{D}_i,t=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_t=0^t_\textit{max-1}∑_λ∈{X_t}∑_g^(i)∈λ({P}_i,t·{A}_i,t)-{D}_i,t\\
&=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_g^(i)∈{G}∑_t=0^\textit{len(g^(i))-1}({P}_i,t·{A}_i,t)-{D}_i,t=L_PRM({G})\end{split}
$$
∎
In other words, the standard GRPO objective defined in Equation 2a automatically induces the PRM and PRM-aware objective defined in Equations 3 - 4.
### 3.2 Empirical Analysis
The theoretical analysis in Section 3.1 shows that the GRPO objective induces a PRM: it remains to be shown, however, that this induced PRM is non-trivial. We refer to the set ${B}({G})$ of process sets as trivial if it contains only singleton sets And, by definition, ${G}$ itself, with $s({G})=e({G})=0$ . —i.e. ${B}({G})=\{{G}\}∪\{\{g^(i)\}\hskip 2.84526pt|\hskip 2.84526ptg^(i)∈{G}\}$ .
If ${B}({G})$ is trivial, then there are no meaningful process steps within each trajectory, and only the outcome reward has an effect on learning. On the other hand, if ${B}({G})$ is not trivial, then Theorem 1 entails that the standard GRPO objective of Equation 2a induces a non-trivial PRM.
To analyze the complexity of real-world, GRPO-derived step-level rewards, we empirically evaluated ${B}({G})$ structures generated during standard GRPO training: for each group ${G}$ , we computed its ${B}({G})$ tree, and measured the number of intermediate nodes between the root ${G}$ and each terminal node $\{g^(i)\}$ (path depth), as a proxy for the complexity of the ${B}({G})$ structure.
In addition, for each completion $g^(i)∈{G}$ , we counted the number of tokens $n^(i)_\textit{term}=e(\{g^(i)\})-s(\{g^(i)\})$ contained in the process step corresponding to the terminal node $\{g^(i)\}$ —i.e. the number of tokens unique to $g^(i)$ —and calculated the intermediate proportion ${p}_i$ of $g^(i)$ : ${p}_i=(\textit{len}(g^(i))-n^(i)_\textit{term})/\textit{len}(g^(i))$ . Higher values of ${p}_i$ indicate that a greater proportion of the trajectory $g^(i)$ belongs to intermediate process steps and is therefore assigned non-trivial step-level reward.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Line Charts: Training Metrics Comparison (Group Size 36 vs. 6)
### Overview
This image displays a 2x3 grid of line charts comparing three specific metrics—**Validation Reward**, **Path Depth**, and **Intermediate Proportion**—across two different experimental configurations: **Group Size = 36** (top row) and **Group Size = 6** (bottom row). The charts illustrate how these metrics evolve over the course of training steps.
### Components/Axes
* **Layout**: 2 rows, 3 columns.
* **Top Row (Group Size = 36)**: X-axis range is 0 to ~270 steps.
* **Bottom Row (Group Size = 6)**: X-axis range is 0 to ~1700 steps.
* **Axes**:
* **X-axis**: Labeled "STEP" for all charts.
* **Y-axis**: Varies by metric (Validation Reward, Path Depth, Intermediate Proportion).
* **Legends**:
* Present in "Path Depth" and "Intermediate Proportion" charts.
* **Mean**: Solid blue line.
* **Median**: Dashed orange line.
---
### Detailed Analysis
#### Top Row: Group Size = 36
1. **Validation Reward (Top-Left)**
* **Trend**: The line shows a general upward trend with a notable dip around step 100.
* **Data Points**: Starts at ~0.20 (step 0). Rises to ~0.27 (step 50), dips to ~0.23 (step 100), rises to ~0.32 (step 150), dips slightly to ~0.30 (step 200), and ends at ~0.34 (step 270).
2. **Path Depth (Top-Center)**
* **Trend**: Both Mean and Median lines slope upward. The Median line exhibits a "staircase" pattern, indicating discrete jumps in depth values.
* **Data Points**: Starts at ~3.5. At step 270, the Mean is ~9.5, while the Median is significantly higher at ~12.5.
3. **Intermediate Proportion (Top-Right)**
* **Trend**: Both lines show an exponential-like growth curve, starting near 0 and rapidly accelerating toward 1.0.
* **Data Points**: Starts at 0.0. At step 270, both Mean and Median converge near 1.0.
#### Bottom Row: Group Size = 6
1. **Validation Reward (Bottom-Left)**
* **Trend**: Highly volatile upward trend. The reward fluctuates significantly throughout the training process.
* **Data Points**: Starts at 0.20. Rapidly climbs to ~0.36 (step 200). Experiences a sharp drop to ~0.25 (step 750). Recovers and peaks at ~0.38 (step 1700).
2. **Path Depth (Bottom-Center)**
* **Trend**: Both lines show a step-like upward progression. The Median (orange) is consistently higher than the Mean (blue) after step 400.
* **Data Points**: Starts at 1.0. At step 1700, the Mean is ~3.3, while the Median reaches 4.0.
3. **Intermediate Proportion (Bottom-Right)**
* **Trend**: Similar to the top-right chart, this shows an exponential growth curve, but over a much longer step duration (1700 steps).
* **Data Points**: Starts at 0.0. At step 1700, the Mean is ~0.7, while the Median is higher, at ~0.9.
---
### Key Observations
* **Scale Disparity**: The "Group Size = 6" configuration allows for significantly more training steps (~1700) compared to "Group Size = 36" (~270). This suggests that smaller group sizes are computationally cheaper per step, allowing for longer training durations.
* **Volatility**: The "Validation Reward" for Group Size 6 is much more volatile than Group Size 36. This is typical in reinforcement learning where smaller batch sizes (or group sizes) introduce more noise into the gradient estimates.
* **Median vs. Mean**: In the "Path Depth" and "Intermediate Proportion" charts, the Median (orange dashed) is frequently higher than the Mean (blue solid), suggesting a right-skewed distribution of values in the underlying data.
### Interpretation
The data suggests an ablation study or hyperparameter tuning experiment comparing the effects of "Group Size" on model training dynamics.
* **Efficiency vs. Stability**: The Group Size 36 configuration reaches a reasonable reward (~0.34) very quickly (270 steps) with a smoother curve. The Group Size 6 configuration requires much longer training (1700 steps) to reach a slightly higher reward (~0.38), but the path to get there is noisy and unstable.
* **Convergence**: The "Intermediate Proportion" charts show that regardless of group size, the model eventually learns to maximize this proportion (approaching 1.0), though the rate of convergence is stretched over the different step scales.
* **Conclusion**: If computational resources are limited, Group Size 36 is likely the superior choice for rapid prototyping due to the smoother convergence. If the goal is to squeeze out maximum performance and the system can handle longer training times, Group Size 6 may be preferable, provided the volatility can be managed.
</details>
Figure 2: Validation reward (exact-match accuracy; left), ${B}({G})$ root-to-terminal path depth (center), and proportions of trajectories spanned by intermediate (non-terminal) process steps (right) for GRPO runs with group sizes of 36 (top) and 6 (bottom).
#### Experimental Setup.
We trained two DeepSeek-R1-Distill-Qwen-1.5B models (DeepSeek-AI, 2025) on the OpenRS (Dang & Ngo, 2025) dataset using the standard GRPO algorithm and objective of Equation 2. We selected 125 OpenRS examples at random to serve as a validation set.
The first model trained for 1675 steps with a group size of six and a learn rate of $6× 10^-6$ . The second was trained with a group size of 36 and a learn rate of $10^-6$ for 275 steps (due to the larger group size). Both models were trained with a maximum new token limit of 4096, a batch size of four, and a temperature of 0.75. Additional training details are located in Appendix B.
#### Results.
Figure 2 shows that both path depth and intermediate proportion increase drastically as validation reward saturates, for group sizes of six and 36. These results are supported by Yu et al. (2025), who find that entropy decreases sharply as GRPO training progresses: this indicates that increasingly rich PRM-inducing structures arise as the model converges on a locally optimal policy.
In addition, found that only twelve of 6,700 ${B}({G})$ structures were trivial with a group size of six ( $∼$ 0.2%). With a group size of 36, zero trivial ${B}({G})$ structures arose out of the 1,100 generated groups. Examples of non-trivial ${B}({G})$ structures from this experiment are given in Appendix C.
In conjunction with the theoretical analysis of Section 3.1, the results of this experiment demonstrate that GRPO induces a non-trivial PRM under real-world conditions. In Section 4, we show that this induced PRM carries a serious flaw that is detrimental to RL training, and propose a minor modification to the GRPO algorithm to mitigate this shortcoming.
## 4 Proposed Approach: $λ$ -GRPO
Viewing the GRPO objective in terms of process set partitions ${X}_t$ (see Equation 6), we note that the contribution of each trajectory $g^(i)∈{G}$ to the loss at index $t$ is identical to that of all other trajectories in the process set $λ^(i,t)$ (where $\hat{P}_t(λ)$ , $\hat{D}_t(λ)$ , and $\hat{A}(λ)$ are defined as in Equation 5):
$$
\begin{split}\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_g^(i)∈{G}&∑_t=0^\textit{len(g^(i))-1}({P}_i,t·{a}_i)-{D}_i,t=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_t=0^t_\textit{max-1}∑_λ∈{X_t}∑_g^(i)∈λ({P}_i,t·{A}_i,t)-{D}_i,t\\
=&\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_t=0^t_\textit{max-1}∑_λ∈{X_t}|λ|·((\hat{P}_t(λ)·\hat{A}(λ))-\hat{D}_t(λ))\end{split} \tag{7}
$$
The contribution of each process set $λ∈{B}({G})$ to the overall loss, $\hat{P}_t(λ)·\hat{A}(λ)-\hat{D}_t(λ)$ , is scaled by $|λ|$ : this carries the danger of harming exploration (for $\hat{A}(λ)<0$ ) and exploitation (for $\hat{A}(λ)>0$ ). Consider some process set $λ$ with $|λ|\gg 1$ . If $\hat{A}(λ)>0$ , then the increase in probability assigned to the process step corresponding to $λ$ by $π_θ$ under GRPO is compounded by a factor of $|λ|$ , decreasing the likelihood of exploring process steps that are dissimilar from $λ$ in subsequent training episodes.
Conversely, if $\hat{A}(λ)<0$ , then the decrease in probability assigned to $λ$ under GRPO is compounded by a factor of $|λ|$ , decreasing the likelihood of exploiting high-reward trajectories in $λ$ . To illustrate, consider the group ${G}$ in Figure 2, assume ${r}_1={r}_2={r}_6=0.5$ , ${r}_4={r}_5=0$ , ${r}_3=1$ , and let $λ=\{g^(3),g^(4),g^(5)\}$ . Then $\hat{A}(λ)=-0.22$ : despite the fact that $g^(3)$ has the highest reward, the probability of the sub-trajectory $g^(3)_:4$ is decreased under the GRPO objective, thereby decreasing the overall likelihood of generating the completion $g^(3)$ . The term $|λ|$ in Equation 7 then scales this decrease in probability by a factor of three.
We propose scaling the token-level loss for $g^(i)_t$ by $|λ^(i,t)|^-1$ ( $λ$ -GRPO; Equation 8): this has the effect of canceling out the term $|λ|$ in Equation 7, so that each process set contributes equally to the loss at index $t$ .
$$
\begin{split}L_λ-GRPO({G})&=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_g^(i)∈{G}∑_t=0^\textit{len(g^(i))-1}\frac{({P}_i,t·{a}_i)-{D}_i,t}{|λ^(i,t)|}\\
&=\frac{1}{∑_g^(i)∈{G}\textit{len}(g^(i))}·∑_t=0^t_\textit{max-1}∑_λ∈{X_t}(\hat{P}_t(λ)·\hat{A}(λ))-\hat{D}_t(λ)\end{split} \tag{8}
$$
## 5 Experiments
To evaluate our proposed approach, we trained DeepSeek-R1-Distill-Qwen-1.5B and Llama-3.2-1B-Instruct https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct with the $λ$ -GRPO (Equation 8) objective on the OpenRS dataset of Section 3.2, and compared them to standard GRPO (Equation 2a) models trained with an identical setup. All models were evaluated on an OpenRS validation set and five downstream reasoning benchmarks (see Section 5.1).
### 5.1 Experimental Setup
All models were trained for 1,000 steps with a group size of six, a batch size of four, a maximum of 4,096 new tokens, and a temperature of 0.75. We conducted two sets of trials across the two models (for a total of four trials): in the first, we set the KL coefficient $β=0.0$ , and in the second $β=0.04$ . The Qwen models were trained with a learn rate of $10^-6$ ; the Llama models were trained with a learn rate of $5× 10^-7$ for the $β=0.0$ trial and $10^-7$ for $β=0.04$ (as training was highly unstable with higher learn rates for Llama). Additional training details are located in Appendix B.
We evaluated the models on the AIME24 https://huggingface.co/datasets/AI-MO/aimo-validation-aime, MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024), AMC23 https://huggingface.co/datasets/AI-MO/aimo-validation-amc, Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024) benchmarks, using the LightEval framework Habib et al. (2023); Dang & Ngo (2025). All models were evaluated at the checkpoint corresponding to the step at which they achieved maximum validation accuracy. As in the experiment in Section 3.2, we withheld 125 examples as a validation set.
### 5.2 Results and Discussion
<details>
<summary>x3.png Details</summary>
### Visual Description
## Line Charts: Validation Reward Comparison for Qwen and Llama Models
### Overview
This image displays four distinct line charts comparing the performance of two optimization methods, "GRPO" and "$\lambda$-GRPO" (lambda-GRPO), across two different base models (Qwen and Llama) and two different $\beta$ (beta) values (0.0 and 0.04). The charts track "VALIDATION REWARD" over training "STEP" counts. In all four scenarios, the $\lambda$-GRPO method (represented by the purple line) consistently achieves higher peak validation rewards than the standard GRPO method (represented by the cyan line).
### Components/Axes
* **X-Axis:** Labeled "STEP". The scale ranges from 0 to approximately 900+ across all charts.
* **Y-Axis:** Labeled "VALIDATION REWARD". The scale varies by chart to accommodate the specific reward ranges of the models.
* **Legend:** Located in the bottom-right corner of each chart.
* **Cyan Line:** Labeled "GRPO".
* **Purple Line:** Labeled "$\lambda$-GRPO" (where $\lambda$ is the Greek letter lambda).
* **Annotations:**
* **Vertical Dashed Lines:** Indicate the step at which the peak reward occurs for each method (Purple dashed line for $\lambda$-GRPO, Cyan dashed line for GRPO).
* **Horizontal Dashed Line:** Indicates the peak reward level achieved by the $\lambda$-GRPO method.
* **Coordinate Labels:** Text labels in brackets/parentheses indicate the specific (Step, Reward) coordinates for the peak values of both methods.
---
### Detailed Analysis
#### 1. Qwen ($\beta=0.0$)
* **Trend:** Both lines show an upward trend initially. The purple line ($\lambda$-GRPO) rises more sharply and maintains a higher average reward throughout the training process compared to the cyan line (GRPO).
* **Peak Values:**
* **$\lambda$-GRPO (Purple):** Peaks at step **325** with a reward of **0.429**.
* **GRPO (Cyan):** Peaks at step **900** with a reward of **0.373**.
#### 2. Qwen ($\beta=0.04$)
* **Trend:** Both lines exhibit significant volatility. The purple line ($\lambda$-GRPO) generally stays above the cyan line.
* **Peak Values:**
* **$\lambda$-GRPO (Purple):** Peaks at step **400** with a reward of **0.381**.
* **GRPO (Cyan):** Peaks at step **600** with a reward of **0.373**.
#### 3. Llama ($\beta=0.0$)
* **Trend:** The purple line ($\lambda$-GRPO) shows a rapid ascent to its peak, while the cyan line (GRPO) remains consistently lower.
* **Peak Values:**
* **$\lambda$-GRPO (Purple):** Peaks at step **150** with a reward of **0.19**.
* **GRPO (Cyan):** Peaks at step **350** with a reward of **0.159**.
#### 4. Llama ($\beta=0.04$)
* **Trend:** Both lines show a gradual upward trend with fluctuations. The purple line ($\lambda$-GRPO) consistently outperforms the cyan line.
* **Peak Values:**
* **$\lambda$-GRPO (Purple):** Peaks at step **325** with a reward of **0.143**.
* **GRPO (Cyan):** Peaks at step **700** with a reward of **0.135**.
---
### Key Observations
* **Performance Superiority:** In every single chart, the $\lambda$-GRPO (purple) method achieves a higher peak validation reward than the standard GRPO (cyan) method.
* **Convergence Speed:** The $\lambda$-GRPO method consistently reaches its peak reward at an earlier training step than the GRPO method.
* **Consistency:** The performance gap is consistent across both model architectures (Qwen and Llama) and both hyperparameter settings ($\beta=0.0$ and $\beta=0.04$).
### Interpretation
The data demonstrates that the $\lambda$-GRPO optimization strategy is more effective than the standard GRPO method for these specific models.
* **Efficiency:** Because $\lambda$-GRPO reaches higher rewards at earlier steps, it suggests that the $\lambda$ modification likely introduces a mechanism that accelerates convergence or provides better gradient stability during the training process.
* **Robustness:** The fact that this trend holds true across different models (Qwen vs. Llama) and different $\beta$ values suggests that $\lambda$-GRPO is a robust improvement over the baseline GRPO, rather than a hyperparameter-specific artifact.
* **Peircean Investigative Note:** The consistent "earlier peak" for $\lambda$-GRPO suggests that standard GRPO might be prone to slower learning or potentially overfitting/degrading later in the training cycle, whereas $\lambda$-GRPO captures the optimal policy more efficiently.
</details>
Figure 3: Models’ validation accuracy across training steps. Peak accuracy is highlighted by vertical, dashed lines.
| Model Qwen | $β$ — | Version Base | AIME24 0.2000 ±0.0743 | MATH-500 0.8300 ±0.0168 | AMC23 0.7500 ±0.0693 | Minerva 0.2978 ±0.0278 | OB 0.5096 ±0.0193 | Avg. 0.5175 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0 | GRPO | 0.3333 ±0.0875 | 0.7660 ±0.0190 | 0.6250 ±0.0775 | 0.2500 ±0.0263 | 0.4444 ±0.0191 | 0.4837 | |
| $λ$ -GRPO (ours) | 0.3667 ±0.0895 | 0.8460 ±0.0162 | 0.7500 ±0.0693 | 0.2904 ±0.0276 | 0.5348 ±0.0192 | 0.5576 | | |
| 0.04 | GRPO | 0.3000 ±0.0851 | 0.8660 ±0.0152 | 0.7500 ±0.0693 | 0.2610 ±0.0267 | 0.5200 ±0.0192 | 0.5394 | |
| $λ$ -GRPO (ours) | 0.4000 ±0.0910 | 0.8340 ±0.0167 | 0.8000 ±0.0641 | 0.2978 ±0.0278 | 0.5378 ±0.0192 | 0.5739 | | |
| Llama | — | Base | 0.0000 ±0.0000 | 0.2280 ±0.0188 | 0.0750 ±0.0422 | 0.0478 ±0.0130 | 0.0563 ±0.0089 | 0.0814 |
| 0.0 | GRPO | 0.0000 ±0.0000 | 0.2300 ±0.0188 | 0.0750 ±0.0422 | 0.0551 ±0.0130 | 0.0607 ±0.0089 | 0.0842 | |
| $λ$ -GRPO (ours) | 0.0000 ±0.0000 | 0.2620 ±0.0197 | 0.1250 ±0.0530 | 0.0515 ±0.0134 | 0.0622 ±0.0092 | 0.1001 | | |
| 0.04 | GRPO | 0.0000 ±0.0000 | 0.2180 ±0.0185 | 0.1750 ±0.0608 | 0.0515 ±0.0134 | 0.0533 ±0.0087 | 0.0996 | |
| $λ$ -GRPO (ours) | 0.0333 ±0.0333 | 0.2560 ±0.0195 | 0.0750 ±0.0422 | 0.0735 ±0.0159 | 0.0489 ±0.0083 | 0.0973 | | |
Table 1: Exact-match accuracy for the base and GRPO-/ $λ$ -GRPO-trained Llama and Qwen models on downstream reasoning datasets ( $OB=OlympiadBench$ ). The best results in each column are indicated in bold, and the best results within each model type (i.e. Llama or Qwen) are underlined. Confidence intervals are subscripted. For each $λ$ -GRPO-trained model, results are given in green if it outperforms its GRPO-trained counterpart and the base model; yellow if it outperforms only its GRPO-trained counterpart; orange if it only improves over the base model; and red if it fails to outperform either model (see Table 2 in the Appendix for exact differences).
All four $λ$ -GRPO models reach a higher validation accuracy in fewer steps than their GRPO counterparts (see Figure 3): on average, $λ$ -GRPO represents a more than 10% increase over the standard GRPO validation accuracy—in less than half of the number of training steps.
This increase in validation accuracy corresponds to improved performance on downstream reasoning tasks (see Table 1). In total, the $λ$ -GRPO models outperform standard GRPO on 15/20 cells (excluding average performance) in Table 1, and they improve over the base Llama/Qwen models on 14/20 cells. Only the Llama $λ$ -GRPO model with $β=0.04$ failed to outperform its GRPO counterpart on average downstream performance—this model still outperformed standard GRPO on a majority (3/5) of the tasks.
In addition, these performance gains result in effectively zero training slowdown: as we are merely detecting ${B}({G})$ structures that occur during training—rather than generating them (e.g. Kazemnejad et al., 2025; Hou et al., 2025; Yang et al., 2025a) —the added computational cost from $λ$ -GRPO (vs. standard GRPO) is negligible.
## 6 Related Work
#### Monte Carlo Sampling for Finer-Grained Rewards.
As discussed in Section 2.2, the labor-intensive human annotation required to obtain step-level rewards for PRM training has driven the development of heuristic methods for PRMs and PRM-like finer-grained reward signals, in particular those based on Monte Carlo estimation. Kazemnejad et al. (2025) replace the critic model in PPO with Monte Carlo estimation: multiple completions are sampled for each step, and the value for that step is derived from their mean outcome reward. On the other hand, Wang et al. (2024) train a neural PRM, using Monte Carlo estimation in a similar manner to Kazemnejad et al. (2025) in order to obtain step-level training rewards, and thereby avoid the need for costly human annotation.
Xie et al. (2024) generate step-level preference data for Direct Preference Optimization (DPO; Rafailov et al., 2023) training via Monte Carlo Tree Search and outcome-level rewards: sequences of overlapping trajectories are obtained by forking trajectories at defined split points to construct tree structures similar to the $B(G)$ trees introduced in Section 3.1. The daughter nodes with the highest and lowest mean reward are then selected as the preferred and dispreferred (respectively) sequences for step-level DPO. Similarly, Hou et al. (2025) construct $B(G)$ -like trees by splitting generated trajectories at high-entropy tokens to create multiple completions to the same initial trajectory prefix. Subtrajectory rewards are then derived from the mean reward of the corresponding node’s daughters in the tree structure. Yang et al. (2025a) employ an analogous approach to generate step-level rewards for GRPO training. Unlike standard GRPO, however, the advantages for each node (step) are computed relative to the rewards of node’s sisters in the tree structure, rather than the entire group.
These methods are orthogonal to our approach: they apply Monte Carlo estimation to explicitly construct step-level reward signals from outcome-level rewards, while we leverage the implicit step-level rewards already present in standard GRPO.
#### PRMs with GRPO.
Aside from Yang et al. (2025a), GRPO has been employed with PRMs in Shao et al. (2024) and Feng et al. (2025). Shao et al. (2024) modify the advantage computation of GRPO to account for step-level rewards: normalized rewards are computed relative to all step-level rewards of all trajectories in the group, and the advantage for each step is the sum of the normalized reward of each subsequent step in its trajectory. Feng et al. (2025) construct a two-level variant of GRPO, in which standard, trajectory-level GRPO advantage is combined with a step-level GRPO advantage, and step-level groups are dynamically computed according to the similarity of steps across trajectories.
Our results in Sections 3 and 5 call into question the necessity of adapting GRPO to step-level rewards, given the rich step-level reward signal already present in the outcome-level variant of the algorithm.
## 7 Conclusion
In this paper, we demonstrated both theoretically and empirically that the standard GRPO algorithm induces a PRM that derives step-level rewards via Monte Carlo estimation. We then showed that this hidden PRM is faulty, and as a result is potentially detrimental to exploration and exploitation. To mitigate this flaw, we introduced a process-step-aware scaling factor to GRPO to derive the $λ$ -GRPO objective. Models trained with $λ$ -GRPO reach higher validation accuracy faster than with standard GRPO, and achieve improved performance on downstream reasoning tasks.
Our results indicate that it is possible to leverage the existing PRM structure inherent in the outcome-based GRPO algorithm, rather than employing costly, explicitly-defined PRMs. The limitations of this work are discussed in Appendix A.
## Reproducibility Statement
We provide the complete proof of Theorem 1 in the main body of the paper. All settings and hyperparameters for the experiments conducted in this work are given in Sections 3.2 / 5.1 and Appendix B. All code is available on GitHub: https://github.com/coli-saar/grpo-prm/.
## References
- Cui et al. (2025) Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
- Dang & Ngo (2025) Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn’t. arXiv preprint arXiv:2503.16219, 2025.
- DeepSeek-AI (2025) DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
- Feng et al. (2025) Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978, 2025.
- Habib et al. (2023) Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/lighteval.
- He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828–3850, 2024.
- Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
- Hou et al. (2025) Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. TreeRL: LLM reinforcement learning with on-policy tree search. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12355–12369, 2025.
- Kazemnejad et al. (2025) Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training of LLMs. In Forty-second International Conference on Machine Learning, 2025.
- Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
- Li & Li (2025) Wendi Li and Yixuan Li. Process reward model with q-value rankings. In The Thirteenth International Conference on Learning Representations, 2025.
- Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
- Luo et al. (2025) Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2, 2025. Notion Blog.
- Muennighoff et al. (2025) Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
- Qian et al. (2025) Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025.
- Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 53728–53741, 2023.
- Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- Setlur et al. (2025) Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, 2025.
- Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
- Sullivan et al. (2025) Michael Sullivan, Mareike Hartmann, and Alexander Koller. Procedural environment generation for tool-use agents. arXiv preprint arXiv:2506.11045, 2025.
- Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
- Wang et al. (2024) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439, 2024.
- Xie et al. (2024) Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, 2024.
- Yang et al. (2025a) Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. Treerpo: Tree relative policy optimization. arXiv preprint arXiv:2506.05183, 2025a.
- Yang et al. (2025b) Zonglin Yang, Zhexuan Gu, Houduo Qi, and Yancheng Yuan. Accelerating rlhf training with reward variance increase. arXiv preprint arXiv:2505.23247, 2025b.
- Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
- Yu et al. (2025) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
- Zhang et al. (2025) Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025.
| Model Qwen | $β$ 0.0 | Version Base | AIME24 +0.1667 | MATH-500 +0.0160 | AMC23 0.0000 | Minerva -0.0074 | OB +0.0252 | Avg. +0.0401 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GRPO | +0.0334 | +0.0800 | +0.1250 | +0.0404 | +0.0904 | +0.0738 | | |
| 0.04 | Base | +0.2000 | +0.0040 | +0.0500 | 0.0000 | +0.0282 | +0.0564 | |
| GRPO | +0.1000 | -0.0320 | +0.0500 | +0.0368 | +0.0178 | +0.0345 | | |
| Llama | 0.0 | Base | 0.0000 | +0.0340 | +0.0500 | +0.0037 | +0.0059 | +0.0187 |
| GRPO | 0.0000 | +0.0320 | +0.0500 | -0.0036 | +0.0015 | +0.0160 | | |
| 0.04 | Base | +0.0333 | +0.0280 | 0.0000 | +0.0257 | -0.0074 | +0.0159 | |
| GRPO | +0.0333 | +0.0380 | -0.1000 | +0.0220 | -0.0044 | -0.0022 | | |
Table 2: Difference in accuracy between the $λ$ -GRPO-trained models, and their corresponding base models and standard-GRPO-trained counterparts. Positive differences (i.e. $λ$ -GRPO outperforms the comparison model) are highlighted in green; negative differences (i.e. the comparison model outperforms $λ$ -GRPO) are highlighted in red. For example, the top-most entry in the AIME24 column indicates that the $λ$ -GRPO Qwen model with $β=0.0$ outperformed the base DeepSeek-R1-Distill-Qwen-1.5B by 0.1667 on the AIME24 benchmark.
## Appendix A Limitations
Due to computational resource constraints, we were only able to conduct the experiments in Sections 3.2 and 5 with relatively small models: 1.5 billion (Qwen) and 1 billion (Llama) parameters. Similarly, we only use one dataset for RL training in both experiments—although OpenRS is a combination of the s1 (Muennighoff et al., 2025) and DeepScaleR Luo et al. (2025) datasets. Future work should extend our findings regarding the non-triviality of the GRPO-induced PRM and the effectiveness of $λ$ -GRPO to larger models and more diverse (training) datasets.
Finally, the objective of this work is to expose the PRM induced by the GRPO algorithm, and to highlight the deficiencies of that PRM as described in Section 4. To that end, our proposed $λ$ -GRPO method does not actually remedy the anti-exploitation effect of the GRPO-induced PRM—it merely lessens its impact. In future work, we intend to investigate more extensive modifications to the GRPO algorithm, with the goal of entirely solving the problems laid out in Section 4.
## Appendix B Experimental Setup
All experiments were conducted on a single NVIDIA H100 GPU. We trained all models with 24 gradient accumulation steps per step and a generation batch size of 6. The models were evaluated on the validation split every 25 training steps.
We additionally hard-coded the generation procedure to halt after “\boxed{…}” was detected: this was to prevent the model from generating multiple boxed answers for a single prompt.
## Appendix C ${B}({G})$ Structure Examples
The following (Figures 4, 5, 6) represent ${B}({G})$ structures on groups generated during the group size 6 trial of the experiment in Section 3.2. A full trajectory is reconstructed by tracing the unique path from the root to a terminal node. The root ( red) corresponds to the prompt/query $q$ . Terminal nodes ( yellow) denote singleton process steps $\{g^(i)\}$ ; each non-terminal node $λ$ (white; including the root) denotes the process step corresponding to the set of all terminal nodes dominated by $λ$ . For the sake of presentation, overlong terminal steps are truncated with “…”.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Diagram: Mathematical Problem Decomposition Tree
### Overview
The image displays a hierarchical tree diagram representing a "Chain of Thought" or problem-decomposition process. The root node contains a formal mathematical problem statement, while the subsequent branches illustrate various ways to restate, parse, or "unpack" that problem into conversational or analytical segments. The diagram appears to visualize how an AI or a student might break down a complex mathematical prompt.
### Components/Axes
* **Structure:** A top-down tree diagram with a single root node and multiple levels of branching.
* **Nodes:** Rectangular boxes with rounded corners.
* **Color Coding:**
* **Red/Pink (Top):** The root node containing the original problem statement.
* **White (Intermediate):** Nodes acting as connectors or partial phrases.
* **Yellow/Cream (Leaf/Terminal):** Nodes containing restatements or detailed breakdowns of the problem.
* **Flow:** Directed arrows indicate the progression from the formal problem to various conversational restatements.
### Detailed Analysis
#### Root Node (Top, Center)
* **Content:** "Find, with proof, all pairs of positive integers $(n,d)$ with the following property: for every integer $S$, there exists a unique non-decreasing sequence of $n$ integers $a_1, a_2, ..., a_n$ such that $a_1 + a_2 + ... + a_n = S$ and $a_n - a_1 = d$."
#### Level 1 Branches
* **Left Branch (Yellow):** "Okay, so I have this problem where I need to find all pairs of positive integers $(n, d)$ such that for every integer $S$, there exists a unique non-decreasing sequence of $n$ integers ..."
* **Right Branch (White):** "Alright, so I"
#### Level 2 Branches (from "Alright, so I")
* **Left Branch (Yellow):** "'ve been given this interesting problem to solve. It asks to find all pairs of positive integers $(n, d)$ such that for every integer $S$, there exists a unique non-decreasing sequence of $n$ integers ..."
* **Right Branch (White):** "have this problem"
#### Level 3 Branches (from "have this problem")
* **Left Branch (White):** "where I need to find all pairs of positive integers $(n, d)$ such that for every integer $S$, there exists a unique non-decreasing sequence of $n$ integers $a$"
* **Level 4 (Left-most, Yellow):** "_1, a_2, ..., a_n where the sum of these a_i's is S, and the difference between the last term a_n and the first term a_1 is equal to d.\n\nOkay, let me parse this again. We need pairs (n, d) where:\n\n..."
* **Level 4 (Left-center, Yellow):** "1, a2, ..., an where the sum of these integers is S and the difference between the last and the first term is d. Hmm, okay, let me try to unpack this step by step.\n\nFirst, let me make sure I understand ..."
* **Center Branch (Yellow):** "to solve: find all pairs of positive integers $(n, d)$ such that for every integer $S$, there exists a unique non-decreasing sequence of $n$ integers $a_1, a_2, ..."
* **Right Branch (Yellow):** "here: I need to find all pairs of positive integers $(n, d)$ such that for every integer $S$, there exists a unique non-decreasing sequence of $n$ integers $a_1, a_2, ..."
### Key Observations
* **Redundancy:** There is a high degree of repetition across the nodes. The core mathematical problem is restated in almost every branch, indicating a process of "thinking aloud" or iterative prompt refinement.
* **Progression:** The diagram moves from a formal, rigid mathematical statement at the top to more conversational, "human-like" phrasing as it descends.
* **Decomposition:** The bottom-most nodes (Level 4) are the only ones that attempt to break down the mathematical constraints (sum $S$ and difference $d$) into actionable steps, whereas the higher nodes are primarily restating the prompt.
### Interpretation
This diagram is a visual representation of an LLM (Large Language Model) or a human problem-solver's internal reasoning process.
1. **Prompt Engineering/Chain-of-Thought:** The structure suggests a "Chain of Thought" prompting technique where the model is forced to restate the problem multiple times to ensure it has parsed the constraints correctly before attempting a solution.
2. **Mathematical Context:** The problem itself is a classic number theory/combinatorics problem involving integer partitions. The model is attempting to identify the conditions under which a partition of $S$ into $n$ parts is unique given a fixed range $d$.
3. **Self-Correction:** The presence of phrases like "Okay, let me parse this again" and "Hmm, okay, let me try to unpack this step by step" in the bottom nodes suggests that the system is actively attempting to self-correct or clarify its understanding of the variables $n$, $d$, and $S$ before proceeding to the actual calculation.
</details>
Figure 4: ${B}({G})$ structure from step 1 (see the beginning of Appendix C for additional details).
<details>
<summary>x5.png Details</summary>
### Visual Description
## Diagram: Probability Problem Solving Flowchart
### Overview
The image is a hierarchical flowchart illustrating the step-by-step solution to a probability problem involving combinatorics. The problem asks for the probability of drawing two balls of the same color from a bag containing 3 black balls and 2 red balls. The diagram branches into two distinct, though largely redundant, pedagogical paths to reach the final solution.
### Components/Axes
* **Structure:** The diagram flows from a single root node at the top, branching into two main paths (left and right), which further subdivide into calculation steps.
* **Node Types:**
* **White Boxes:** Contain problem statements, conceptual explanations, and logical steps.
* **Light Yellow/Tan Boxes:** Contain specific mathematical calculations, formulas, and final results.
* **Flow:** Arrows indicate the logical progression from the problem statement to the final calculation.
### Detailed Analysis
#### Root Node (Top)
* **Text:** "In a bag, there are 5 balls of the same size, including 3 black balls and 2 red balls. If two balls are drawn at random, the probability of drawing two balls of the same color is ( ).
A: 1/5
B: 3/10
C: 2/5
D: 1/2"
#### Left Branch
* **Node 1 (White):** "Given a bag with 5 balls of the same size, including 3 black balls and 2 red balls, we need to find the probability of drawing two balls of the same color when two balls are drawn at random. First, we calculate the total number of ways to draw"
* **Node 2 (Yellow):** "two balls from the bag. This is given by the combination formula $\binom{n}{k}$, where n is the total number of balls and k is the number of balls to draw. Here, n = 5 and k = 2: ..."
* **Node 3 (Yellow):** "$\binom{5}{2} = \frac{5!}{2!(5-2)!} = \frac{5 \times 4}{2 \times 1} = 10$. Next, we calculate the number of ways to draw 2 black balls from the 3 available: $\binom{3}{2} = ...$"
* **Node 4 (Yellow):** "$\binom{5}{2} = \frac{5!}{2!(5-2)!} = \frac{5 \times 4}{2 \times 1} = 10$. Next, we calculate the number of ways to draw 2 black balls from the 3 black balls: ..."
* **Node 5 (Yellow - Bottom Left):** "$\frac{\text{ways to draw 2 same color}}{\text{Total number of ways to draw 2 balls}} = \frac{4}{10} = \frac{2}{5}$. Thus, the probability of drawing two balls of the same color is: $\boxed{\frac{2}{5}}$"
#### Right Branch
* **Node 1 (White):** "To determine the probability of drawing two balls of the same color from a bag containing 3 black balls and 2 red balls, we can follow these steps: 1. Calculate the total number of balls: 3 (black) + 2 (red) = 5 balls. 2. Determine the"
* **Node 2 (Yellow):** "total number of ways to draw 2 balls from 5: $\binom{5}{2} = \frac{5!}{2!(5-2)!} = \frac{5 \times 4}{2 \times 1} = 10$. 3. Calculate the number of ways to draw 2 black balls: $\binom{3}{2} = \frac{3!}{2!(3-2)!} = \frac{3 \times 2}{2 \times 1} = 3$"
* **Node 3 (Yellow):** "number of ways to draw 2 balls from 5: $\binom{5}{2} = \frac{5!}{2!(5-2)!} = \frac{5 \times 4}{2 \times 1} = 10$. 3. Calculate the number of ways to draw 2 black balls: $\binom{3}{2} = \frac{3!}{2!(3-2)!} = \frac{3 \times 2}{2 \times 1} = 3$. 4. Calculate the number of ways to draw 2 red balls: $\binom{2}{2} = \frac{2!}{2!(2-2)!} = 1$. 5. Find the total number of ways to draw 2 balls of the same color: 3 (black) + 1 (red) = 4. 6. Calculate the probability: $\text{Probability} = \frac{\text{Number of"
* **Node 4 (Yellow - Bottom Right):** "favorable outcomes}}{\text{Total number of outcomes}} = \frac{4}{10} = \frac{2}{5}$. Thus, the probability of drawing two balls of the same color is: $\boxed{C}$"
### Key Observations
* **Redundancy:** The diagram presents two paths that essentially calculate the same mathematical problem. The right path is more comprehensive, explicitly calculating both black and red ball combinations, whereas the left path is fragmented and repetitive.
* **Formatting Errors:** The text contains several LaTeX rendering artifacts (e.g., `\text{}`, `\binom`) that suggest this diagram was generated by a script or software that did not fully render the mathematical notation.
* **Consistency:** Despite the messy formatting, the mathematical logic is consistent across both branches, arriving at the correct probability of 2/5.
* **Final Answer:** The right branch explicitly maps the result to the multiple-choice option "C".
### Interpretation
The data demonstrates a standard combinatorics approach to probability:
1. **Total Outcomes:** Calculated using the combination formula $\binom{5}{2} = 10$.
2. **Favorable Outcomes:**
* Drawing 2 black balls from 3: $\binom{3}{2} = 3$.
* Drawing 2 red balls from 2: $\binom{2}{2} = 1$.
* Total favorable outcomes: $3 + 1 = 4$.
3. **Probability:** $\frac{\text{Favorable}}{\text{Total}} = \frac{4}{10} = \frac{2}{5}$.
The diagram acts as a pedagogical tool, likely generated by an AI or automated tutoring system, attempting to break down the problem into logical steps. The presence of two branches suggests an attempt to provide alternative explanations or a "show your work" feature, though the execution is visually cluttered and contains rendering errors. The final answer, 2/5, corresponds to option C in the original problem statement.
</details>
Figure 5: ${B}({G})$ structure from step 1001 (see the beginning of Appendix C for additional details).
<details>
<summary>x6.png Details</summary>
### Visual Description
## Flowchart: Mathematical Probability Problem Solving Process
### Overview
The image displays a hierarchical flowchart representing a step-by-step logical derivation for a probability problem. The diagram illustrates an AI-style "Chain of Thought" process, where a mathematical problem is broken down into multiple, sometimes redundant, reasoning paths. The flow moves from the top problem statement down through various branches of logic, ultimately converging on a geometric probability approach.
### Components/Axes
* **Structure:** A top-down tree diagram with branching nodes.
* **Node Types:**
* **Top Node (Red/Pink):** The primary problem statement.
* **Intermediate Nodes (White):** Core logical steps and reasoning.
* **Supporting Nodes (Tan/Yellow):** Explanatory details and context.
* **Connectors:** Black arrows indicating the flow of logic.
### Detailed Analysis
The text contains raw LaTeX markup (e.g., `\n`, `\geq`, `\times`). The transcription below preserves the exact formatting found in the image.
**1. Top Node (Problem Statement)**
* "Let $a$ and $b$ be given positive real numbers, with $a<b$. If two points are selected at random from a straight line segment of length $b$, what is the probability that the distance between them is at least $a$?"
**2. First Branch (Initial Processing)**
* **Center Node:** `\n\nGiven two`
**3. Second Branch (Splitting)**
* **Left Path:** "points selected at random from a straight line segment of length \( b \), we need to find the probability that the distance between them is at least \( a \), where ("
* **Right Path:** "positive real numbers \(a\) and \(b\) such that \(a < b\). We need to find the probability that the distance between two randomly selected points on a"
**4. Third Branch (Further Decomposition)**
* **Leftmost Tan Box:** "a \( b \) are positive real numbers with \( a < b \).\n\nFirst, we recognize that the problem can be modeled using a uniform distribution over the interval \([0, b]\) ..."
* **Middle White Box:** "0 < a < b \).\n\nFirst, we"
* **Middle-Right Tan Box:** "line segment of length \( (b\) is at least \( (a\).\n\nFirst, we note that the total number of possible pairs of points is \( (b^2 \), since each point can be anywhere along the segment of length \( (b\)..."
* **Rightmost Tan Box:** "straight line segment of length \( (b\) is at least \( (a\).\n\nFirst, we note that the total number of possible pairs of points is represented by the number of ways to choose 2 points from a segment of length \( (b\) ..."
**5. Fourth Branch (Deepening Logic)**
* **Left Tan Box (from Middle White Box):** "note that the total number of ways to select two points from a segment of length \( (b \) is represented by the number of possible positions for these points. Since the points are selected continuously, we can model this using a unit square where each axis ..."
* **Center White Box:** "recognize that the problem can be modeled using a uniform distribution over the interval \( ([0, b]). Let the positions of the two points be \( (X \) and \( (Y \), where \( (X \) and \( (Y \) are independent random variables uniformly distributed over \( ([0, b]).\n\nThe distance between the two points is \( |X - Y| \geq a \).\n\nTo solve this, we consider the sample space"
**6. Fifth Branch (Final Conclusion/Setup)**
* **Bottom Left Tan Box:** "\( [0, b] \times [0, b] \), where each point \( ((X, Y) \) represents a possible outcome. The area of this sample space is \( (b^2 \).\n\nWe are interested in the region where \( |X - Y| \geq a \). This can be split into two regions:\n1. ..."
* **Bottom Right Tan Box:** ", which is the set of all possible pairs \( ((X, Y) \). This can be represented as a square in the \( (XY \)-plane with side length \( (b \). The area of this square is \( (b^2 \).\n\nWe are interested in the region where \( |X - Y| \geq a \) ..."
### Key Observations
* **Redundancy:** The diagram is highly repetitive. Multiple branches explore the same concepts (e.g., defining the sample space as a square of side $b$ and area $b^2$) using slightly different phrasing.
* **Formatting:** The text includes raw LaTeX syntax and newline characters (`\n`), indicating this is likely a raw output from an LLM (Large Language Model) or a similar automated reasoning system.
* **Logic Flow:** The diagram effectively demonstrates a "Chain of Thought" process where the system generates multiple potential paths to solve the problem, eventually converging on the standard geometric probability solution (using a square sample space and the condition $|X-Y| \geq a$).
### Interpretation
The data demonstrates a classic geometric probability problem. The problem asks for the probability that the distance between two points chosen uniformly at random on a line segment of length $b$ is at least $a$.
The reasoning process shown in the diagram correctly identifies the standard solution method:
1. **Model:** Treat the two points as independent random variables $X$ and $Y$ on the interval $[0, b]$.
2. **Sample Space:** The sample space is a square of area $b^2$ in the $XY$-plane.
3. **Condition:** The favorable region is defined by $|X - Y| \geq a$.
4. **Geometric Solution:** This region corresponds to the two triangles in the corners of the square (the area of the square minus the central strip where $|X - Y| < a$).
The diagram is an artifact of an AI reasoning process, showing the "exploration" phase where the model generates multiple, overlapping explanations for the same mathematical setup before settling on the final geometric formulation.
</details>
Figure 6: ${B}({G})$ structure from step 1673 (see the beginning of Appendix C for additional details).