# Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning
**Authors**: Xintong Li, Sha Li, Rongmei Lin, Hongye Jin, Linwei Li, Hejie Cui, Sarah Zhang, Chia-Yuan Chang, Kewei Cheng, Besnik Fetahu, Priyanka Nigam, Jingbo Shang, Bing Yin
## Abstract
Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy. Prior reinforcement learning approaches typically rely on a single outcome reward with trajectory-level length penalties, which cannot distinguish essential from redundant reasoning steps and therefore yield blunt compression. Although recent work incorporates step-level signals, such as offline pruning, supervised data construction, or verifier-based intermediate rewards, reasoning length is rarely treated as an explicit step-level optimization objective during RL. We propose S tep- w ise A daptive P enalization (SwAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution. We estimate step importance from the model’s on-policy log-probability improvement toward the correct answer, then treat excess length as a penalty mass redistributed to penalize low-importance steps more heavily while preserving high-importance reasoning. We optimize with a unified outcome–process advantage within group-relative policy optimization. Experiments demonstrate that SwAP reduces reasoning length by 64.3% on average while improving accuracy by 5.7% relative to the base model.
Machine Learning, ICML
## 1 Introduction
Large language models (LLMs) have achieved strong performance in complex reasoning tasks through Chain-of-Thought (CoT) prompting, which encourages the model to decompose a problem into intermediate reasoning steps (Achiam et al., 2023; Guo et al., 2025). While this enables models to navigate complex logical structures, it often leads to overthinking: models generate excessively long reasoning chains dominated by redundant or low-value steps without improving final accuracy (Sui et al., 2025). Such unnecessary verbosity substantially increases inference cost and latency, and can even degrade performance by introducing opportunities for hallucination in later stages of generation (Li et al., 2025c; Liu and Wang, 2025). *[Error downloading image: 2603.00296v1/x1.png]*
Figure 1: Step-level redundancy in reasoning. We segment the model’s responses into steps by token length and use the change in the probability of generating the ground truth answer to measure each step’s progress. (a) Distribution of step-wise information gain: most steps contribute little or no progress toward the correct answer, and high-gain steps are rare (4% with gain $>$ 0.5). (b) Longer reasoning trajectories contain a lower fraction of high-gain steps. (c) In SwAP, in addition to the outcome reward $r_{o}$ , each step also gets assigned a progress reward $r_{s}$ . When the response is correct but exceeds the reference length, the penalty (dashed grey area) gets distributed to each step based on its progress (darker shades represent more progress).
To mitigate overthinking, recent work has increasingly turned to post-training strategies, including supervised preference learning for shorter responses (Muennighoff et al., 2025), distillation from long to short CoT models (Li et al., 2025b), and reinforcement learning (RL) with explicit length constraints (Li et al., 2025a). Among these, RL-based approaches offer a particularly flexible mechanism to trade off accuracy and efficiency. Most existing methods, however, operate at the trajectory level, for example, by imposing global length penalties, fixed token budgets, or difficulty-dependent reasoning modes (Aggarwal and Welleck, 2025; Lyu et al., 2025). Although effective in reducing the overall length of the reasoning, these coarse-grained strategies suffer from a fundamental limitation by treating all the reasoning steps equally valuable.
In practice, reasoning trajectories are highly heterogeneous at the step level (Bogdan et al., 2025). Some steps introduce critical information that decisively pivots the solution toward correctness, while others correspond to redundant verification, reformulation, or low-utility exploration. As a result, trajectory-level penalties often induce indiscriminate compression, risking the removal of essential reasoning while preserving irrelevant text (Yu et al., 2025). Figure 1 empirically demonstrates this imbalance. As shown in (a), the majority of reasoning steps yield negligible or no information gain toward the final answer, with only a small fraction contributing substantial progress. Moreover, (b) shows that longer reasoning trajectories contain a markedly lower density of high-impact steps, indicating that redundancy accumulates as reasoning length increases. Together, these observations indicate that overthinking is not simply a function of how much reasoning is performed, but of where redundant computation accumulates within a trajectory.
These findings highlight two principles that are largely underexplored in existing reward design. First, length regularization should be aware of the difficulty. Harder problems naturally require longer reasoning and should not be penalized identically to easier ones (Xiang et al., 2025; Song and Zheng, 2025). Second, and more critically, length control should be conduct at the step level, rather than applied uniformly across an entire trajectory within a single trajectory, different reasoning steps contribute unequally to correctness and should therefore be treated asymmetrically (Song et al., 2025). While recent work has begun to incorporate step-level information, such as offline pruning, supervised data construction, or verifier-based intermediate rewards, most approaches either use step signals outside of RL or rely on auxiliary models to assess step correctness (Yue et al., 2025; He et al., 2025; Wu et al., 2025). As a result, reasoning length itself is rarely treated as an explicit step-level optimization objective during RL, and existing methods lack a principled mechanism for deciding which steps to shorten and by how much when a trajectory becomes overly long.
In this work, we propose S tep- w ise A daptive P enalization (SwAP), a fine-grained RL framework that allocates length reduction across reasoning steps based on their intrinsic contribution to solving the problem. Rather than relying on external reward models or heuristic estimators, SwAP derives step importance directly from the model’s own behavior. Specifically, we quantify the information gain of each reasoning step by measuring the improvement in the log-probability assigned to the correct answer after that step. Steps that induce large gains are interpreted as carrying substantial reasoning mass and should be preserved, while steps with minimal or no gain are treated as low-impact and preferentially penalized.
To preserve correctness while enforcing efficiency, we implement this algorithm via a stepwise weighted length penalty mechanism. For each prompt, we define a reference length that is based on the difficulty of the correct responses and compute a global penalty when a trajectory exceeds this target. This penalty mass is then redistributed across steps according to their importance, so that low-utility steps absorb most of the penalty while high-impact steps are protected from aggressive compression. This design enables adaptive, fine-grained length control without requiring pre-specified token budgets or external supervision. To ensure correctness while enforcing efficiency, we integrate the stepwise signal with trajectory-level outcome rewards through a unified outcome–process advantage under Group Relative Policy Optimization (GRPO) (Guo et al., 2025). The outcome advantage promotes globally correct solutions, while a backward-propagated process advantage assigns credit to tokens based on the cumulative contribution of future reasoning steps. Together, this unified formulation decomposes learning into complementary roles: outcome rewards gate correctness, while progress-aware step rewards shape where and how reasoning is compressed.
Experiments on five mathematical reasoning benchmarks demonstrate that SwAP achieves a substantially better accuracy–efficiency trade-off than prior methods. For example, on DeepSeek-Distill-Qwen-1.5B, SwAP reduces reasoning length by 64.3% on average while improving accuracy by 5.7% relative to the base model. On the 7B model, SwAP reduces token usage by over 50% while consistently matching or exceeding the strongest baselines on the hardest benchmarks, including AIME24, AIME25, and OlympiadBench. These results indicate that significant overthinking persists even in large reasoning models, and that stepwise optimization provides a principled and effective path toward efficient reasoning.
## 2 Related Work
#### Efficient reasoning in LLMs.
Modern reasoning LLMs often suffer from overthinking, producing unnecessarily long CoT that increase latency and inference cost without improving accuracy (Feng et al., 2025; Sui et al., 2025). Recent work on efficient reasoning spans both supervised fine-tuning and preference-based post-training. For instance, O1-Pruner (Luo et al., 2025a) applies length-harmonized fine-tuning to prune redundant reasoning while preserving performance, while AdaptThink (Zhang et al., 2025) introduces hybrid strategies that dynamically switch between “thinking” and “no-thinking” modes based on problem difficulty. SABER (Zhao et al., 2025) further enables user-controllable, token-budgeted reasoning at inference time. Complementary analyses highlight that inefficiency is not only a matter of how much reasoning is generated, but also where redundancy accumulates. RC-R1 (Cheng et al., 2025) explicitly targets such redundant segments while preserving a sufficient core of reasoning steps. In parallel, recent studies on reasoning distillation and trace selection show that data quality and selection criteria strongly shape distilled reasoning behavior (Li et al., 2025b; Chen et al., 2025), underscoring the importance of fine-grained control over reasoning structure.
#### RL for reducing overthinking.
A prominent line of work leverages RL to trade off accuracy and efficiency through explicit length constraints. Early approaches such as L1 (Aggarwal and Welleck, 2025) demonstrate that RL can enforce strong length control under fixed token budgets. Subsequent methods introduce progressively tighter constraints or iterative pruning strategies to compress long CoT (Hou et al., 2025), adaptive reward shaping and target-length policies (Liu et al., 2025), and accuracy–length control rules or hierarchical budgeting mechanisms (Li et al., 2025a; Lyu et al., 2025). While effective at reducing overall reasoning length, most of these approaches apply penalties uniformly across the trajectory, leading to indiscriminate compression. More recent step-aware RL methods address this limitation by incorporating verifiable intermediate feedback (Wang et al., 2025). VSRM assigns rule-based verifiable rewards to encourage effective steps and penalize ineffective ones (Yue et al., 2025), while SmartThinker combines supervised fine-tuning with step-wise RL to guide reasoning refinement (He et al., 2025). These advances highlight the importance of step-level credit assignment for suppressing overthinking.
## 3 Framework
We introduce a progress-aware stepwise reward framework that redistributes trajectory-level length penalties across reasoning steps based on their marginal contribution to reaching the correct answer, and optimizes this signal jointly with outcome correctness under GRPO. Our approach consists of three stages: estimating intrinsic step importance from on-policy log-probability improvements, allocating length penalties to individual steps according to their importance, and integrating the resulting stepwise signal with the outcome reward through a unified outcome–process advantage estimator. The full algorithm is summarized in Algorithm 1.
### 3.1 Step Reward Measurement
To assign step rewards, we define a robust step segmentation scheme and introduce an intrinsic measure of step importance based on answer log-probability improvement.
#### Step segmentation by token budget.
Unlike prior work that relies on heuristic sentence boundaries or explicit reasoning transition words (e.g., wait, however, therefore) (Besta et al., 2025; Wang et al., 2023; Xu et al., 2025), we define reasoning steps using a simple and robust token-based segmentation scheme.
Given a generated response, we extract the reasoning portion (e.g., content before </think>) and partition it into at most $K$ steps $\{s_{k}\}_{k=1}^{K}$ by grouping approximately a fixed token budget $M$ per step. If the response would exceed $K$ steps, we adapt $M$ upward to ensure a bounded number of steps. We additionally align step boundaries to the next new line to produce cleaner, sentence-level segments. Under this construction, each step corresponds to a contiguous span of tokens and forms a semantically coherent chunk of reasoning.
#### Step reward via answer log-probability gain.
We define step importance intrinsically using the model’s own confidence toward the correct answer. Intuitively, a reasoning step is important if it substantially increases the likelihood of the final answer.
Let $x$ denote the prompt, $a^{\star}$ the ground-truth answer, and $\{s_{1},\ldots,s_{K}\}$ the segmented reasoning steps. For each prefix consisting of the first $k$ steps, we construct the conditional context $x_{k}=(x,s_{1},\ldots,s_{k})$ , and compute the average per-token log-probability of the correct answer,
$$
\ell_{k}=\frac{1}{|a^{\star}|}\log p_{\theta}\!\left(a^{\star}\mid x_{k}\right).
$$
We also compute a baseline score $\ell_{0}$ using the prompt without any reasoning steps. We then define a progress-aware step reward as monotone incremental information gain
$$
\Delta_{k}=\max\Bigl(0,\;\ell_{k}-\max_{j<k}\ell_{j}\Bigr),\qquad k=1,\ldots,K. \tag{1}
$$
This construction attributes positive reward only when a step improves the model’s confidence beyond all previous steps, preventing later redundant steps from accumulating spurious credit. Steps that do not introduce new information receive zero reward. Each $\Delta_{k}$ is assigned to the end of the corresponding step in the token sequence, yielding sparse, step-aligned reward markers that are compatible with on-policy reinforcement learning.
### 3.2 Step-Weighted Length Penalty Redistribution
To discourage overlong reasoning while preserving essential computation, we introduce a step-weighted length penalty that selectively penalizes low-importance reasoning steps when a trajectory exceeds an adaptive length target.
Under GRPO, responses are generated in groups corresponding to the same prompt. For each group, we define a target length $L_{\text{target}}$ as the median length of correct responses within the group, which serves as a difficulty-aware reference. Given a response of length $L$ , if $L\leq L_{\text{target}}$ , no length penalty is applied. Otherwise, we compute a total penalty mass
$$
P=\lambda\cdot\frac{L-L_{\text{target}}}{L_{\text{target}}}, \tag{2}
$$
where $\lambda$ controls the overall strength of length regularization. Rather than applying this penalty uniformly across tokens, we redistribute it according to step importance. Let $g_{k}=\ell_{k}-\ell_{k-1}$ denote the local log-probability gain associated with step $k$ . We assign each step a penalty weight
$$
w_{k}\propto\exp(-g_{k}/\tau),
$$
where $\tau$ is a temperature parameter. This construction assigns larger weights to steps with smaller gains, while protecting high-impact steps from aggressive penalization. The weights are normalized such that $\sum_{k=1}^{K}w_{k}=1$ , and each step $k$ receives a penalty of $-P\cdot w_{k}$ .
Finally, we combine intrinsic progress reward and length penalty into a single step-level reward,
$$
r_{k}=\Delta_{k}-P\cdot w_{k}, \tag{3}
$$
where $\Delta_{k}$ is the monotone information gain defined in Eq. 1. This step-weighted redistribution mechanism ensures that excess length is preferentially removed from low-utility reasoning segments, enabling fine-grained and adaptive reasoning compression without indiscriminately shortening the entire trajectory.
| Method | MATH-500 | AMC23 | AIME24 | AIME25 | OlympiadBench | Avg. | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc $\uparrow$ | Len $\downarrow$ | Acc $\uparrow$ | Len $\downarrow$ | Acc $\uparrow$ | Len $\downarrow$ | Acc $\uparrow$ | Len $\downarrow$ | Acc $\uparrow$ | Len $\downarrow$ | Acc $\uparrow$ | Len $\downarrow$ | |
| DeepSeek-Distill-Qwen-1.5B | | | | | | | | | | | | |
| Original | 83.4 | 4777 | 70.0 | 7722 | 26.7 | 12571 | 23.5 | 12247 | 35.1 | 11557 | 47.7 | 9775 |
| ThinkPrune | 83.1 | 2332 | 72.5 | 3791 | 24.7 | 7469 | 20.4 | 7024 | 37.8 | 5634 | 47.7 | 5250 |
| LC-R1 | 81.9 | 2249 | 68.2 | 3607 | 23.3 | 7061 | 18.7 | 6647 | 33.3 | 5199 | 45.1 | 4953 |
| AdaptThink | 83.9 | 1863 | 70.0 | 3628 | 31.3 | 7236 | 22.3 | 7195 | 38.1 | 4900 | 49.1 | 4964 |
| Laser-D | 85.2 | 2589 | 74.2 | 4227 | 30.8 | 7929 | 22.1 | 7387 | 41.6 | 6498 | 50.8 | 5726 |
| Laser-DE | 85.1 | 2821 | 75.1 | 4503 | 31.0 | 8330 | 21.2 | 7954 | 42.0 | 6819 | 50.9 | 6085 |
| SwAP (ours) | 86.5 | 1597 | 78.3 | 2645 | 33.9 | 4915 | 23.3 | 4627 | 45.1 | 3646 | 53.4 | 3486 |
| DeepSeek-Distill-Qwen-7B | | | | | | | | | | | | |
| Original | 93.0 | 3743 | 84.6 | 5794 | 52.3 | 10579 | 37.5 | 11076 | 53.6 | 10222 | 64.2 | 8283 |
| L1-Exact | 91.9 | 2186 | 81.1 | 2589 | 43.5 | 4026 | 28.7 | 4042 | 49.6 | 3438 | 59.0 | 3256 |
| L1-Max | 92.0 | 2134 | 79.5 | 2594 | 44.5 | 4139 | 31.4 | 4093 | 47.5 | 3392 | 59.0 | 3270 |
| LC-R1 | 89.9 | 1568 | 81.6 | 2922 | 49.5 | 6972 | 36.2 | 7657 | 52.4 | 5678 | 61.9 | 4959 |
| AdaptThink | 91.8 | 1825 | 80.0 | 3835 | 52.9 | 8600 | 37.5 | 9548 | 54.5 | 7856 | 63.3 | 6333 |
| Laser-D | 93.5 | 1961 | 87.0 | 2925 | 55.6 | 6108 | 37.9 | 6629 | 59.7 | 5244 | 66.7 | 4573 |
| Laser-DE | 93.6 | 1898 | 85.0 | 2901 | 53.5 | 5968 | 37.9 | 6161 | 58.3 | 5111 | 65.7 | 4408 |
| SwAP (ours) | 93.6 | 1573 | 85.6 | 2685 | 56.5 | 5744 | 38.1 | 5598 | 60.3 | 4760 | 66.8 | 4072 |
Table 1: Performance and reasoning length on mathematical reasoning benchmarks. Avg. denotes the average pass@1 across all datasets.
### 3.3 Unified Outcome–Process Advantage
We integrate stepwise rewards into GRPO by constructing a unified token-level advantage that combines standard group-relative outcome supervision with a backward-propagated process signal derived from step rewards. This formulation preserves the stability of outcome-based GRPO while enabling fine-grained efficiency optimization at the reasoning-step level.
#### Group-relative outcome advantage.
For each query $x$ , we samples a group of $N$ responses $\{y_{i}\}_{i=1}^{N}$ . Each response receives a sparse outcome reward $r^{\text{out}}_{i}\in\mathbb{R}$ , defined as positive if the final answer is correct and zero otherwise. Following the standard GRPO, outcome rewards are normalized within the group using mean $\mu_{\text{out}}(x)$ and standard deviation $\sigma_{\text{out}}(x)$ , resulting in the outcome advantage
$$
A^{\text{out}}_{i}=\frac{r^{\text{out}}_{i}-\mu_{\text{out}}(x)}{\sigma_{\text{out}}(x)+\varepsilon}. \tag{4}
$$
#### Step-level rewards.
Each response $y_{i}$ is segmented into $K_{i}$ reasoning steps, with step rewards $\{r_{i,k}\}_{k=1}^{K_{i}}$ defined in Eq. (3) and placed at the end token of each step. To stabilize training, step rewards are normalized within each group using statistics computed over correct trajectories only,
$$
\tilde{r}_{i,k}=\frac{r_{i,k}-\mu_{\text{step}}(x)}{\sigma_{\text{step}}(x)+\varepsilon}, \tag{5}
$$
where $\mu_{\text{step}}(x)$ and $\sigma_{\text{step}}(x)$ denote the mean and standard deviation of step rewards across correct responses for the prompt $x$ .
#### Backward-propagated process advantage.
Let $e_{i,k}$ denote the token index corresponding to the end of step $k$ in response $y_{i}$ . We define a token-level process advantage by accumulating normalized step rewards from future reasoning steps,
$$
A^{\text{proc}}_{i,t}=\sum_{k:\;t\leq e_{i,k}}\tilde{r}_{i,k}. \tag{6}
$$
This reverse cumulative construction assigns each token credit proportional to the cumulative contribution of future reasoning steps, allowing step-level efficiency signals to influence earlier generation decisions.
Algorithm 1 SwAP
0: Policy $\pi_{\theta}$ , dataset $\mathcal{D}$ , group size $N$ , weights $(\beta,\theta)$
1: for $t=1,\ldots,T$ do
2: Sample prompts $x\sim\mathcal{D}$ and rollouts $\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)$
3: for each rollout $y_{i}$ do
4: Segment into steps $\{s_{i,k}\}_{k=1}^{K_{i}}$ via token budget $M$
5: Calculate step information gain $\Delta_{i,k}$ (Eq. 1)
6: Penalty redistribution: compute penalty mass $P_{i}$ and weight $w_{i,k}\propto\exp(-g_{i,k}/\tau)$
7: Step reward $r_{i,k}=\Delta_{i,k}-P_{i}w_{i,k}$ (Eq. 3)
8: end for
9: Outcome advantage $A_{i}^{\mathrm{out}}=\dfrac{r_{i}^{\mathrm{out}}-\mu_{\mathrm{out}}(x)}{\sigma_{\mathrm{out}}(x)+\varepsilon}$
10: Normalize step rewards on correct rollouts: $\tilde{r}_{i,k}$ (Eq. 5)
11: Process advantage $A_{i,t}^{\mathrm{proc}}$ (Eq. 6)
12: Unified advantage $A_{i,t}=\beta A_{i}^{\mathrm{out}}+\theta\,\mathbb{I}[r_{i}^{\mathrm{out}}>0]A_{i,t}^{\mathrm{proc}}$
13: Update $\theta$ with clipped GRPO objective using $A_{i,t}$ (Eq. 8)
14: end for
#### Unified advantage.
Finally, we combine outcome and process advantages into a single token-level signal,
$$
A_{i,t}=\beta\,A^{\text{out}}_{i}\;+\;\theta\,\mathbb{I}\!\left[r^{\text{out}}_{i}>0\right]\cdot A^{\text{proc}}_{i,t}, \tag{7}
$$
where $\beta,\theta>0$ control the trade-off between global correctness and local efficiency. Importantly, the process term is gated by correctness, ensuring that step-level signals influence optimization only for correct trajectories and preventing noisy step rewards from corrupting learning on incorrect samples.
Let $\pi_{\theta_{\text{old}}}$ denote the behavior policy used to generate rollouts. We optimize the standard clipped GRPO objective using the unified advantage:
$$
\displaystyle\mathcal{L}_{\text{GRPO}}(\theta)=- \displaystyle\mathbb{E}_{x,i,t}\Big[\min\big(r_{i,t}(\theta)A_{i,t}, \displaystyle\operatorname{clip}(r_{i,t}(\theta),1-\epsilon,1+\epsilon)A_{i,t}\big)\Big] \tag{8}
$$
where $r_{i,t}(\theta)=\frac{\pi_{\theta}\!\left(y_{i,t}\mid x,\,y_{i,<t}\right)}{\pi_{\theta_{\text{old}}}\!\left(y_{i,t}\mid x,\,y_{i,<t}\right)}$ is the token-wise importance ratio.
Algorithm 1 summarizes the overall SWAP training procedure, including step importance estimation, penalty redistribution, and unified advantage construction under GRPO.
## 4 Experiments
### 4.1 Experimental Setup
| Original | 83.4 | 4777 | 70.0 | 7722 | 26.7 | 12571 | 23.5 | 12247 | 35.1 | 11557 | 47.7 | 9775 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Outcome-Only | 79.5 | 1534 | 70.6 | 2953 | 29.8 | 6287 | 21.8 | 6048 | 41.3 | 4433 | 48.6 | 4251 |
| Step-Only | 79.9 | 1526 | 67.8 | 2915 | 27.1 | 6019 | 20.8 | 5872 | 31.5 | 3689 | 45.4 | 4004 |
| No-Penalty | 85.7 | 2594 | 77.1 | 4169 | 32.5 | 8027 | 24.3 | 7602 | 42.7 | 6237 | 52.5 | 5726 |
| Static-Penalty | 83.7 | 1276 | 72.8 | 1899 | 25.2 | 3264 | 20.6 | 2702 | 43.0 | 2110 | 49.1 | 2250 |
| Uniform-Penalty | 86.4 | 2064 | 77.8 | 3469 | 32.9 | 6568 | 25.4 | 6163 | 44.5 | 5118 | 53.4 | 4676 |
| SwAP (ours) | 86.5 | 1597 | 78.3 | 2645 | 33.9 | 4915 | 23.3 | 4627 | 45.1 | 3646 | 53.4 | 3486 |
Table 2: Performance and reasoning length of different model variants. All results are based on the DeepSeek-Distill-Qwen-1.5B model. AVG denotes the average pass@1 across all datasets.
#### Models and Training Data.
We evaluate SwAP on two widely used reasoning models, DeepSeek-R1-Distill-Qwen-1.5B and DeepSeek-R1-Distill-Qwen-7B (DeepSeek-AI, 2025). Both models demonstrate strong mathematical reasoning capability but frequently exhibit overthinking behavior, producing unnecessarily long chains-of-thought. For training, we use the DeepScaleR-Preview dataset (Luo et al., 2025b), which contains 40K high-quality mathematical reasoning problems collected from AIME, AMC, Omni-Math (Gao et al., 2024), and STILL (Min et al., 2024). This dataset provides diverse difficulty levels and supports studying both reasoning accuracy and efficiency.
#### Baselines.
We compare SwAP against a diverse set of state-of-the-art methods for efficient reasoning, covering three representative categories. Length-penalty-based RL methods include ThinkPrune (Hou et al., 2025), which progressively enforces stricter token limits during RL to prune redundant reasoning; LC-R1 (Cheng et al., 2025), which combines GRPO with global length and compression rewards to remove invalid post-solution reasoning; and two variants of the LASER (Liu et al., 2025) framework, LASER-D and LASER-DE, which apply dynamic and difficulty-aware length reward shaping, with LASER-DE additionally encouraging exploration on incorrect trajectories. Adaptive thinking-mode methods include AdaptThink (Zhang et al., 2025), which trains models to dynamically select between thinking and no-thinking modes based on problem difficulty. Length-control methods include L1-Exact and L1-Max (Aggarwal and Welleck, 2025), which optimize reasoning accuracy under prompt-specified exact or maximum length constraints via length-controlled policy optimization.
#### Evaluation Benchmarks.
We evaluate all methods on five mathematical reasoning benchmarks spanning a wide range of difficulty levels: MATH-500 (Hendrycks et al., 2021), AMC23 (MAA, n.d.b), AIME24, AIME25 (MAA, n.d.a), and OlympiadBench (He et al., 2024). Evaluation is performed with temperature $T=0.6$ , top- $p=0.95$ , and a maximum generation length of 32,768 tokens. For each benchmark, we generate 16 samples per query and report both pass@1 accuracy and the average response length in tokens, capturing the trade-off between correctness and reasoning efficiency.
#### Implementation Details.
We implement SwAP using the VeRL framework (Sheng et al., 2025). During training, we use a batch size of 128 and generate 16 rollouts per prompt with a maximum response length of 12,000 tokens and sampling temperature 0.6. Training is conducted for two epochs for both model sizes, resulting in a total of 623 steps.
For step segmentation, we allocate approximately 350 tokens per step and cap the maximum number of reasoning steps at 25. We set the length penalty coefficient $\lambda=1$ to apply the full penalty mass when responses exceed the target length. The outcome advantage weight $\beta$ is set to 1, while the process advantage weight $\theta$ is set to 0.3 to avoid overly aggressive early length reduction. Training of the 1.5B model is conducted on a single node with $8\times$ H100 GPUs, whereas the 7B model is trained on a single node with $8\times$ H200 GPUs.
### 4.2 Main results
*[Error downloading image: 2603.00296v1/x2.png]*
Figure 2: Comparison of Pass@1 accuracy vs. token usage across math benchmarks. SwAP establishes the Pareto frontier for best performance under every token budget.
Table 1 reports results on five mathematical reasoning benchmarks for both model scales. Across all settings, SwAP significantly reduces reasoning length while simultaneously improving accuracy, achieving the strongest accuracy–efficiency trade-off among all methods.
Relative to the base models, SwAP substantially compresses reasoning traces without sacrificing correctness. For the 1.5B model, SwAP reduces average response length by 64.3% while improving average accuracy by 5.7%, with consistent gains across all benchmarks and particularly large improvements on harder datasets such as AIME24 and OlympiadBench. A similar trend holds for the 7B model, where SwAP reduces reasoning length by 50.8% on average and improves accuracy on all benchmarks, achieving the strongest results on AIME24, AIME25, and OlympiadBench. These results suggest that substantial overthinking is present even among larger reasoning models, especially on difficult problems, and that progress-aware stepwise length reduction can effectively remove redundant computation.
Compared with efficient reasoning baselines, SwAP consistently achieves a more favorable accuracy–efficiency balance. Methods such as ThinkPrune and LC-R1 achieve notable length reductions but often degrade accuracy, especially on harder benchmarks, suggesting that trajectory-level penalties may indiscriminately remove essential reasoning steps alongside redundant ones. For the 7B model, L1-Exact and L1-Max enforce strict token budgets, reducing token usage by around 60% but decreasing average accuracy by 5.2%, with particularly large losses on challenging datasets. Among all baselines, LASER-D and LASER-DE are the most competitive. However, SwAP consistently matches or exceeds their accuracy while using substantially fewer tokens. For example, on the 1.5B model, SwAP improves accuracy on AIME24 and OlympiadBench while reducing reasoning length by 35–45% relative to LASER variants. Similar trends are observed for the 7B model, where SwAP achieves the best or tied-best accuracy on the hardest benchmarks while producing among the shortest reasoning traces overall. *[Error downloading image: 2603.00296v1/x3.png]*
Figure 3: Impact of step advantage weight $\theta$ on model performance and response length. A moderate value of $\theta\in[0.2,0.4]$ achieves the best balance. Hard datasets such as AIME 2024/2025 are more sensitive to a large $\theta$ value.
### 4.3 Reward Components and Penalty Strategies
We study the contribution of each component in our framework using the DeepSeek-Distill-1.5B model. As shown in Table 2, we compare our unified approach against variants that isolate specific reward signals and penalty mechanisms.
Necessity of Unified Advantages. We first evaluate the performance of using either outcome or step signals in isolation. The Outcome-Only variant, which applies a trajectory-level length penalty to the final reward, successfully reduces reasoning length by over 56% on average. However, it suffers from a significant drop in accuracy (e.g., -3.9% on MATH-500), indicating that global signals induce indiscriminate compression that inadvertently removes steps essential for correctness. Conversely, the Step-Only variant, relying solely on intrinsic progress, is the only method that performs worse than the baseline model (-2.3%). This highlights that while stepwise rewards provide fine-grained signals, they lack the global grounding necessary to preserve overall logical integrity. These results confirm that the synergy between global outcome advantages and local process advantages is critical for balanced optimization.
Length Penalty Design and Distribution. The absence of an explicit penalty (No-Penalty) results in the longest reasoning trajectories among all RL variants. While this setting achieves strong accuracy, its average length remains 64% higher than SwAP, demonstrating that intrinsic step rewards alone are insufficient to suppress redundancy without explicit cost pressure. Static-Penalty applies a uniform per-step penalty, while Uniform-Penalty distributes the total penalty mass evenly across steps. Both reduce reasoning length more aggressively than No-Penalty; however, Static-Penalty severely degrades performance on challenging benchmarks such as AIME, where uniform discouragement likely suppresses the depth of reasoning required for complex problem solving. Uniform-Penalty achieves better accuracy, matching SwAP ’s average performance (53.4%), but remains substantially less efficient. In comparison, SwAP reduces the average reasoning length from 4,676 to 3,486 tokens while preserving the same accuracy. By dynamically reallocating penalties toward low-importance steps, SwAP selectively prunes redundant explorations while preserving the critical logical pivots necessary for correct solutions.
## 5 Analysis
### 5.1 Performance under Different Budgets
Figure 2 presents the trade-off between Pass@1 accuracy and token usage on three mathematical reasoning benchmarks, comparing SwAP with state-of-the-art length-control methods. Across all datasets and inference budgets, SwAP consistently achieves superior accuracy under comparable or smaller reasoning budgets.
On MATH-500, SwAP maintains strong performance even under aggressive compression. Specifically, it achieves approximately 87.5% accuracy with around 3.3k tokens, and still preserves over 86% accuracy when the budget is reduced to roughly 1.6k tokens. In contrast, baseline methods such as ThinkPrune and AdaptThink drop to around 84–85% accuracy and exhibit pronounced degradation once the budget falls below 2.5k tokens. This gap suggests that SwAP more effectively suppresses low-utility reasoning steps while preserving logical integrity. Under tighter budgets (below 3k tokens), baseline methods experience sharp accuracy declines, while SwAP remains substantially more robust. On the more challenging AIME25 benchmark, SwAP continues to dominate the trade-off frontier. At roughly 5.8k tokens, SwAP attains about 26.2% accuracy, outperforming all baselines operating at similar or larger token budgets. Even when reduced to approximately 4.6k tokens, SwAP maintains over 23% accuracy, whereas other methods at comparable budgets remain below this level or require significantly longer responses. *[Error downloading image: 2603.00296v1/x4.png]*
Figure 4: Training dynamics of different model ablations. SwAP achieves the best validation performance, with a final average response length similar to Outcome-Only (length penalty assigned at the trajectory level).
### 5.2 Effect of Step Advantage Coefficient $\theta$
We analyze the effect of the step advantage weight $\theta$ on accuracy and reasoning length in Figure 3. $\theta$ controls the strength of step-level process supervision, with larger values imposing stronger penalties on low-importance reasoning steps. Across all benchmarks, increasing $\theta$ leads to a monotonic reduction in response length. For small values of $\theta$ ( $\leq 0.2$ ), SwAP preserves high accuracy while substantially reducing reasoning length, exhibiting a favorable efficiency–accuracy trade-off. For example, on MATH-500, accuracy remains above 87.5% as the average length decreases from 3,374 tokens at $\theta=0.05$ to 1,868 tokens at $\theta=0.2$ . As $\theta$ increases further, we observe a transition regime around $\theta\approx 0.4$ – $0.5$ , where accuracy begins to degrade more noticeably.
This effect is more pronounced on harder benchmarks such as AIME24 and AIME25, where accuracy declines sharply beyond $\theta\approx 0.4$ , indicating that overly strong step penalization starts to remove essential reasoning steps. For large values of $\theta$ ( $\theta\geq 0.7$ ), response length saturates across all datasets. For instance, on AIME25, increasing $\theta$ from 0.7 to 1.0 yields only marginal length reduction (from 1,455 to 1,318 tokens), while accuracy remains low and relatively stable. These results indicate that moderate values of $\theta$ (approximately 0.2–0.4) achieve the best balance between efficiency and accuracy. In contrast, overly large $\theta$ leads to diminishing returns in length reduction and sharp accuracy degradation. This behavior confirms that process advantages must be carefully weighted to suppress redundant reasoning without eliminating necessary intermediate supervision.
### 5.3 Training Dynamics
To study how different reward and penalty designs influence optimization, we analyze the training dynamics of response length and validation accuracy shown in Figure 4. We compare standard GRPO, Outcome-Only (outcome reward with a global length penalty), No-Penalty (combined outcome and step rewards without length penalties), and SwAP.
Standard GRPO reduces response length slowly and converges to long reasoning traces (around 5,500 tokens), with relatively low validation accuracy. Adding a global length penalty accelerates early length reduction; however, its accuracy remains noticeably lower, indicating that trajectory-level penalties alone encourage indiscriminate compression without improving reasoning quality. In contrast, SwAP steadily reduces response length throughout training while simultaneously achieving the highest validation accuracy. Compared to Outcome-Only and GRPO, our model converges to substantially shorter responses with better accuracy, and compared to No-Penalty, it achieves similar or higher accuracy with significantly fewer tokens. These dynamics demonstrate that combining outcome grounding with progress-aware, step-weighted penalties enables both efficient compression and stable performance improvement during training.
### 5.4 Additional Analysis
We provide further analysis in Appendix A to examine the robustness of our stepwise reward design. We study the sensitivity to step segmentation granularity in Figure 5. Extremely small step sizes lead to over-fragmentation and distorted importance estimation, while overly large step sizes weaken resolution and reduce length control. A moderate segmentation budget achieves the best accuracy–efficiency trade-off. In addition, Figure 6 analyze the distribution of step-wise information gain before and after training. We observe a clear shift toward positive gain after optimization, indicating that low-utility reasoning steps are selectively suppressed while high-impact transitions are preserved. Detailed experimental results and additional discussions are provided in Appendix.
## 6 Conclusion
We presented SwAP, a step-wise length penalization framework that addresses overthinking in large reasoning models through fine-grained reinforcement learning. By measuring each reasoning step’s intrinsic contribution via on-policy information gain and redistributing length penalties accordingly, SwAP enables selective compression that preserves critical reasoning while eliminating redundancy. Integrated with a unified outcome–process advantage under GRPO, our approach balances correctness and efficiency without relying on external reward models or heuristic budgets. Extensive experiments demonstrate that step-level optimization yields a substantially improved accuracy–efficiency trade-off across mathematical benchmarks. Our findings suggest that overthinking is fundamentally a step-level phenomenon, and that stepwise credit assignment is a principled direction for efficient reasoning in future large-scale models.
## Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
## References
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1.
- P. Aggarwal and S. Welleck (2025) L1: controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697. Cited by: §1, §2, §4.1.
- M. Besta, J. Barth, E. Schreiber, A. Kubicek, A. Catarino, R. Gerstenberger, P. Nyczyk, P. Iff, Y. Li, S. Houliston, et al. (2025) Reasoning language models: a blueprint. arXiv preprint arXiv:2501.11223. Cited by: §3.1.
- P. C. Bogdan, U. Macar, N. Nanda, and A. Conmy (2025) Thought anchors: which llm reasoning steps matter?. arXiv preprint arXiv:2506.19143. Cited by: §1.
- X. Chen, Z. Sun, G. Wenjin, M. Zhang, Y. Chen, Y. Sun, H. Su, Y. Pan, D. Klakow, W. Li, et al. (2025) Unveiling the key factors for distilling chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 15094–15119. Cited by: §2.
- Z. Cheng, D. Chen, M. Fu, and T. Zhou (2025) Optimizing length compression in large reasoning models. arXiv preprint arXiv:2506.14755. Cited by: §2, §4.1.
- DeepSeek-AI (2025) DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. External Links: 2501.12948, Link Cited by: §4.1.
- S. Feng, G. Fang, X. Ma, and X. Wang (2025) Efficient reasoning models: a survey. arXiv preprint arXiv:2504.10903. Cited by: §2.
- B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, et al. (2024) Omni-math: a universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985. Cited by: §4.1.
- D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1, §1.
- C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun (2024) OlympiadBench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. External Links: 2402.14008 Cited by: §4.1.
- X. He, X. Ling, and J. Liu (2025) Smartthinker: learning to compress and preserve reasoning by step-level length control. arXiv preprint arXiv:2507.04348. Cited by: §1, §2.
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021) Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: §4.1.
- B. Hou, Y. Zhang, J. Ji, Y. Liu, K. Qian, J. Andreas, and S. Chang (2025) Thinkprune: pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296. Cited by: §2, §4.1.
- R. Li, Z. Luo, Q. Zhang, R. Li, B. Zhou, A. Payani, and X. Du (2025a) AALC: large language model efficient reasoning via adaptive accuracy-length control. arXiv preprint arXiv:2506.20160. Cited by: §1, §2.
- Y. Li, Y. Emad, K. Padthe, J. Lanchantin, W. Yuan, T. Nguyen, J. Weston, S. Li, D. Wang, I. Kulikov, et al. (2025b) NaturalThoughts: selecting and distilling reasoning traces for general reasoning tasks. arXiv preprint arXiv:2507.01921. Cited by: §1, §2.
- Z. Li, J. Zhong, Z. Zheng, X. Wen, Z. Xu, Y. Cheng, F. Zhang, and Q. Xu (2025c) Compressing chain-of-thought in llms via step entropy. arXiv preprint arXiv:2508.03346. Cited by: §1.
- W. Liu, R. Zhou, Y. Deng, Y. Huang, J. Liu, Y. Deng, Y. Zhang, and J. He (2025) Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint arXiv:2505.15612. Cited by: §2, §4.1.
- X. Liu and L. Wang (2025) Answer convergence as a signal for early stopping in reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 17907–17918. Cited by: §1.
- H. Luo, L. Shen, H. He, Y. Wang, S. Liu, W. Li, N. Tan, X. Cao, and D. Tao (2025a) O1-pruner: length-harmonizing fine-tuning for o1-like reasoning pruning. arXiv preprint arXiv:2501.12570. Cited by: §2.
- M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, T. Zhang, L. E. Li, et al. (2025b) Deepscaler: surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog. Cited by: §4.1.
- S. Lyu, L. Wu, Y. Yan, X. Wu, H. Li, Y. Shen, P. Jiang, W. Lu, J. Xiao, and Y. Zhuang (2025) Hierarchical budget policy optimization for adaptive reasoning. arXiv preprint arXiv:2507.15844. Cited by: §1, §2.
- MAA (n.d.a) American invitational mathematics examination (AIME). Mathematical Association of America (MAA). Note: Mathematics Competition Series External Links: Link Cited by: §4.1.
- MAA (n.d.b) American mathematics competitions (AMC 10/12). Mathematical Association of America (MAA). Note: Mathematics Competition Series External Links: Link Cited by: §4.1.
- Y. Min, Z. Chen, J. Jiang, J. Chen, J. Deng, Y. Hu, Y. Tang, J. Wang, X. Cheng, H. Song, et al. (2024) Imitate, explore, and self-improve: a reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413. Cited by: §4.1.
- N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. B. Hashimoto (2025) S1: simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20286–20332. Cited by: §1.
- G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279–1297. Cited by: §4.1.
- F. Song, S. Wei, B. Gao, Y. Wang, W. Luo, W. Li, L. Yao, W. Xiong, L. Chen, T. Liu, et al. (2025) Mitigating overthinking through reasoning shaping. arXiv preprint arXiv:2510.09535. Cited by: §1.
- M. Song and M. Zheng (2025) Walk before you run! concise llm reasoning via reinforcement learning. arXiv preprint arXiv:2505.21178. Cited by: §1.
- Y. Sui, Y. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, N. Zou, et al. (2025) Stop overthinking: a survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419. Cited by: §1, §2.
- X. Wang, L. Caccia, O. Ostapenko, X. Yuan, W. Y. Wang, and A. Sordoni (2023) Guiding language model reasoning with planning tokens. arXiv preprint arXiv:2310.05707. Cited by: §3.1.
- Z. Wang, Z. Gao, and G. Ke (2025) Masked-and-reordered self-supervision for reinforcement learning from verifiable rewards. arXiv preprint arXiv:2511.17473. Cited by: §2.
- J. Wu, Y. Xiong, X. Li, Z. Hu, T. Yu, R. Wang, X. Chen, J. Shang, and J. McAuley (2025) Ctrls: chain-of-thought reasoning via latent state-transition. arXiv preprint arXiv:2507.08182. Cited by: §1.
- V. Xiang, C. Blagden, R. Rafailov, N. Lile, S. Truong, C. Finn, and N. Haber (2025) Just enough thinking: efficient reasoning with adaptive length penalties reinforcement learning. arXiv preprint arXiv:2506.05256. Cited by: §1.
- Y. Xu, X. Guo, Z. Zeng, and C. Miao (2025) Softcot: soft chain-of-thought for efficient reasoning with llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23336–23351. Cited by: §3.1.
- S. Yu, Y. Xiong, J. Wu, X. Li, T. Yu, X. Chen, R. Sinha, J. Shang, and J. McAuley (2025) Explainable chain-of-thought reasoning: an empirical analysis on state-aware reasoning dynamics. arXiv preprint arXiv:2509.00190. Cited by: §1.
- C. Yue, C. Dong, Y. Gao, H. He, J. Chai, G. Yin, and W. Lin (2025) Promoting efficient reasoning with verifiable stepwise reward, 2025. URL https://arxiv. org/abs/2508.10293. Cited by: §1, §2.
- J. Zhang, N. Lin, L. Hou, L. Feng, and J. Li (2025) Adaptthink: reasoning models can learn when to think. arXiv preprint arXiv:2505.13417. Cited by: §2, §4.1.
- K. Zhao, Y. Zhao, J. Song, S. He, L. Zhang, Q. Zhang, and T. Li (2025) Saber: switchable and balanced training for efficient llm reasoning. arXiv preprint arXiv:2508.10026. Cited by: §2.
## Appendix A Appendix
*[Error downloading image: 2603.00296v1/x5.png]*
Figure 5: Comparison of Pass@1 accuracy vs. step length across four math benchmarks.
### A.1 Effect of Step Size on Step Importance Estimation
We analyze the impact of step size $M$ in token-based step segmentation, which determines the granularity of step importance estimation. Results are shown in Figure 5. When the step size is too small, the response is split into many short steps, leading to excessive penalization of low-gain segments. For example, with $M=150$ , accuracy on MATH-500 drops to 83.6% despite a short average length of 1,353 tokens, compared to 86.5% at $M=350$ . Similar degradation is observed on AMC23 and AIME24, indicating that overly fine-grained steps distort step importance estimation. Conversely, overly large step sizes reduce the resolution of step rewards and weaken length control. On AIME24, increasing $M$ from 350 to 650 increases reasoning length from 4,915 to 7,658 tokens while accuracy decreases from 33.9% to 29.0%. Across all benchmarks, $M=350$ achieves the best accuracy–efficiency trade-off, motivating our default choice.
### A.2 Distribution of Step-wise Information Gain
*[Error downloading image: 2603.00296v1/x6.png]*
Figure 6: Distribution of step-wise information gain for the original model (red) and SwAP (blue), computed from 50k randomly sampled reasoning steps. After training, SwAP exhibits a clear shift toward positive information gain, indicating that low-importance reasoning steps are suppressed while informative steps are preserved.
Figure 6 compares the distribution of step-wise information gain between the original model and SwAP, computed from 50k randomly sampled reasoning steps. After training with stepwise reward penalization, the distribution shifts noticeably toward positive information gain. In contrast to the original model, where a large fraction of steps exhibit near-zero or negative gain, SwAP concentrates mass on steps that contribute meaningful progress toward the final answer. This indicates that SwAP effectively suppresses low-utility or redundant reasoning steps, preserving primarily high-impact reasoning transitions.