2601.03559v1

# DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs Abstract Chain-of-Thought (CoT) reasoning improves multi-step mathematical problem solving in large language models but remains vulnerable to exposure bias and error accumulation, as early mistakes propagate irreversibly through autoregressive decoding. In this work, we propose DiffCoT, a diffusion-styled CoT framework that reformulates CoT reasoning as an iterative denoising process. DiffCoT integrates diffusion principles at the reasoning-step level via a sliding-window mechanism, enabling unified generation and retrospective correction of intermediate steps while preserving token-level autoregression. To maintain causal consistency, we further introduce a causal diffusion noise schedule that respects the temporal structure of reasoning chains. Extensive experiments on three multi-step CoT reasoning benchmarks across diverse model backbones demonstrate that DiffCoT consistently outperforms existing CoT preference optimization methods, yielding improved robustness and error-correction capability in CoT reasoning. DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs Shidong Cao 1, Hongzhan Lin 1, Yuxuan Gu 2, Ziyang Luo 1, Jing Ma 1 1 Hong Kong Baptist University 2 Harbin Institute of Technology {cssdcao,cshzlin,majing}@comp.hkbu.edu.hk 1 Introduction Large Language Models (LLMs) (Brown et al., 2020; Achiam et al., 2023; Guo et al., 2025) have marked a major breakthrough in natural language processing, achieving competitive performance across a wide range of mathematical reasoning tasks. A widely adopted technique in LLMs, Chain-of-Thought (CoT) reasoning (Wei et al., 2022; Kojima et al., 2022), enhances mathematical problem-solving by decomposing tasks into step-by-step intermediate reasoning. However, CoT reasoning is highly sensitive to potential errors, where mistakes introduced in the early stages can propagate through later steps, often leading to incorrect final answers (Wang et al., 2023; Lyu et al., 2023). This highlights a central research challenge: reducing errors by guiding LLMs to identify and follow reliable reasoning paths in long thought chains. Recent work has sought to optimize the exploration of reasoning branches in LLMs to reduce errors in CoT, by coupling generation with external critics or search procedures, such as Monte Carlo Tree Search (MCTS), to prune faulty trajectories (Li et al., 2024; Qin et al., 2024; Xi et al., 2024). In contrast, Preference Optimization (PO) methods (Lai et al., 2024; Zhang et al., 2024; Xu et al., 2025) aligned the policy with implicit preferences distilled from reasoning chains, so that decoding itself favors correct trajectories rather than relying on post-hoc filtering. However, beyond the substantial inference-time overhead of MCTS, these methods share a fundamental limitation that motivates rethinking of the CoT paradigm. Specifically, they all execute CoT reasoning in a strictly step-by-step manner, with each step conditioned on previous ones. Under static teacher-forcing supervision, the model is exposed only to correct prefixes during training, whereas at inference time the prefixes may contain errors that accumulate across steps, resulting in exposure bias that commonly plagues autoregressive generation (Bengio et al., 2015) and imitation learning (Ross et al., 2011). <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Existing Methods for CoT vs. DiffCoT ### Overview The image compares two problem-solving frameworks: **(a) Existing Methods for Chain-of-Thought (CoT) Reasoning** and **(b) DiffCoT (Diffusion-based CoT)**. Both diagrams illustrate step-by-step reasoning processes for solving a math problem involving Mrs. Snyder's monthly income. The diagrams use color-coded nodes (blue = chosen, gray = rejected, red = error) and arrows to represent logical flow. --- ### Components/Axes #### Diagram (a): Existing Methods for CoT - **Nodes**: - **Node 1**: Input (Question about Mrs. Snyder's income). - **Nodes 2–4**: Intermediate reasoning steps (Step 1–4). - **Node 4 (Red)**: Error response (incorrect final answer). - **Arrows**: - Solid arrows indicate progression through steps. - Dashed arrows (not present in (a)) likely represent refinement paths in (b). - **Text**: - **Question**: "Mrs. Snyder used to spend 40% of her monthly income on rent and utilities. Her salary was recently increased by $600 so now her rent and utilities now only amount to 25% of her previous monthly income. How much was her previous monthly income?" - **Steps**: 1. Let previous income = x. Increased income = x + 600. 2. 25% of x + 600 = rent/utilities. 3. 0.4x = 0.25(x + 600). Solve for x. 4. Rearranged equation: x = 4000 (incorrect). #### Diagram (b): DiffCoT - **Nodes**: - **Nodes 1–3**: Initial reasoning steps (Step 1–3). - **Node 4 (Red)**: Error response (incorrect final answer). - **Refined Nodes 3–4**: Corrected steps (Refined Step 3–4). - **Arrows**: - Solid arrows for initial flow. - Dashed arrows labeled "Diffusion" and "AR" (Augmentation/Refinement) show iterative corrections. - **Text**: - **Refined Steps**: 3. 0.25(x + 600) = 0.4x. Solve for x. 4. 0.15x = 150. Divide both sides by 150. --- ### Detailed Analysis #### Diagram (a): Existing Methods for CoT 1. **Step 1**: Defines variables (previous income = x, increased income = x + 600). 2. **Step 2**: Establishes relationship: 25% of increased income = rent/utilities. 3. **Step 3**: Sets up equation: 0.4x = 0.25(x + 600). 4. **Step 4**: Incorrect rearrangement: x = 4000 (error). #### Diagram (b): DiffCoT 1. **Initial Steps 1–3**: Same as (a) but with explicit error flags (⚠️) on Nodes 3 and 4. 2. **Refined Step 3**: Corrects equation to 0.25(x + 600) = 0.4x. 3. **Refined Step 4**: Solves 0.15x = 150 → x = 1000 (correct). --- ### Key Observations 1. **Error in Existing Methods**: - Step 4 in (a) incorrectly rearranges 0.15x = 600 to x = 4000 (should be x = 4000/0.15 ≈ 26,666.67). - DiffCoT identifies and corrects this via algebraic refinement. 2. **DiffCoT Workflow**: - Uses diffusion/AR to iteratively refine steps, isolating errors (e.g., Node 4 in (b) is marked as an error before correction). - Final answer (x = 1000) matches the problem’s constraints: - Previous income = $1000 → Increased income = $1600. - 25% of $1600 = $400 (rent/utilities), which is 40% of $1000. 3. **Color Coding**: - Red nodes (errors) in both diagrams highlight critical flaws in reasoning. - Gray nodes (rejected) in (b) suggest alternative paths discarded during refinement. --- ### Interpretation - **Problem-Solving Framework**: - Existing CoT methods (a) follow linear reasoning but fail to validate intermediate steps, leading to errors. - DiffCoT (b) introduces iterative refinement (diffusion/AR) to detect and correct errors, improving accuracy. - **Mathematical Insight**: - The error in (a) arises from misapplying percentage changes. DiffCoT’s refined steps explicitly isolate variables (e.g., 0.15x = 150) to avoid miscalculations. - **Practical Implications**: - DiffCoT’s structured refinement mirrors human cognitive processes (e.g., backtracking, hypothesis testing), making it more robust for complex reasoning tasks. - The diagrams emphasize the importance of validating intermediate steps in automated reasoning systems. --- **Note**: All values and steps are transcribed directly from the image. No additional languages or hidden data were detected. </details> Figure 1: Comparison of our proposed DiffCoT with existing CoT reasoning approaches: (a) Existing step-by-step CoT Reasoning methods adopt teacher-forcing training, where each step depends on the ground-truth output of the previous one. At inference time, this assumption breaks, causing exposure bias and leading to error accumulation. (b) DiffCoT performs CoT reasoning along both the noise (diffusion) and temporal (autoregressive) dimensions, enabling iterative correction of prior mistakes and effectively mitigating exposure bias. To mitigate the limitations above, we formulate CoT reasoning as a globally revisable trajectory, in which intermediate steps are not fixed once generated but can be adaptively corrected in light of later context. Rather than defining reasoning as a strictly forward, prefix-conditioned process driven by human priors (see Fig. 1), our method jointly models all reasoning steps within a diffusion paradigm (Dhariwal and Nichol, 2021), enabling robust reasoning under realistic noise from a global perspective with long-term revision guided by future signals. Starting from a corrupted or noisy reasoning chain, the entire reasoning trajectory can be iteratively updated, where intermediate revisions and forward progression are seamlessly integrated into a unified refinement process. By unifying iterative revision and forward continuation within the same diffusion-based refinement process, our design philosophy aim to reduce the discrepancy between training-time supervision and inference-time reasoning dynamics, leading to more stable reasoning behavior during inference. To this end, we propose Diff usion-Styled C hain o f T hought, DiffCoT, which integrates diffusion principles into the CoT framework to mitigate the exposure bias of static teacher-forcing and address the pointwise and local-step supervision issues of prior preference optimization solutions. Specifically, 1) DiffCoT modifies only the step-level generation strategy while preserving token-level autoregression, making it straightforward to fine-tune from existing autoregressive models. 2) By incorporating sliding windows and causal noise masks, DiffCoT unifies generation and revision within a versatile framework, to balance autoregressive and diffusion-styled reasoning. 3) Moreover, DiffCoT employs sentence-level noise injection to convert the reasoning paradigm into iterative denoising, progressively refining corrupted trajectories into coherent chains of thought, which captures the distributional evolution of reasoning errors to enable recovery from diverse and compounding mistakes. Altogether, DiffCoT could establish a unified and adaptive paradigm to CoT reasoning in LLMs. Our contributions are summarized as follows: - We propose DiffCoT, a novel diffusion-styled CoT framework that reformulates multi-step reasoning as an iterative denoising process, effectively mitigating exposure bias and error accumulation in autoregressive CoT reasoning. - We introduce a step-level diffusion-based learning strategy that unifies generation and revision through a sliding-window mechanism, enabling retrospective correction of earlier reasoning steps while preserving token-level autoregression. - We design a causal noise schedule that explicitly encodes the temporal dependency of reasoning steps, balancing global error correction with the causal structure required for coherent reasoning. - Extensive experiments on three public mathematical reasoning benchmarks demonstrate that DiffCoT outperforms State-of-The-Art (SoTA) PO methods, yielding robust gains and substantially improved error-correction capability in CoT. 2 Preliminaries Chain-of-Thought Reasoning Given a question prompt $p$ , the CoT paradigm (Wei et al., 2022) explicitly unfolds the reasoning process into a sequence of reasoning steps $s_{1:K}$ whose final element $s_{K}$ corresponds to the answer $a$ . The conditional distribution of a complete reasoning trace can be expressed as: $$ p_{\theta}(s_{1:K}\mid p)=\prod_{t=1}^{K}\pi_{\theta}(s_{k}\mid p,s_{<k}), \tag{1} $$ where $s_{k}$ denotes the $k$ -th reasoning step, $k≤ K$ is the step budget, and $\pi_{\theta}$ is the Auto-Regressive (AR) policy, i.e., the conditional distribution over the next step given the prompt $p$ and the previously generated steps. On annotated CoT data $(p,s_{1:K})$ , the model is trained by maximizing the conditional likelihood: $$ \mathcal{L}_{\mathrm{CoT}}=-\sum_{k=1}^{K}\log\pi_{\theta}(s_{k}\mid p,s_{<k}). \tag{2} $$ Diffusion Models Diffusion models (Dhariwal and Nichol, 2021) define a generative framework where data samples are gradually corrupted into a tractable noise distribution through a forward process, and a model is trained to approximate the corresponding reverse process. Let $x_{0}\sim\varphi(x_{0})$ denote a data sample from real data distribution $\varphi$ . The forward process produces a sequence $\{x_{t}\}_{t=1}^{T}$ by applying a noise operator $\mathcal{O}$ at each step: $$ x_{t}=\mathcal{D}(x_{t-1},\eta_{t}),\quad t=1,\dots,T, \tag{3} $$ where $t$ denotes the diffusion step index, $\eta_{t}$ is a noise variable. $\mathcal{O}$ could be a deterministic or structured corruption operator. The generative process is defined by learning a parameterized reverse transition $p_{\theta}(x_{t-1}\mid x_{t})$ that reconstructs the original sample through stepwise denoising. Training minimizes the discrepancy between the predicted $\hat{x}_{t-1}=f_{\theta}(x_{t},t)$ and the ground-truth corrupted data $x_{t-1}$ obtained from the forward process: $$ \begin{split}&\mathcal{L}_{\mathrm{Diffusion}}=\\ &\mathbf{E}_{x_{0}\sim\varphi(x_{0}),\,t\sim\{1,\dots,T\}}\Big[\ell\big(f_{\theta}(x_{t},t),\,x_{t-1}\big)\Big],\end{split} \tag{4} $$ where $f_{\theta}(x_{t},t)$ predicts the reconstruction of $x_{t-1}$ from $x_{t}$ , $\ell(·,·)$ is a reconstruction loss. In inference, sampling begins from $x_{T}$ and iteratively applies the learned reverse transitions until $x_{0}$ is obtained. 3 Methodology During CoT reasoning, LLMs are typically trained only on correct trajectories (Wei et al., 2022), while at inference time they may condition on erroneous intermediate steps (Lyu et al., 2023). Moreover, step-wise optimization focuses on local alignment and ignores future signals within the reasoning trajectory, limiting global consistency (Zhang et al., 2024). Together, these issues lead to error accumulation (Yoon et al., 2025; Chen et al., 2024a), motivating a rethinking of CoT reasoning under preference optimization. To address these issues, we argue that effective CoT reasoning requires trajectory-level revision to enforce global consistency and recover from corrupted intermediate steps. As error accumulation leads to noisy reasoning chains at inference time, a mechanism for global recovery becomes essential (Lyu et al., 2023; Wang et al., 2023). Motivated by the robustness of diffusion models in reconstructing structured data from noise (Ho et al., 2020; Li et al., 2022), we aim to develop a diffusion-styled preference optimization paradigm to reformulate CoT reasoning. To this end, we propose Diff usion-styled C hain o f T hought (DiffCoT), which models CoT reasoning in a diffusion-styled framework. We first present a step-level forward noising process for CoT using reward-ranked candidates (§ 3.1), and then apply a diffusion sliding window to iteratively denoise past steps while generating new ones (§ 3.2). We further introduce causal diffusion noise to strengthen causal consistency across reasoning steps (§ 3.3). An overview is shown in Figure 2. 3.1 Diffusion-Styled Noising for CoT In this section, we design a diffusion-styled forward noising process for CoT reasoning at the reasoning-step level, where step-level noise is induced by ranking candidate responses for the same step according to their reward scores. Previous preference optimization methods overlooked the distribution between high-reward and low-reward reasoning steps and relied on isolated step-wise adjustments that fail to mitigate the exposure bias inherent in teacher forcing. To this end, in our principle, higher-reward candidates are treated as lower-noise reasoning states, while lower-reward candidates correspond to progressively higher-noise states in the forward process. This ranking forms a progression from clean to corrupted reasoning states under diffusion-styled noise, enabling distribution-aware modeling of reasoning steps. Specifically, we implement forward noising by first collecting CoT reasoning trajectories for each question to construct the training set $\mathcal{D}$ . As shown in Fig. 2 (a), given a question prompt $p$ , we follow the standard CoT data generation process by employing MCTS (Browne et al., 2012), to gradually build a search tree for step-by-step solution exploration. The root corresponds to $p$ , and each child node represents an intermediate step $s$ . A complete path from the root to a terminal node $s_{K}$ yields a trajectory $\mathbf{tra}=p\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{K}$ , where each step $s_{k}$ is annotated with its reward $r(s_{k})$ . In contrast to conventional CoT trajectory construction Zhang et al. (2024), at each step $k$ we collect several candidate responses. These responses are scored using either an external reward model or rollout-based success rates. In our forward noising view, the candidate with the highest reward, denoted as $s^{\sigma^{0}=w}_{k}$ , is regarded as the lowest-noise state for step $k$ , and the remaining candidates are ordered by their rewards to form $\{s^{\sigma^{1:T}}_{k}\}$ , which we interpret as states with gradually increasing noise. Step-level noise is thus measured by the deviation of lower-reward responses from this best trajectory, indicating the corruption level at the step granularity. This design ensures that the final generation remains coherent while providing a set of forward diffusion states for subsequent denoising and preference optimization. Under this step-level forward noising, DiffCoT defines a diffusion-styled distribution that spans low-noise to high-noise reasoning states (Chen et al., 2025). Note that our data construction does not distinguish between “correct” and “incorrect” labels at the collection stage as in (Lai et al., 2025); instead, it uniformly gathers diverse responses for each step throughout the entire trajectory. This global view provides distribution-aware data to support both generation and refinement. In this way, error patterns are implicitly encoded in trajectory-level modeling, allowing robust reasoning without additional manual annotations of corrective steps. 3.2 Reformulating CoT Reasoning as Denoising <details> <summary>figs/method.png Details</summary>  ### Visual Description ## Diagram: Multi-Stage Decision and Diffusion Model Architecture ### Overview The image presents a technical diagram illustrating a multi-stage decision-making process combined with a diffusion model. It is divided into three sections: 1. **Best Trajectory Selection** (a) 2. **Diffusion Sliding Window** (b) 3. **Diffusion Model Heatmap** (c) ### Components/Axes #### (a) Best Trajectory Selection - **Nodes**: Labeled 1–4, with subscripts indicating state variations (e.g., {s₁₁₄}, {s₂₁₄}, etc.). - **Arrows**: Directed from a central node labeled "P" to nodes 1–4, with a highlighted "Best trajectory" path (blue line) connecting nodes 1→2→3→4. - **Sorting**: A dashed arrow labeled "Sort by Reward" points downward from node 4. - **Subscripts**: Nodes are annotated with subscripts like {s₁₁₄}, {s₂₁₄}, {s₃₁₄}, {s₄₁₄}, suggesting hierarchical or contextual state definitions. #### (b) Diffusion Sliding Window - **Nodes**: Labeled 1–4, with subscripts {s₁₁₄}, {s₂₁₄}, {s₃₁₄}, {s₄₁₄}. - **Sliding Window**: A blue dashed box highlights nodes 1→2→3, labeled "Diffusion Sliding Window." - **DPO Outcomes**: - **DPO lose**: Red arrow pointing to a network labeled "LLM" (Large Language Model). - **DPO win**: Blue arrow pointing to a network labeled "DPO win." - **Legend**: Located on the right, with red for "DPO lose" and blue for "DPO win." #### (c) Diffusion Model Heatmap - **Axes**: - **X-axis**: "Reward (Denoising)" with labels l < p < n < w. - **Y-axis**: "S₁" to "S₄" (states). - **Heatmap**: Gradient from light orange (low reward) to dark orange (high reward). - **Arrows**: - **AR (Adaptive Reward)**: Red arrow pointing upward from the heatmap. - **DiffCoT (Diffusion CoT)**: Blue arrow pointing upward from the heatmap. - **Denoising Arrows**: - **Step-level denoising**: Orange arrow labeled "Denoising at step level." - **Reward-level denoising**: Blue arrow labeled "Denoising at reward level." ### Detailed Analysis #### (a) Best Trajectory Selection - The "Best trajectory" (blue line) connects nodes 1→2→3→4, indicating an optimal path. - Nodes are annotated with subscripts (e.g., {s₁₁₄}), suggesting state-specific parameters or constraints. - The "Sort by Reward" instruction implies a post-processing step to prioritize trajectories based on reward metrics. #### (b) Diffusion Sliding Window - The sliding window (blue box) spans nodes 1→2→3, emphasizing sequential state transitions. - **DPO lose** (red) and **DPO win** (blue) outcomes are tied to the LLM and DPO win networks, respectively. - The legend confirms color coding: red for negative outcomes (DPO lose) and blue for positive outcomes (DPO win). #### (c) Diffusion Model Heatmap - The heatmap visualizes the relationship between reward (denoising) and states (S₁–S₄). - **AR** and **DiffCoT** arrows suggest mechanisms to adjust reward or denoising strategies. - The gradient indicates that higher rewards (darker orange) correlate with specific states (e.g., S₄). ### Key Observations 1. **Best Trajectory**: The highlighted path (1→2→3→4) is the optimal sequence, prioritized by reward. 2. **Sliding Window Dynamics**: The diffusion process evaluates sequential states (1→2→3) to determine outcomes (DPO lose/win). 3. **Heatmap Trends**: The gradient and arrows suggest that denoising strategies (AR, DiffCoT) influence reward outcomes, with step-level denoising affecting intermediate states and reward-level denoising impacting final rewards. ### Interpretation - The diagram illustrates a **reinforcement learning framework** where decisions (nodes 1–4) are optimized via reward-based sorting. - The **diffusion sliding window** acts as a mechanism to evaluate sequential state transitions, with DPO outcomes (lose/win) determining the model's adaptability. - The **heatmap** quantifies the relationship between denoising steps and rewards, showing that higher rewards (S₄) are associated with specific denoising strategies (AR, DiffCoT). - **DPO lose/win** outcomes are critical for refining the model, as they directly influence the LLM's performance. - The **step-level vs. reward-level denoising** arrows imply a hierarchical approach: step-level adjustments optimize intermediate states, while reward-level adjustments target final outcomes. This architecture likely supports a **reinforcement learning with diffusion-based exploration**, where the model balances exploration (diffusion) and exploitation (reward optimization) to identify optimal trajectories. </details> Figure 2: DiffCoT Framework and Training Data Construction: (a) Step-level forward noising: MCTS-based data generation defines step-level noise by reward-ranking multiple candidates, yielding states ranging from clean to corrupted. (b) Sliding-window denoising: a diffusion sliding window refines previously generated CoT steps while producing the next step in an autoregressive manner. (c) Causal diffusion noise: a step-dependent schedule assigns stronger noise to later steps to encode the causal order of the reasoning chain. In this section, our DiffCoT framework reformulates CoT reasoning beyond the teacher-forcing paradigm as a diffusion-styled denoising process to mitigate exposure bias at inference time. To couple this with the autoregressive nature of CoT, we introduce a diffusion sliding-window mechanism that operates directly on the CoT reasoning process. Within this sliding window, previously generated CoT steps are progressively denoised from high-noise toward low-noise reasoning states, thereby facilitating self-correction, while advancing the window naturally aligns with the autoregressive generation of subsequent CoT steps. Thus, instead of costly training a separate diffusion language model from scratch, we leverage diffusion-styled data to efficiently fine-tune pre-trained LLM $\pi$ for CoT reasoning. Specifically, for the iterative denoising process, the model $\pi$ takes the question prompt $p$ together with the preceding reasoning steps $\mathbf{s}_{1:k}$ as input. To enable variable-length generation when applying diffusion to thought chains while integrating generation with self-correction, the model maintains a diffusion sliding window of size $m$ and stride $n$ . At denoising iteration $t$ , the window containing previously generated CoT steps $\{s_{k-m}^{\sigma},...,s_{k}^{\sigma}\}$ is updated to a lower-noise version $\{s_{k-m}^{\sigma^{\prime}},...,s_{k}^{\sigma^{\prime}}\}$ , where $\sigma=\sigma_{k}^{(t)}$ is defined as a function of the step index $k$ and the denoising iteration $t$ , while $\sigma$ and $\sigma^{\prime}$ denote the noise levels before and after refinement, respectively. Simultaneously, as the window shifts forward by one step, the model predicts the next step $s_{k+1}^{\sigma}$ , initialized at a high-noise state. Iterating this denoising process eventually yields a clean trajectory $s_{1:K}^{\sigma^{0}}$ : $$ \pi_{\theta}\!\big(p,\,\mathbf{s}_{1:k}^{\,\sigma}\big)\;\mapsto\;\underbrace{\mathbf{s}_{k-m:k}^{\,\sigma^{\prime}}}_{\text{refined past}}\;\oplus\;\underbrace{\mathbf{s}_{k+1}^{\,\sigma}}_{\text{predicted future}}\,. \tag{5} $$ An illustration of the denoising process is shown in Fig. 2 (b). In sequence modeling, teacher-forcing next-token prediction can be viewed as masking along the reasoning-step axis $s$ , whereas diffusion corresponds to masking along the noise axis $\sigma$ , where $\sigma^{T}$ approaches pure white noise after $T$ iterations. To unify both views, we denote $s_{k}^{\sigma}$ as the $k$ -th CoT step under noise level $\sigma$ , where $\sigma$ is instantiated as $\sigma_{k}^{t}$ , i.e., the diffusion noise strength assigned to the $k$ -th step at denoising iteration $t$ . Building on the above reasoning process, we now introduce the training objective in DiffCoT. Our goal is to optimize a model $\pi_{\theta}$ that takes the question prompt $p$ together with the preceding reasoning steps $\mathbf{s}_{1:k}$ as input. We construct the win sequence $s_{k-m:k+1}^{w}$ by combining the denoised steps $\{s_{k-m}^{\sigma^{\prime}},...,s_{k}^{\sigma^{\prime}}\}$ with a lower-noise variant of $s_{k+1}^{\sigma_{k+1}^{\prime}}$ . Similarly, the lose sequence $s_{k-m:k+1}^{l}$ is formed by combining the unrefined steps $\{s_{k-m}^{\sigma},...,s_{k}^{\sigma}\}$ with the highest-noise variant of $s_{k+1}^{\sigma_{k+1}}$ . The prefix condition is the past text $s_{1:k-1}$ . To optimize the LLM on this preference pair, we adopt the Direct Preference Optimization (DPO) loss (Rafailov et al., 2023): $$ \begin{split}\mathcal{L}_{i}(\pi_{\theta};\pi_{\textrm{ref}})=-\log\phi\Big(&\beta\log\frac{\pi_{\theta}(s_{:k+1}^{w}\mid p,s_{1:k})}{\pi_{\textrm{ref}}(s_{:k+1}^{w}\mid p,s_{1:k})}\\ -\beta\log&\frac{\pi_{\theta}(s_{:k+1}^{l}\mid p,s_{1:k})}{\pi_{\textrm{ref}}(s_{:k+1}^{l}\mid p,s_{1:k})}\Big),\end{split} \tag{6} $$ where $s_{:k+1}$ denotes the subsequence $s_{k-m:k+1}$ , $\phi(·)$ denotes the sigmoid function, and $\beta$ is a factor controlling the strength of the preference signal. Note that our conditional prefix $s_{1:k-1}^{\sigma}$ differs from that used in standard DPO: instead of being composed exclusively of preferred (i.e., clean) steps, it combines clean steps from $1$ to $k-m$ with noisy steps within the sliding window. In this manner, the hybrid construction could alleviate the exposure bias by training the model to make preference-consistent updates even when conditioned on partially corrupted reasoning prefixes. 3.3 Causal Diffusion Noise Modeling CoT reasoning with diffusion poses a significant challenge to causality. Conventional full-sequence diffusion models are inherently non-causal (Ho et al., 2020; Dhariwal and Nichol, 2021), which contrasts sharply with the causal nature of CoT reasoning. While our backbone $\pi$ retains an AR token-level generation process, prior studies on diffusion models indicate that relying solely on the model’s ability to capture causality is inadequate, especially when reasoning must be performed over noisy or perturbed data (Li et al., 2022). Inspired by Diffusion Forcing (Chen et al., 2024a), as shown in Fig. 2 (c), we leverage noise as a mechanism to inject causal priors into diffusion sliding window.In conventional full-sequence diffusion, the noise strength $\sigma^{(t)}$ depends solely on the denoising iteration $t$ and is shared across all tokens. Such uniform noise injection is ill-suited for CoT reasoning, as it limits the model’s ability to capture step-wise causal dependencies. To overcome this, we redefine the noise schedule as $\sigma^{t}_{k}$ , a joint function of reasoning step $k$ and iteration $t$ (see § 3.2). Within the diffusion sliding window, $\sigma^{t}_{k}$ follows a progressive schedule, where earlier steps are perturbed with weaker noise, while later steps are perturbed with stronger noise. The noise schedule is formally defined as follows: $$ \begin{split}&F(s_{j-m+1},\ldots,s_{j};j)=\\ &\big(s_{j-m+1}^{\sigma^{0}},\;s_{j-m+2}^{\sigma^{1}},\;\ldots,\;s_{j}^{\sigma^{T}}\big),\end{split} \tag{7} $$ where the diffusion sliding window has size $m$ and stride $1$ , and at the $j$ -th denoising iteration the window advances to generate $s_{j}$ . In this manner, our framework could better stabilize the causal chain and enhance self-correction in subsequent reasoning steps even if the current step is erroneous. 4 Experiment | GSM8K SVAMP M-L1 | 37.2 49.6 31.3 | 39.3 50.1 33.4 | 37.6 49.4 32.2 | 37.7 48.8 31.5 | 36.1 48.6 33.8 | 38.1 48.4 32.0 | 39.6 50.4 34.3 | 65.5 83.5 61.1 | 64.3 84.9 62.0 | 65.7 84.2 60.1 | 65.1 82.7 60.8 | 64.9 85.2 61.7 | 65.9 85.9 63.0 | 66.2 85.5 63.6 | 62.0 80.1 51.2 | 64.0 80.7 52.4 | 61.8 82.3 51.8 | 63.0 80.3 51.6 | 63.3 79.4 51.8 | 64.7 83.0 52.7 | 65.4 83.2 53.3 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | M-L2 | 18.4 | 18.5 | 18.1 | 17.9 | 19.3 | 17.6 | 19.5 | 39.1 | 39.2 | 38.4 | 38.7 | 39.9 | 37.1 | 39.3 | 31.9 | 34.0 | 32.6 | 33.1 | 35.7 | 34.2 | 35.3 | | M-L3 | 12.5 | 13.2 | 12.6 | 12.9 | 12.6 | 13.7 | 14.2 | 25.2 | 26.1 | 25.3 | 25.5 | 26.2 | 25.8 | 26.9 | 16.0 | 17.0 | 16.2 | 16.4 | 17.2 | 17.9 | 18.0 | | M-L4 | 5.9 | 6.1 | 6.6 | 6.4 | 6.7 | 7.6 | 8.0 | 16.5 | 15.0 | 15.5 | 15.6 | 15.8 | 17.4 | 17.5 | 9.0 | 10.2 | 8.3 | 8.5 | 8.9 | 9.9 | 10.5 | | M-L5 | 1.9 | 2.3 | 1.4 | 2.9 | 3.0 | 3.1 | 3.3 | 4.4 | 5.4 | 5.2 | 3.7 | 4.2 | 4.7 | 5.5 | 1.0 | 1.2 | 0.7 | 0.7 | 2.4 | 1.4 | 2.9 | Table 1: Test accuracy (%) on GSM8K, SVAMP, and MATH. M-L1 to M-L5 denote different difficulty levels in the MATH dataset, where L1 is the easiest and L5 is the most challenging, while Step denotes Step-DPO and FStep denotes Full-Step-DPO. The best and second results are in bold and underlined. 4.1 Settings Models and Datasets We conduct experiments on three representative backbone models: Llama3-8B (Touvron et al., 2023), Qwen3-8B (Yang et al., 2025), and Qwen3-4B (Yang et al., 2025). We primarily evaluate our DiffCoT on three public mathematical reasoning benchmarks, specifically GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MATH (Hendrycks et al., 2021). The MATH is divided into five difficulty levels, as defined in the original dataset. More data statistics and preparation are provided in Appendix § A. Baselines We compare DiffCoT with the following SoTA baselines: 1) CoT (Wei et al., 2022): generates step-by-step reasoning before the final answer, evaluated with greedy decoding. 2) ToT (Yao et al., 2023): explores multiple reasoning paths via tree search. 3) TS-SFT (Feng et al., 2023): applies supervised fine-tuning on reasoning paths obtained from ToT. 4) CPO (Zhang et al., 2024): performs preference optimization at the step level, directly aligning intermediate reasoning steps via contrastive pairs. 5) Step-DPO (Lai et al., 2025): step-level preference optimization by explicitly collecting erroneous steps and applying preference optimization to correct them. 6) Full-Step-DPO (Xu et al., 2025): further generalizes step-wise optimization to the entire reasoning trajectory. More implementation details regarding model training and data construction are provided in Appendix § B. 4.2 Main Results As shown in Table 1, DiffCoT can be directly applied to modern instruction-tuned language models, such as Qwen3 and Llama3, via standard fine-tuning. This design enables seamless integration into existing training pipelines and allows DiffCoT to deliver consistent performance improvements across models of varying sizes, demonstrating strong scalability across different base models. From the result, we can observe that: 1) Baselines guided by LLM self-verification signals, such as CPO and ToT, exhibit notable instability across models and datasets. For example, when applied to Qwen3-8B, CPO results in performance degradation on the MATH-1 dataset while achieving competitive performance on SVAMP, indicating sensitivity to both model and task distributions. 2) Step-level preference learning baselines, including Step-DPO and Full-Step-DPO, generally achieve great improvements but still suffer from occasional performance drops under certain settings. This suggests that relying solely on local step-wise optimization may not be sufficient to ensure stable reasoning behavior across diverse evaluation scenarios. 3) Overall, DiffCoT consistently outperforms existing PO approaches across the majority of benchmarks and evaluation settings considered. Although it does not achieve the single best result in every individual setting, such as on SVAMP with the Qwen3-8B backbone, where Full-Step-DPO attains slightly higher accuracy, DiffCoT consistently ranks among the top-performing methods across all settings. Moreover, the improvements achieved by DiffCoT are substantial in magnitude and stable across different model sizes and datasets, highlighting its robustness and reliability. 4.3 Ablation Study To validate the effectiveness of the proposed DiffCoT method, we perform an ablation study on its key components. We select two representative base language models from different model families with varied sizes, Llama3-8B and Qwen3-4B, and evaluate them on two general mathematical reasoning benchmarks, GSM8K and SVAMP. To investigate the impact of incorporating diffusion into CoT reasoning, we evaluate various window sizes to explore how this factor influences the model’s performance. Specifically, when the diffusion window size and stride are set to 1, our approach essentially degenerates into an AR method, albeit with differently constructed prefixes. On the other hand, when the sliding diffusion window and stride are set to the number of steps $K$ , the method reverts to a purely diffusion-based approach. As shown in Table 2, it can be observed that performance degrades when the window size and stride is set to 1 or when it becomes too large. We attribute this phenomenon to a trade-off between causal connectivity and error-correction capability. Strengthening the ability to revise earlier steps often comes at the expense of weakening top-down causal reasoning. When the model operates entirely in AR mode, it exhibits the strongest causal reasoning ability but suffers from exposure bias during testing. In contrast, when the model fully adopts the diffusion mode, an excessively long window introduces noise: it disrupts the structural coherence of reasoning and prolonged denoising fluctuations undermine the causal nature of inference. | Llama3-8B - window size, stride=1 - window size, stride= $K$ | 39.6 36.3 -3.3 30.3 -9.3 | 50.4 48.3 -2.1 42.6 -7.8 | | --- | --- | --- | | - causal noise | 35.5 -4.1 | 48.2 -2.2 | | Qwen3-4B | 65.4 | 83.2 | | - window size, stride=1 | 63.0 -2.4 | 80.2 -3.0 | | - window size, stride= $K$ | 56.7 -8.7 | 73.9 -9.3 | | - causal noise | 61.9 -3.5 | 79.1 -4.1 | Table 2: Ablative results on the general datasets GSM8K and SVAMP, where $K$ is the number of reasoning steps. Numbers shown in the upper-right corner of each cell indicate the relative change in accuracy rate compared to the full DiffCoT model. We further conduct an ablation study to verify the effectiveness of the causal diffusion noise. Specifically, we disrupt our proposed noise scheduling by randomly shuffling the data order used for noising, instead of following the accuracy-based progression. This modification effectively breaks the causal structure of the noise. Results in Table 2 show that this ablation leads to substantial performance degradation across different models and datasets, demonstrating that our causal noise scheduling is a critical component of DiffCoT. 4.4 Analysis Case Study To further illustrate the model’s robustness to accumulated imperfections in intermediate reasoning, we present qualitative case studies that contrast effective and suboptimal reasoning trajectories. In particular, we focus on challenging examples in which the model introduces semantically irrelevant or weakly informative steps at an early stage of reasoning. Although such steps are not necessarily incorrect in isolation, they tend to accumulate and hinder progress toward the correct solution under standard step-by-step reasoning. As shown in Fig. 3, DiffCoT is able to progressively refine earlier reasoning steps through the diffusion process. Instead of rigidly committing to the initial reasoning prefix, the model gradually improves previously generated steps while producing subsequent ones, effectively revising semantically unhelpful content introduced earlier in the trajectory. This allows the model to move away from locally coherent but globally suboptimal reasoning paths and ultimately reach the correct solution. It highlights a key distinction between DiffCoT and conventional autoregressive reasoning. While autoregressive models tend to propagate early semantic noise forward without revision, DiffCoT leverages its denoising dynamics to refine the entire reasoning trajectory, including previously generated steps, resulting in more robust and globally consistent reasoning behavior. <details> <summary>figs/case.png Details</summary>  ### Visual Description ## Text Screenshot: Download Time Calculation Problem ### Overview The image contains a mathematical problem about calculating download time with an interruption, followed by two sets of reasoning steps (labeled "Past Reasoning Steps" and "Next Generation"). The problem involves a 200 GB file download interrupted by a system restart, with calculations for total time. --- ### Components/Axes - **Text Structure**: - **Question**: Describes the scenario (200 GB file, 2 GB/min speed, 40% interruption at 20 minutes). - **Past Reasoning Steps**: Four steps with color-coded calculations (blue, red). - **Next Generation**: Five steps with color-coded calculations (blue, red). --- ### Detailed Analysis #### Past Reasoning Steps: 1. **Step 1**: - **Text**: "40% of the way through the download is 0.4 * 200 GB = 80 GB." - **Calculation**: 40% of 200 GB = 80 GB. 2. **Step 2**: - **Text**: "80 GB / 2 GB per minute = 40 minutes to download the first 80 GB." - **Calculation**: Time = 80 GB ÷ 2 GB/min = 40 minutes. 3. **Step 3**: - **Text**: "20 minutes for the restart." - **Calculation**: Restart duration = 20 minutes. 4. **Step 4**: - **Text**: "200 GB - 80 GB = 120 GB remaining after the restart." - **Calculation**: Remaining data = 120 GB. #### Next Generation: 1. **Step 3**: - **Text**: "20 minutes for the restart, so total time so far is 40 + 20 = 60 minutes." - **Calculation**: Total time after restart = 60 minutes. 2. **Step 4**: - **Text**: "Then she downloads the full 200 GB from scratch, taking 100 minutes after the restart starts." - **Calculation**: Time for full download = 200 GB ÷ 2 GB/min = 100 minutes. 3. **Step 5**: - **Text**: "The total time can be calculated by adding the time before and after the interruption." - **Calculation**: Total time = 60 + 100 = 160 minutes. --- ### Key Observations - **Discrepancy in Approaches**: - **Past Reasoning**: Assumes Carla resumes the download after the restart (120 GB remaining). - **Next Generation**: Incorrectly assumes Carla restarts the download from the beginning (200 GB). - **Color Coding**: - Blue text highlights calculations (e.g., 40% = 80 GB, 80 GB ÷ 2 GB/min). - Red text emphasizes critical values (e.g., 20 minutes restart, 120 GB remaining). --- ### Interpretation - **Correct Approach**: The "Past Reasoning Steps" correctly calculate the total time as **40 minutes (initial download) + 20 minutes (restart) + 60 minutes (remaining 120 GB)** = **120 minutes total**. - **Flawed Logic in Next Generation**: The "Next Generation" steps erroneously restart the download from scratch, leading to an inflated total time of **160 minutes**. This highlights a misunderstanding of resumable downloads. - **Critical Insight**: The problem tests understanding of partial progress preservation after interruptions, a common real-world scenario in file transfers. --- ### Final Answer (Extracted from Image) The total download time is **120 minutes** (Past Reasoning Steps). The "Next Generation" steps incorrectly calculate 160 minutes by restarting the download. </details> Figure 3: Example illustrating how DiffCoT modifies early-stage reasoning shift steps. The steps highlighted in blue represent the diffusion sliding window. <details> <summary>x2.png Details</summary>  ### Visual Description ## Line Graphs: Accuracy vs. Noise Intensity Coefficient (α) for Different Models ### Overview The image contains three line graphs comparing the performance of two methods—**DiffCoT** (blue) and **Full-Step DPO** (orange)—across three language models: **Llama3-8B**, **Qwen3-8B**, and **Qwen3-4B**. Each graph plots accuracy (y-axis, %) against the noise intensity coefficient (α, x-axis, ranging from 0 to 0.4). Both methods show declining accuracy as α increases, but **DiffCoT** consistently outperforms **Full-Step DPO** across all models. --- ### Components/Axes - **X-axis**: Noise Intensity Coefficient (α) with markers at 0, 0.1, 0.2, 0.3, 0.4. - **Y-axis**: Accuracy (Acc, %) with increments of 5%. - **Legends**: - Blue line: **DiffCoT** - Orange line: **Full-Step DPO** - **Graph Titles**: - Llama3-8B (left) - Qwen3-8B (center) - Qwen3-4B (right) --- ### Detailed Analysis #### Llama3-8B - **DiffCoT**: Starts at ~38% (α=0), declines to ~19% (α=0.4). - **Full-Step DPO**: Starts at ~38% (α=0), declines to ~15% (α=0.4). - **Trend**: Both methods show a steep downward slope, with **DiffCoT** maintaining a ~3–5% advantage. #### Qwen3-8B - **DiffCoT**: Starts at ~66% (α=0), declines to ~42% (α=0.4). - **Full-Step DPO**: Starts at ~66% (α=0), declines to ~40% (α=0.4). - **Trend**: Similar to Llama3-8B, but with higher absolute accuracy values. The gap between methods narrows slightly at α=0.4. #### Qwen3-4B - **DiffCoT**: Starts at ~64% (α=0), declines to ~41% (α=0.4). - **Full-Step DPO**: Starts at ~64% (α=0), declines to ~38% (α=0.4). - **Trend**: Steeper decline compared to other models. **DiffCoT** retains a ~3% edge at α=0.4. --- ### Key Observations 1. **Consistent Performance Gap**: **DiffCoT** outperforms **Full-Step DPO** across all models and α values. 2. **Model Sensitivity**: - **Qwen3-4B** shows the steepest decline in accuracy for both methods, suggesting higher sensitivity to noise. - **Llama3-8B** exhibits the least sensitivity, with a smaller drop in accuracy. 3. **Noise Impact**: Accuracy decreases non-linearly as α increases, with sharper drops at higher α values. --- ### Interpretation The data demonstrates that **DiffCoT** is more robust to noise than **Full-Step DPO** across all tested models. This could indicate that **DiffCoT**’s methodology (e.g., incremental noise injection during training) better prepares models for noisy environments. The steeper decline in **Qwen3-4B** suggests architectural or training differences that make it more vulnerable to noise. Practically, **DiffCoT** may be preferable for applications requiring noise resilience, while **Full-Step DPO** might suffice in cleaner environments. Further investigation into the noise-handling mechanisms of these methods could clarify their relative strengths. </details> Figure 4: Correction success rate under stochastic prefix corruption, where noise is injected at the midpoint of the reasoning trajectory with probability $\omega$ . Error Accumulation Analysis We further analyze the model’s ability to recover from accumulated imperfections in intermediate reasoning steps. We consider a correction-oriented setting in which the model is deliberately conditioned on prefixes that contain semantically suboptimal or noisy reasoning steps, and is then required to continue the reasoning process to reach a correct final answer. To this end, we introduce controlled perturbations at an intermediate stage of the reasoning process. Specifically, when the reasoning reaches approximately half of the trajectory, each preceding step is independently perturbed with probability $\omega$ . A perturbation replaces the original step with a low-reward alternative sampled from model-generated trajectories that are semantically plausible but less aligned with the optimal reasoning path. The noise strength $\omega$ therefore governs the degree of accumulated semantic drift in the prefix, enabling a systematic evaluation of robustness under varying levels of intermediate reasoning noise. We compare different training strategies under this protocol using the correction success rate, defined as the proportion of cases in which the model produces a correct final answer despite being conditioned on a perturbed prefix. Experiments are conducted on 300 GSM8K problems with multiple backbone models, comparing our approach against Full-Step-DPO. As shown in Fig. 4, our method consistently achieves substantially higher correction success rates across all settings, demonstrating a stronger ability to compensate for accumulated intermediate noise and to maintain globally coherent reasoning, rather than brittle continuation of imperfect early steps. In other words, our model can mitigate the impact of early-stage reasoning drift, rather than rigidly propagating locally coherent but globally suboptimal trajectories in CoT reasoning. 5 Related Work Preference Learning Preference learning has recently emerged as an effective paradigm for aligning LLMs with human preferences by contrasting desirable and undesirable responses (Ouyang et al., 2022; Liu et al., 2025b), with methods such as DPO showing strong performance on general language tasks (Rafailov et al., 2023). However, extending these gains to mathematical reasoning remains challenging, largely due to the coarse granularity of solution-level supervision, which fails to localize and correct intermediate reasoning errors (Chen et al., 2024b). To address this limitation, recent work incorporates step-level preference signals to align intermediate reasoning processes (Lai et al., 2024; Lu et al., 2024), while Full-Step-DPO further optimizes entire reasoning trajectories using global objectives (Xu et al., 2025). Despite these advances, most existing step-wise methods adopt a teacher-forcing paradigm and reason exclusively over clean prefixes during training, leading to error accumulation and degraded robustness during inference. Accordingly, our work reconceptualizes CoT reasoning as a globally revisable trajectory, in which previously generated steps remain malleable to correction in light of future context. Mathematical Reasoning Mathematical reasoning is widely regarded as a challenging capability for large language models, and prior work has explored several directions to improve it. One line of research strengthens base models through continual pre-training on large-scale mathematical corpora or supervised fine-tuning on synthetic datasets distilled from stronger models (Azerbayev et al., 2023; Zhihong et al., 2024; Xu et al., 2024; Mitra et al., 2024). Another line improves test-time performance by increasing computational budgets, such as generating and reranking multiple solutions using outcome- or process-level rewards, or adopting reward-guided decoding strategies (Guan et al., 2025; Wu et al., 2024; Wang et al., 2024). Recently, reinforcement learning and preference-based optimization have been explored to directly align reasoning behaviors via trajectory- or step-level supervision, aiming to improve robustness beyond supervised objectives alone (Pal et al., 2024; Wang et al., 2024). But they largely operate in a forward-only manner and lack mechanisms for revising corrupted intermediate reasoning, motivating our diffusion-styled reformulation of CoT reasoning to enable effective error correction. 6 Conclusion and Future Work In this work, we proposed a novel DiffCoT framework to enhance mathematical reasoning and alleviate error accumulation by integrating diffusion steps into autoregressive generation. Our method utilizes a sliding diffusion window with causal noise to refine intermediate steps while maintaining consistent reasoning chains. Extensive experiments confirm that our proposed DiffCoT outperforms existing CoT methods, offering a robust solution for multi-step CoT reasoning. Future work will explore the scalability of our paradigm across more backbone models and broader reasoning domains. Limitations Although we have conducted extensive experiments on DiffCoT and observed strong empirical performance, several limitations remain. First, the data construction and training of DiffCoT follow an off-policy paradigm, where preference data are collected using a policy that differs from the one being optimized. Such a mismatch between the behavior policy and the training policy may introduce distribution shift, biased value estimation, and training instability, especially when scaling to more difficult datasets or longer reasoning chains. Second, similar to Diffusion Forcing, DiffCoT breaks the local Markov property of prefix conditioned generation by revisiting and modifying historical reasoning steps. While this violation enables stronger capabilities, it also increases the uncertainty and controllability challenges during generation, typically requiring more training iterations and larger amounts of data to achieve stable convergence. References - Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. - Azerbayev et al. (2023) Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.06786. - Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28. - Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS). - Browne et al. (2012) Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43. - Chen et al. (2024a) Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. 2024a. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125. - Chen et al. (2024b) Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024b. Step-level value preference optimization for mathematical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7889–7903, Miami, Florida, USA. Association for Computational Linguistics. - Chen et al. (2025) Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Ziyang Wang, Tony Quek, Joey Tianyi Zhou, Soujanya Poria, and Zuozhu Liu. 2025. DiffPO: Diffusion-styled preference optimization for inference time alignment of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18910–18925, Vienna, Austria. Association for Computational Linguistics. - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. - Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Quinn Nichol. 2021. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS). - Feng et al. (2023) Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2023. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179. - Guan et al. (2025) Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rStar-math: Small LLMs can master math reasoning with self-evolved deep thinking. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 20640–20661. PMLR. - Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. - Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. - Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS). - Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. - Kojima et al. (2022) Takeshi Kojima, Shixiang Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916. - Lai et al. (2024) Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629. - Lai et al. (2025) Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2025. Step-DPO: Step-wise preference optimization for long-chain reasoning of LLMs. - Li et al. (2022) Xiang Lisa Li, Tianyu Gong, Tian He, Dan Jurafsky, Percy Liang, and Steven CH Zhao. 2022. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems (NeurIPS). - Li et al. (2024) Yanhong Li, Chenghao Yang, and Allyson Ettinger. 2024. When hindsight is not 20/20: Testing limits on reflective thinking in large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3741–3753. - Liu et al. (2025a) Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025a. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. - Liu et al. (2025b) Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, and 1 others. 2025b. Lipo: Listwise preference optimization through learning-to-rank. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2404–2420. - Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. - Lu et al. (2024) Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning. CoRR. - Lyu et al. (2023) Qing Lyu, Tongshuang Wu, Fei Sha, and Graham Neubig. 2023. Faithful chain-of-thought reasoning. Preprint, arXiv:2301.13379. - Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. Preprint, arXiv:2402.14830. - Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. - Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228. - Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191. - Qin et al. (2024) Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and 1 others. 2024. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982. - Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems. - Ross et al. (2011) Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings. - Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. - Wang et al. (2024) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439, Bangkok, Thailand. Association for Computational Linguistics. - Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR). - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. - Wu et al. (2024) Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. 2024. Enhancing mathematical reasoning in llms by stepwise correction. Preprint, arXiv:2410.12934. - Xi et al. (2024) Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, and 1 others. 2024. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579. - Xu et al. (2025) Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, and Anh Tuan Luu. 2025. Full-step-DPO: Self-supervised preference optimization with step-wise rewards for mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24343–24356, Vienna, Austria. Association for Computational Linguistics. - Xu et al. (2024) Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Zhao Wenyi, Jie Tang, and Yuxiao Dong. 2024. ChatGLM-math: Improving math problem-solving in large language models with a self-critique pipeline. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9733–9760, Miami, Florida, USA. Association for Computational Linguistics. - Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. - Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. - Yoon et al. (2025) Jaesik Yoon, Hyeonseo Cho, Doojin Baek, Yoshua Bengio, and Sungjin Ahn. 2025. Monte carlo tree diffusion for system 2 planning. In Forty-second International Conference on Machine Learning. - Zhang et al. (2024) Xuan Zhang, Chao Dut, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024. Chain of preference optimization: improving chain-of-thought reasoning in llms. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 333–356. - Zhihong et al. (2024) Shao Zhihong, Wang Peiyi, Zhu Qihao, Xu Runxin, Song Junxiao, Zhang Mingchuan, Y. K. Li, Y. Wu, and Guo Daya. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Appendix A Data Statistics We primarily evaluate our DiffCoT on mathematical reasoning benchmarks, specifically SVAMP (Patel et al., 2021), MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021). The MATH is divided into five difficulty levels, as defined in the original dataset. To reduce randomness and better showcase the experimental results, we sample test instances separately from each of five levels. To maintain a reasonable computational budget, particularly given the high cost of Tree-of-Thought (Yao et al., 2023) search, we restrict each dataset to at most 300 test samples through random sampling. For training, we similarly sample 300 instances from each dataset to construct the training set. For a fair comparison, all baseline methods use the same dataset configuration. We provide the detailed data statistics in Table 3. | GSM8K SVAMP MATH-L1 | 5 3 3 | 500 500 300 | 300 300 300 | 5 5 5 | 8 8 8 | | --- | --- | --- | --- | --- | --- | | MATH-L2 | 4 | 300 | 300 | 5 | 8 | | MATH-L3 | 4 | 300 | 300 | 5 | 8 | | MATH-L4 | 5 | 300 | 300 | 5 | 8 | | MATH-L5 | 5 | 300 | 300 | 5 | 8 | Table 3: Dataset-specific configurations for reasoning experiments. $\#\mathrm{Step}\ K$ denotes the maximum number of reasoning steps; $\#\mathrm{Train}$ and $\#\mathrm{Test}$ denote the numbers of training and test samples, respectively; $\#\mathrm{Cand.}$ denotes the number of candidate thoughts sampled at each step; $\#\mathrm{Roll.}$ denotes the number of Monte-Carlo rollouts used for evaluating each candidate. MATH-L1–L5 correspond to increasing difficulty levels in the MATH dataset. Appendix B Implementation Details Data Generation via Rollout-Based Thought Search We generate candidate contexts by conducting an iterative thought search starting from each original problem. Specifically, at reasoning step $t$ , we maintain a prefix that contains all previously selected thoughts. Conditioned on the current prefix, we sample four candidate thoughts from the base model. In parallel, we query DeepSeek-V3.2 Liu et al. (2025a) to produce an additional reference golden candidate thought under the same prefix. This golden candidate is used only during data collection and is not available at inference time. Figure 5 provides a representative example illustrating the constructed candidate thoughts and their corresponding step-wise evaluations under this procedure. To evaluate each candidate thought, we estimate its utility via Monte-Carlo rollouts. Concretely, we perform $R=8$ independent rollouts conditioned on the concatenation of the prefix and the candidate thought. Each rollout is decoded until termination to obtain a final answer. We then compute the empirical success rate as the fraction of rollouts that yield the correct answer. We continue the search by greedily selecting the candidate with the highest success rate, appending it to the prefix, and repeating the above procedure until the reasoning process terminates. We provide a detailed example with step-wise reasoning annotations in Figure 5. Model Training For efficient fine-tuning, we employ Low-Rank Adaptation (LoRA) (Hu et al., 2022) with a rank of 8 and $\alpha=16$ , where LoRA adapters are inserted into the q_proj and v_proj linear projections of every self-attention layer. Model training is conducted using the DPO (Rafailov et al., 2023) loss with a regularization coefficient of $\beta=0.4$ , optimized by AdamW (Loshchilov and Hutter, 2017) with a cosine learning rate schedule. We train for 3 epochs with a learning rate of $1\mathrm{e}{-5}$ , a warm-up ratio of 0.05, and a global batch size of 32. During decoding, the temperature is fixed at $0.4$ . Compared results (p < 0.05 under t-test) are averaged over three random 3 runs. All experiments are conducted on NVIDIA A100 GPUs with 80GB memory. Training and inference on 500 GSM8K samples require approximately 11 GPU-hours in total. Baseline Implementation The backbone models are implemented by adopting the ‘Meta-Llama-3-8B https://huggingface.co/meta-llama/Meta-Llama-3-8B ’, ‘Qwen3-8B https://huggingface.co/Qwen/Qwen3-8B ’, and ‘Qwen3-4B-Instruct-2507 https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 ’ versions. For Full-Step-DPO, we do not retrain the Process Reward Model (PRM) introduced in Full-Step-DPO. In our implementation, we retain the proposed global step-wise loss formulation, but replace the PRM-based step rewards with empirical success rates estimated from Monte-Carlo rollouts. Specifically, we collect extra complete reasoning trajectories from rollouts and use their final outcomes to compute success-based step scores, which serve as substitutes for PRM outputs in the global loss. Moreover, for CPO and ToT baselines, we follow the implementation described in the CPO paper, where preference pairs are constructed via LLM self-evaluation of intermediate reasoning steps. This design choice highlights the fundamental distinction between CPO-style self-judgment-based supervision and Step-DPO-style supervision derived from execution or outcome feedback. Appendix C Discussion with Reinforcement Learning PO methods generally follow a paradigm: trajectory generation followed by trajectory-level reweighting. The model first expands a full reasoning path via longitudinal generation. Alignment is then applied horizontally across completed trajectories using either reinforcement learning or preference optimization, discouraging low-reward reasoning paths and favoring high-reward ones. While this paradigm of vertical generation followed by horizontal RL-based refinement has achieved broad empirical success, it inevitably suffers from error accumulation, as early mistakes in the generated trajectory can propagate and constrain subsequent optimization. Chen et al. (2025) theoretically showed that the denoising process of diffusion models, which gradually transforms low-quality samples into high-quality ones, is equivalent to preference optimization. Building on this insight, DiffCoT introduces a diffusion sliding window, tightly coupling longitudinal reasoning generation with horizontal preference optimization within a unified framework. From this perspective, we believe that future extensions of our method can naturally integrate reinforcement learning, leveraging RL to directly optimize the denoising dynamics of the diffusion sliding window. Appendix D Generative AI Usage AI assistants were used in a limited and supportive manner during the preparation of this manuscript, primarily for language polishing, formatting suggestions, and improving clarity of presentation. Question. In a truck, there are 26 pink hard hats, 15 green hard hats, and 24 yellow hard hats. If Carl takes away 4 pink hard hats, and John takes away 6 pink hard hats and twice as many green hard hats as the number of pink hard hats that he removed, calculate the total number of hard hats that remained in the truck. Prompt. You are an expert math problem solver. You must reason step by step and avoid logical or arithmetic mistakes. Solve the following math problem. You MUST use exactly 5 reasoning steps. After the reasoning, output the final answer in the last line using the format: Answer: <final_answer>. Step 1. [0] 26 pink hard hats $-$ 4 pink hard hats $=$ 22 pink hard hats success_rate=0.0 [1] 26 $-$ 4 $=$ ? (number of pink hats after Carl’s removal) success_rate=0.125 [2] Initial number of pink hard hats is 26. success_rate=0.5 [3] We have 26 pink hard hats initially. Carl takes away 4 pink hard hats, leaving us with $26-4=22$ pink hard hats. success_rate=0.75 [4] Calculate the initial total number of hard hats. Initial: pink = 26, green = 15, yellow = 24. Total initial = 26 + 15 + 24 = 65. referenced data Step 2. [0] John removed 6 pink hats and 6 green hats, leaving $15-6=9$ green hats. success_rate=0.0 [1] John removes 6 pink hats, so he removes 6 green hats as well. Thus $15-6=9$ green remain. success_rate=0.125 [2] 6 pink hats are removed by John, leaving $22-6=16$ pink hats. success_rate=0.625 [3] John removes 6 pink hats: $22-6=16$ pink left. He removes twice as many green hats: $2× 6=12$ ,so $15-12=3$ green left. success_rate=1.0 [4] Carl removes 4 pink hard hats. Pink after Carl $26-4=22$ . referenced data Step 3. [0] Yellow hard hats remain unchanged, so we still have 24 yellow hard hats. success_rate=0.75 [1] Yellow hard hats remain unchanged, so we still have 24 yellow hard hats. success_rate=0.875 [2] 24 yellow hard hats remain in the truck and are not affected by the removals. success_rate=0.875 [3] 24 yellow hard hats remain unchanged. success_rate=1.0 [4] John removes 6 pink hard hats. Pink after John $22-6=16$ . referenced data Step 4. [0] 16 pink, 3 green, and 24 yellow hard hats remain in the truck. success_rate=0.75 [1] 16 pink and 3 green hard hats remain; $16+3=19$ . success_rate=0.875 [2] Total remaining hats: $16+3+24=43$ . success_rate=0.875 [3] 16 pink and 3 green hard hats remain, so $16+3=19$ hats across these two colors. success_rate=1.0 [4] John removes twice as many green hard hats as the pink hats he removed (he removed 6 pink hats, so twice that is 12 green hats). Green initially $=15$ .Green after John $15-12=3$ . referenced data Step 5. [0] Total: $3+19+24=?$ success_rate=0.0 [1] Total remaining hats: $19+24=43$ . success_rate=0.875 [2] Combining 19 (pink+green) with 24 yellow gives $19+24=43$ . success_rate=1.0 [3] Total remaining hats: $19+24=43$ . success_rate=1.0 [4] Yellow hats remain unchanged at 24. Total remaining = pink $(16)+$ green $(3)+$ yellow $(24)=43$ . . Figure 5: Representative dataset example with step-wise reasoning annotations.