2601.03559

# DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs Abstract Chain-of-Thought (CoT) reasoning improves multi-step mathematical problem solving in large language models but remains vulnerable to exposure bias and error accumulation, as early mistakes propagate irreversibly through autoregressive decoding. In this work, we propose DiffCoT, a diffusion-styled CoT framework that reformulates CoT reasoning as an iterative denoising process. DiffCoT integrates diffusion principles at the reasoning-step level via a sliding-window mechanism, enabling unified generation and retrospective correction of intermediate steps while preserving token-level autoregression. To maintain causal consistency, we further introduce a causal diffusion noise schedule that respects the temporal structure of reasoning chains. Extensive experiments on three multi-step CoT reasoning benchmarks across diverse model backbones demonstrate that DiffCoT consistently outperforms existing CoT preference optimization methods, yielding improved robustness and error-correction capability in CoT reasoning. DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs Shidong Cao 1, Hongzhan Lin 1, Yuxuan Gu 2, Ziyang Luo 1, Jing Ma 1 1 Hong Kong Baptist University 2 Harbin Institute of Technology {cssdcao,cshzlin,majing}@comp.hkbu.edu.hk 1 Introduction Large Language Models (LLMs) (Brown et al., 2020; Achiam et al., 2023; Guo et al., 2025) have marked a major breakthrough in natural language processing, achieving competitive performance across a wide range of mathematical reasoning tasks. A widely adopted technique in LLMs, Chain-of-Thought (CoT) reasoning (Wei et al., 2022; Kojima et al., 2022), enhances mathematical problem-solving by decomposing tasks into step-by-step intermediate reasoning. However, CoT reasoning is highly sensitive to potential errors, where mistakes introduced in the early stages can propagate through later steps, often leading to incorrect final answers (Wang et al., 2023; Lyu et al., 2023). This highlights a central research challenge: reducing errors by guiding LLMs to identify and follow reliable reasoning paths in long thought chains. Recent work has sought to optimize the exploration of reasoning branches in LLMs to reduce errors in CoT, by coupling generation with external critics or search procedures, such as Monte Carlo Tree Search (MCTS), to prune faulty trajectories (Li et al., 2024; Qin et al., 2024; Xi et al., 2024). In contrast, Preference Optimization (PO) methods (Lai et al., 2024; Zhang et al., 2024; Xu et al., 2025) aligned the policy with implicit preferences distilled from reasoning chains, so that decoding itself favors correct trajectories rather than relying on post-hoc filtering. However, beyond the substantial inference-time overhead of MCTS, these methods share a fundamental limitation that motivates rethinking of the CoT paradigm. Specifically, they all execute CoT reasoning in a strictly step-by-step manner, with each step conditioned on previous ones. Under static teacher-forcing supervision, the model is exposed only to correct prefixes during training, whereas at inference time the prefixes may contain errors that accumulate across steps, resulting in exposure bias that commonly plagues autoregressive generation (Bengio et al., 2015) and imitation learning (Ross et al., 2011). <details> <summary>x1.png Details</summary>  ### Visual Description \n ## Diagram: Chain of Thought (CoT) Comparison - Existing Methods vs. DiffCoT ### Overview The image presents a comparative diagram illustrating two approaches to Chain of Thought (CoT) reasoning: "Existing Methods for CoT" (a) and "DiffCoT" (b). Both diagrams depict a tree-like structure representing the reasoning process, with nodes numbered 1 through 4. The diagrams are accompanied by a word problem and step-by-step solutions demonstrating how each method attempts to solve it. The diagram highlights the differences in how each method handles errors and refines its reasoning. ### Components/Axes The diagram is divided into two main sections, labeled (a) and (b). Each section contains a tree-like diagram, a word problem, and a series of steps outlining the solution process. * **Nodes:** Represent steps in the reasoning process, numbered 1-4. * **Arrows:** Indicate the flow of reasoning. * **Symbols:** * Circle with an arrow pointing inward: Input * Yellow Star: Indicates a step in the reasoning process. * Blue Circle: Chosen response. * Yellow Triangle with exclamation point: Rejected response. * Red 'X' in a circle: Error response. * Green Checkmark: Correct response. * **Labels:** "Existing Methods for CoT", "DiffCoT", "Input", "Chosen response", "Rejected response", "Error response", "Diffusion", "AR", "Initial response", "Refined response", "Final response". * **Text Blocks:** Word problem, step-by-step solutions for both methods. ### Detailed Analysis or Content Details **Section (a): Existing Methods for CoT** * **Word Problem:** "Question: Mrs. Snyder used to spend 40% of her monthly income on rent and utilities. Her salary was recently increased by $600 so her new and utilities only amount to 25% of her monthly income. How much was her previous monthly income?" * **Step 1:** "Let the previous monthly income be x. The increased income is x + 600." * **Step 2:** "25% of x + 600 is the rent and utilities now. 25% of x = 0.25(x + 600)" * **Step 3:** "0.4x = 25% of x + 600. Subtract 25% of x from both sides to get 0.15x = 600." * **Step 4:** "Rearrange the equation to get x = 4000." * **Reasoning Path:** The diagram shows a path from Input (node 1) to node 4. Node 2 is marked with a yellow triangle (rejected response), and node 3 is marked with a red 'X' (error response). The chosen response is node 4 (blue circle). **Section (b): DiffCoT** * **Word Problem:** Same as in section (a). * **Step 3:** "0.4x = 25% of x + 600. Subtract 25% of x from both sides to get 0.15x = 600." * **Step 4:** "Rearrange the equation to get x = 4000." * **Refined Step 3:** "0.25(x + 600) = 0.4x. Solve for x" * **Refined Step 4:** "0.15x = 150. Solve for x." * **Refined Step 4:** "0.15 x = 150. Divide both sides by 0.15 x = 1000" * **Reasoning Path:** The diagram shows an initial path from Input (node 1) to node 4, marked with a red 'X' (error response). A "Diffusion" arrow points to a refined path, starting from node 2, going through node 3 (marked with a green checkmark - correct response), and finally to node 4 (blue circle - final response). The initial response is node 1, the refined response is node 3, and the final response is node 4. ### Key Observations * **Error Handling:** Existing Methods directly lead to an error (node 3 in section a) and then correct it in the final step. DiffCoT uses a "Diffusion" process to refine the reasoning and correct the error earlier in the process (node 3 in section b). * **Refinement:** DiffCoT explicitly shows the refinement steps (Refined Step 3 & 4), demonstrating how the reasoning is adjusted based on the initial error. * **Visual Representation:** The use of colors and symbols effectively highlights the differences in how each method handles errors and refines its reasoning. ### Interpretation The diagram illustrates a key difference between traditional Chain of Thought methods and the proposed DiffCoT approach. Existing methods proceed linearly, potentially leading to errors that are only corrected at the final step. DiffCoT, on the other hand, incorporates a "Diffusion" mechanism that allows for earlier error detection and refinement of the reasoning process. This is visually represented by the branching paths and the correction of the error at node 3 in section (b). The refined steps demonstrate a more iterative and robust approach to problem-solving. The diagram suggests that DiffCoT is more likely to arrive at the correct answer by proactively addressing potential errors during the reasoning process. The word problem itself is a standard algebraic problem, and the diagram focuses on *how* the solution is reached, rather than the solution itself. The diagram is a conceptual illustration of a methodology, not a presentation of empirical data. </details> Figure 1: Comparison of our proposed DiffCoT with existing CoT reasoning approaches: (a) Existing step-by-step CoT Reasoning methods adopt teacher-forcing training, where each step depends on the ground-truth output of the previous one. At inference time, this assumption breaks, causing exposure bias and leading to error accumulation. (b) DiffCoT performs CoT reasoning along both the noise (diffusion) and temporal (autoregressive) dimensions, enabling iterative correction of prior mistakes and effectively mitigating exposure bias. To mitigate the limitations above, we formulate CoT reasoning as a globally revisable trajectory, in which intermediate steps are not fixed once generated but can be adaptively corrected in light of later context. Rather than defining reasoning as a strictly forward, prefix-conditioned process driven by human priors (see Fig. 1), our method jointly models all reasoning steps within a diffusion paradigm (Dhariwal and Nichol, 2021), enabling robust reasoning under realistic noise from a global perspective with long-term revision guided by future signals. Starting from a corrupted or noisy reasoning chain, the entire reasoning trajectory can be iteratively updated, where intermediate revisions and forward progression are seamlessly integrated into a unified refinement process. By unifying iterative revision and forward continuation within the same diffusion-based refinement process, our design philosophy aim to reduce the discrepancy between training-time supervision and inference-time reasoning dynamics, leading to more stable reasoning behavior during inference. To this end, we propose Diff usion-Styled C hain o f T hought, DiffCoT, which integrates diffusion principles into the CoT framework to mitigate the exposure bias of static teacher-forcing and address the pointwise and local-step supervision issues of prior preference optimization solutions. Specifically, 1) DiffCoT modifies only the step-level generation strategy while preserving token-level autoregression, making it straightforward to fine-tune from existing autoregressive models. 2) By incorporating sliding windows and causal noise masks, DiffCoT unifies generation and revision within a versatile framework, to balance autoregressive and diffusion-styled reasoning. 3) Moreover, DiffCoT employs sentence-level noise injection to convert the reasoning paradigm into iterative denoising, progressively refining corrupted trajectories into coherent chains of thought, which captures the distributional evolution of reasoning errors to enable recovery from diverse and compounding mistakes. Altogether, DiffCoT could establish a unified and adaptive paradigm to CoT reasoning in LLMs. Our contributions are summarized as follows: - We propose DiffCoT, a novel diffusion-styled CoT framework that reformulates multi-step reasoning as an iterative denoising process, effectively mitigating exposure bias and error accumulation in autoregressive CoT reasoning. - We introduce a step-level diffusion-based learning strategy that unifies generation and revision through a sliding-window mechanism, enabling retrospective correction of earlier reasoning steps while preserving token-level autoregression. - We design a causal noise schedule that explicitly encodes the temporal dependency of reasoning steps, balancing global error correction with the causal structure required for coherent reasoning. - Extensive experiments on three public mathematical reasoning benchmarks demonstrate that DiffCoT outperforms State-of-The-Art (SoTA) PO methods, yielding robust gains and substantially improved error-correction capability in CoT. 2 Preliminaries Chain-of-Thought Reasoning Given a question prompt $p$ , the CoT paradigm (Wei et al., 2022) explicitly unfolds the reasoning process into a sequence of reasoning steps $s_{1:K}$ whose final element $s_{K}$ corresponds to the answer $a$ . The conditional distribution of a complete reasoning trace can be expressed as: $$ p_{\theta}(s_{1:K}\mid p)=\prod_{t=1}^{K}\pi_{\theta}(s_{k}\mid p,s_{<k}), \tag{1} $$ where $s_{k}$ denotes the $k$ -th reasoning step, $k≤ K$ is the step budget, and $\pi_{\theta}$ is the Auto-Regressive (AR) policy, i.e., the conditional distribution over the next step given the prompt $p$ and the previously generated steps. On annotated CoT data $(p,s_{1:K})$ , the model is trained by maximizing the conditional likelihood: $$ \mathcal{L}_{\mathrm{CoT}}=-\sum_{k=1}^{K}\log\pi_{\theta}(s_{k}\mid p,s_{<k}). \tag{2} $$ Diffusion Models Diffusion models (Dhariwal and Nichol, 2021) define a generative framework where data samples are gradually corrupted into a tractable noise distribution through a forward process, and a model is trained to approximate the corresponding reverse process. Let $x_{0}\sim\varphi(x_{0})$ denote a data sample from real data distribution $\varphi$ . The forward process produces a sequence $\{x_{t}\}_{t=1}^{T}$ by applying a noise operator $\mathcal{O}$ at each step: $$ x_{t}=\mathcal{D}(x_{t-1},\eta_{t}),\quad t=1,\dots,T, \tag{3} $$ where $t$ denotes the diffusion step index, $\eta_{t}$ is a noise variable. $\mathcal{O}$ could be a deterministic or structured corruption operator. The generative process is defined by learning a parameterized reverse transition $p_{\theta}(x_{t-1}\mid x_{t})$ that reconstructs the original sample through stepwise denoising. Training minimizes the discrepancy between the predicted $\hat{x}_{t-1}=f_{\theta}(x_{t},t)$ and the ground-truth corrupted data $x_{t-1}$ obtained from the forward process: $$ \begin{split}&\mathcal{L}_{\mathrm{Diffusion}}=\\ &\mathbf{E}_{x_{0}\sim\varphi(x_{0}),\,t\sim\{1,\dots,T\}}\Big[\ell\big(f_{\theta}(x_{t},t),\,x_{t-1}\big)\Big],\end{split} \tag{4} $$ where $f_{\theta}(x_{t},t)$ predicts the reconstruction of $x_{t-1}$ from $x_{t}$ , $\ell(·,·)$ is a reconstruction loss. In inference, sampling begins from $x_{T}$ and iteratively applies the learned reverse transitions until $x_{0}$ is obtained. 3 Methodology During CoT reasoning, LLMs are typically trained only on correct trajectories (Wei et al., 2022), while at inference time they may condition on erroneous intermediate steps (Lyu et al., 2023). Moreover, step-wise optimization focuses on local alignment and ignores future signals within the reasoning trajectory, limiting global consistency (Zhang et al., 2024). Together, these issues lead to error accumulation (Yoon et al., 2025; Chen et al., 2024a), motivating a rethinking of CoT reasoning under preference optimization. To address these issues, we argue that effective CoT reasoning requires trajectory-level revision to enforce global consistency and recover from corrupted intermediate steps. As error accumulation leads to noisy reasoning chains at inference time, a mechanism for global recovery becomes essential (Lyu et al., 2023; Wang et al., 2023). Motivated by the robustness of diffusion models in reconstructing structured data from noise (Ho et al., 2020; Li et al., 2022), we aim to develop a diffusion-styled preference optimization paradigm to reformulate CoT reasoning. To this end, we propose Diff usion-styled C hain o f T hought (DiffCoT), which models CoT reasoning in a diffusion-styled framework. We first present a step-level forward noising process for CoT using reward-ranked candidates (§ 3.1), and then apply a diffusion sliding window to iteratively denoise past steps while generating new ones (§ 3.2). We further introduce causal diffusion noise to strengthen causal consistency across reasoning steps (§ 3.3). An overview is shown in Figure 2. 3.1 Diffusion-Styled Noising for CoT In this section, we design a diffusion-styled forward noising process for CoT reasoning at the reasoning-step level, where step-level noise is induced by ranking candidate responses for the same step according to their reward scores. Previous preference optimization methods overlooked the distribution between high-reward and low-reward reasoning steps and relied on isolated step-wise adjustments that fail to mitigate the exposure bias inherent in teacher forcing. To this end, in our principle, higher-reward candidates are treated as lower-noise reasoning states, while lower-reward candidates correspond to progressively higher-noise states in the forward process. This ranking forms a progression from clean to corrupted reasoning states under diffusion-styled noise, enabling distribution-aware modeling of reasoning steps. Specifically, we implement forward noising by first collecting CoT reasoning trajectories for each question to construct the training set $\mathcal{D}$ . As shown in Fig. 2 (a), given a question prompt $p$ , we follow the standard CoT data generation process by employing MCTS (Browne et al., 2012), to gradually build a search tree for step-by-step solution exploration. The root corresponds to $p$ , and each child node represents an intermediate step $s$ . A complete path from the root to a terminal node $s_{K}$ yields a trajectory $\mathbf{tra}=p\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{K}$ , where each step $s_{k}$ is annotated with its reward $r(s_{k})$ . In contrast to conventional CoT trajectory construction Zhang et al. (2024), at each step $k$ we collect several candidate responses. These responses are scored using either an external reward model or rollout-based success rates. In our forward noising view, the candidate with the highest reward, denoted as $s^{\sigma^{0}=w}_{k}$ , is regarded as the lowest-noise state for step $k$ , and the remaining candidates are ordered by their rewards to form $\{s^{\sigma^{1:T}}_{k}\}$ , which we interpret as states with gradually increasing noise. Step-level noise is thus measured by the deviation of lower-reward responses from this best trajectory, indicating the corruption level at the step granularity. This design ensures that the final generation remains coherent while providing a set of forward diffusion states for subsequent denoising and preference optimization. Under this step-level forward noising, DiffCoT defines a diffusion-styled distribution that spans low-noise to high-noise reasoning states (Chen et al., 2025). Note that our data construction does not distinguish between “correct” and “incorrect” labels at the collection stage as in (Lai et al., 2025); instead, it uniformly gathers diverse responses for each step throughout the entire trajectory. This global view provides distribution-aware data to support both generation and refinement. In this way, error patterns are implicitly encoded in trajectory-level modeling, allowing robust reasoning without additional manual annotations of corrective steps. 3.2 Reformulating CoT Reasoning as Denoising <details> <summary>figs/method.png Details</summary>  ### Visual Description \n ## Diagram: Diffusion Sliding Window and Trajectory Analysis ### Overview The image presents a diagram illustrating a diffusion sliding window approach for reinforcement learning, likely within the context of Direct Preference Optimization (DPO). It compares a "Best Trajectory" approach with a "Diffusion Sliding Window" method, and then visualizes the underlying diffusion model. The diagram is divided into three sections: (a) Best Trajectory, (b) Diffusion Sliding Window, and (c) Diffusion Model. ### Components/Axes * **Section (a): Best Trajectory** * Nodes labeled 1, 2, 3, and 4 representing states. * Circles representing the "Best Trajectory" {s_1:4}. * Parentheses representing initial state P. * Braces containing state sequences {s₁^p,n,l}, {s₂^p,n,l}, {s₃^p,n,l}, {s₄^p,n,l}. * An arrow labeled "Sort by Reward" pointing downwards. * **Section (b): Diffusion Sliding Window** * Nodes labeled 1, 2, 3, and 4 representing states. * Circles representing state sequences s₁^w, s₂^w, s₃^w, s₄^w. * Parentheses representing initial state P. * States labeled with superscripts 'p' and 'n' (s₁^p, s₂^p, s₃ⁿ, s₄ⁿ). * States labeled with 'l' (s₄^l). * A legend indicating "DPO lose" (red) and "DPO win" (blue) represented by a tangled line graph labeled "LLM". * **Section (c): Diffusion Model** * Vertical axis labeled "Reward (Denoising)". * Horizontal axis labeled "Denoising at reward level". * States labeled S₁, S₂, S₃, and S₄. * A heatmap representing the diffusion model. * Lines representing "AR" (AutoRegressive) and "DiffCoT" (Diffusion Chain of Thought). * "DM" label. ### Detailed Analysis or Content Details **Section (a): Best Trajectory** The diagram shows a sequence of states 1, 2, 3, and 4 forming the best trajectory. The initial state is denoted by P. Below this, multiple possible trajectories are shown, each starting from P and leading to the states 1 through 4, labeled with {s₁^p,n,l}, {s₂^p,n,l}, {s₃^p,n,l}, {s₄^p,n,l}. The arrow "Sort by Reward" indicates that these trajectories are ranked based on their associated rewards. **Section (b): Diffusion Sliding Window** This section illustrates the diffusion sliding window approach. Two trajectories are shown, starting from P. The first trajectory is labeled s₁^w, s₂^w, s₃^w, s₄^w. The second trajectory has states labeled with 'p' and 'n' indicating preference. s₁^p, s₂^p, s₃ⁿ, s₄ⁿ. The state s₄^l is also shown. The legend indicates that the red lines represent "DPO lose" and the blue lines represent "DPO win", visualized as a complex line graph labeled "LLM". **Section (c): Diffusion Model** The heatmap shows the diffusion model's representation of reward. The vertical axis represents the reward level, and the horizontal axis represents the denoising level. The states S₁, S₂, S₃, and S₄ are positioned along the vertical axis. The heatmap is shaded, with darker shades indicating higher values. The line labeled "AR" slopes upwards, and the line labeled "DiffCoT" also slopes upwards, but with a steeper gradient. The point where the lines intersect is labeled "DM". The x-axis is labeled "Denoising at step level" and the y-axis is labeled "Reward (Denoising)". The inequality l < p < n < w is shown along the y-axis. ### Key Observations * The "Best Trajectory" section highlights the concept of selecting the optimal path based on reward. * The "Diffusion Sliding Window" section introduces a method for exploring multiple trajectories and evaluating their preferences (DPO win/lose). * The "Diffusion Model" section visualizes how the model represents reward and denoising levels. * The heatmap in section (c) suggests a correlation between reward and denoising. * The lines AR and DiffCoT represent different approaches to denoising, with DiffCoT exhibiting a stronger relationship between denoising and reward. ### Interpretation The diagram illustrates a novel approach to reinforcement learning using diffusion models and Direct Preference Optimization (DPO). The "Diffusion Sliding Window" method appears to be a way to sample multiple trajectories and learn from preferences, as indicated by the "DPO win/lose" labels. The diffusion model, visualized in section (c), provides a representation of the reward landscape, allowing for more informed denoising and trajectory selection. The comparison between "AR" and "DiffCoT" suggests that the latter approach is more effective at capturing the relationship between denoising and reward. The inequality l < p < n < w likely represents a ranking or ordering of different parameters or stages within the diffusion process. The overall goal seems to be to improve the efficiency and effectiveness of reinforcement learning by leveraging the power of diffusion models and preference-based learning. The diagram suggests a system where the LLM is used to determine the preference between trajectories, and this information is then used to refine the diffusion model. The heatmap in section (c) is a visual representation of the model's understanding of the reward function. </details> Figure 2: DiffCoT Framework and Training Data Construction: (a) Step-level forward noising: MCTS-based data generation defines step-level noise by reward-ranking multiple candidates, yielding states ranging from clean to corrupted. (b) Sliding-window denoising: a diffusion sliding window refines previously generated CoT steps while producing the next step in an autoregressive manner. (c) Causal diffusion noise: a step-dependent schedule assigns stronger noise to later steps to encode the causal order of the reasoning chain. In this section, our DiffCoT framework reformulates CoT reasoning beyond the teacher-forcing paradigm as a diffusion-styled denoising process to mitigate exposure bias at inference time. To couple this with the autoregressive nature of CoT, we introduce a diffusion sliding-window mechanism that operates directly on the CoT reasoning process. Within this sliding window, previously generated CoT steps are progressively denoised from high-noise toward low-noise reasoning states, thereby facilitating self-correction, while advancing the window naturally aligns with the autoregressive generation of subsequent CoT steps. Thus, instead of costly training a separate diffusion language model from scratch, we leverage diffusion-styled data to efficiently fine-tune pre-trained LLM $\pi$ for CoT reasoning. Specifically, for the iterative denoising process, the model $\pi$ takes the question prompt $p$ together with the preceding reasoning steps $\mathbf{s}_{1:k}$ as input. To enable variable-length generation when applying diffusion to thought chains while integrating generation with self-correction, the model maintains a diffusion sliding window of size $m$ and stride $n$ . At denoising iteration $t$ , the window containing previously generated CoT steps $\{s_{k-m}^{\sigma},...,s_{k}^{\sigma}\}$ is updated to a lower-noise version $\{s_{k-m}^{\sigma^{\prime}},...,s_{k}^{\sigma^{\prime}}\}$ , where $\sigma=\sigma_{k}^{(t)}$ is defined as a function of the step index $k$ and the denoising iteration $t$ , while $\sigma$ and $\sigma^{\prime}$ denote the noise levels before and after refinement, respectively. Simultaneously, as the window shifts forward by one step, the model predicts the next step $s_{k+1}^{\sigma}$ , initialized at a high-noise state. Iterating this denoising process eventually yields a clean trajectory $s_{1:K}^{\sigma^{0}}$ : $$ \pi_{\theta}\!\big(p,\,\mathbf{s}_{1:k}^{\,\sigma}\big)\;\mapsto\;\underbrace{\mathbf{s}_{k-m:k}^{\,\sigma^{\prime}}}_{\text{refined past}}\;\oplus\;\underbrace{\mathbf{s}_{k+1}^{\,\sigma}}_{\text{predicted future}}\,. \tag{5} $$ An illustration of the denoising process is shown in Fig. 2 (b). In sequence modeling, teacher-forcing next-token prediction can be viewed as masking along the reasoning-step axis $s$ , whereas diffusion corresponds to masking along the noise axis $\sigma$ , where $\sigma^{T}$ approaches pure white noise after $T$ iterations. To unify both views, we denote $s_{k}^{\sigma}$ as the $k$ -th CoT step under noise level $\sigma$ , where $\sigma$ is instantiated as $\sigma_{k}^{t}$ , i.e., the diffusion noise strength assigned to the $k$ -th step at denoising iteration $t$ . Building on the above reasoning process, we now introduce the training objective in DiffCoT. Our goal is to optimize a model $\pi_{\theta}$ that takes the question prompt $p$ together with the preceding reasoning steps $\mathbf{s}_{1:k}$ as input. We construct the win sequence $s_{k-m:k+1}^{w}$ by combining the denoised steps $\{s_{k-m}^{\sigma^{\prime}},...,s_{k}^{\sigma^{\prime}}\}$ with a lower-noise variant of $s_{k+1}^{\sigma_{k+1}^{\prime}}$ . Similarly, the lose sequence $s_{k-m:k+1}^{l}$ is formed by combining the unrefined steps $\{s_{k-m}^{\sigma},...,s_{k}^{\sigma}\}$ with the highest-noise variant of $s_{k+1}^{\sigma_{k+1}}$ . The prefix condition is the past text $s_{1:k-1}$ . To optimize the LLM on this preference pair, we adopt the Direct Preference Optimization (DPO) loss (Rafailov et al., 2023): $$ \begin{split}\mathcal{L}_{i}(\pi_{\theta};\pi_{\textrm{ref}})=-\log\phi\Big(&\beta\log\frac{\pi_{\theta}(s_{:k+1}^{w}\mid p,s_{1:k})}{\pi_{\textrm{ref}}(s_{:k+1}^{w}\mid p,s_{1:k})}\\ -\beta\log&\frac{\pi_{\theta}(s_{:k+1}^{l}\mid p,s_{1:k})}{\pi_{\textrm{ref}}(s_{:k+1}^{l}\mid p,s_{1:k})}\Big),\end{split} \tag{6} $$ where $s_{:k+1}$ denotes the subsequence $s_{k-m:k+1}$ , $\phi(·)$ denotes the sigmoid function, and $\beta$ is a factor controlling the strength of the preference signal. Note that our conditional prefix $s_{1:k-1}^{\sigma}$ differs from that used in standard DPO: instead of being composed exclusively of preferred (i.e., clean) steps, it combines clean steps from $1$ to $k-m$ with noisy steps within the sliding window. In this manner, the hybrid construction could alleviate the exposure bias by training the model to make preference-consistent updates even when conditioned on partially corrupted reasoning prefixes. 3.3 Causal Diffusion Noise Modeling CoT reasoning with diffusion poses a significant challenge to causality. Conventional full-sequence diffusion models are inherently non-causal (Ho et al., 2020; Dhariwal and Nichol, 2021), which contrasts sharply with the causal nature of CoT reasoning. While our backbone $\pi$ retains an AR token-level generation process, prior studies on diffusion models indicate that relying solely on the model’s ability to capture causality is inadequate, especially when reasoning must be performed over noisy or perturbed data (Li et al., 2022). Inspired by Diffusion Forcing (Chen et al., 2024a), as shown in Fig. 2 (c), we leverage noise as a mechanism to inject causal priors into diffusion sliding window.In conventional full-sequence diffusion, the noise strength $\sigma^{(t)}$ depends solely on the denoising iteration $t$ and is shared across all tokens. Such uniform noise injection is ill-suited for CoT reasoning, as it limits the model’s ability to capture step-wise causal dependencies. To overcome this, we redefine the noise schedule as $\sigma^{t}_{k}$ , a joint function of reasoning step $k$ and iteration $t$ (see § 3.2). Within the diffusion sliding window, $\sigma^{t}_{k}$ follows a progressive schedule, where earlier steps are perturbed with weaker noise, while later steps are perturbed with stronger noise. The noise schedule is formally defined as follows: $$ \begin{split}&F(s_{j-m+1},\ldots,s_{j};j)=\\ &\big(s_{j-m+1}^{\sigma^{0}},\;s_{j-m+2}^{\sigma^{1}},\;\ldots,\;s_{j}^{\sigma^{T}}\big),\end{split} \tag{7} $$ where the diffusion sliding window has size $m$ and stride $1$ , and at the $j$ -th denoising iteration the window advances to generate $s_{j}$ . In this manner, our framework could better stabilize the causal chain and enhance self-correction in subsequent reasoning steps even if the current step is erroneous. 4 Experiment | GSM8K SVAMP M-L1 | 37.2 49.6 31.3 | 39.3 50.1 33.4 | 37.6 49.4 32.2 | 37.7 48.8 31.5 | 36.1 48.6 33.8 | 38.1 48.4 32.0 | 39.6 50.4 34.3 | 65.5 83.5 61.1 | 64.3 84.9 62.0 | 65.7 84.2 60.1 | 65.1 82.7 60.8 | 64.9 85.2 61.7 | 65.9 85.9 63.0 | 66.2 85.5 63.6 | 62.0 80.1 51.2 | 64.0 80.7 52.4 | 61.8 82.3 51.8 | 63.0 80.3 51.6 | 63.3 79.4 51.8 | 64.7 83.0 52.7 | 65.4 83.2 53.3 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | M-L2 | 18.4 | 18.5 | 18.1 | 17.9 | 19.3 | 17.6 | 19.5 | 39.1 | 39.2 | 38.4 | 38.7 | 39.9 | 37.1 | 39.3 | 31.9 | 34.0 | 32.6 | 33.1 | 35.7 | 34.2 | 35.3 | | M-L3 | 12.5 | 13.2 | 12.6 | 12.9 | 12.6 | 13.7 | 14.2 | 25.2 | 26.1 | 25.3 | 25.5 | 26.2 | 25.8 | 26.9 | 16.0 | 17.0 | 16.2 | 16.4 | 17.2 | 17.9 | 18.0 | | M-L4 | 5.9 | 6.1 | 6.6 | 6.4 | 6.7 | 7.6 | 8.0 | 16.5 | 15.0 | 15.5 | 15.6 | 15.8 | 17.4 | 17.5 | 9.0 | 10.2 | 8.3 | 8.5 | 8.9 | 9.9 | 10.5 | | M-L5 | 1.9 | 2.3 | 1.4 | 2.9 | 3.0 | 3.1 | 3.3 | 4.4 | 5.4 | 5.2 | 3.7 | 4.2 | 4.7 | 5.5 | 1.0 | 1.2 | 0.7 | 0.7 | 2.4 | 1.4 | 2.9 | Table 1: Test accuracy (%) on GSM8K, SVAMP, and MATH. M-L1 to M-L5 denote different difficulty levels in the MATH dataset, where L1 is the easiest and L5 is the most challenging, while Step denotes Step-DPO and FStep denotes Full-Step-DPO. The best and second results are in bold and underlined. 4.1 Settings Models and Datasets We conduct experiments on three representative backbone models: Llama3-8B (Touvron et al., 2023), Qwen3-8B (Yang et al., 2025), and Qwen3-4B (Yang et al., 2025). We primarily evaluate our DiffCoT on three public mathematical reasoning benchmarks, specifically GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), and MATH (Hendrycks et al., 2021). The MATH is divided into five difficulty levels, as defined in the original dataset. More data statistics and preparation are provided in Appendix § A. Baselines We compare DiffCoT with the following SoTA baselines: 1) CoT (Wei et al., 2022): generates step-by-step reasoning before the final answer, evaluated with greedy decoding. 2) ToT (Yao et al., 2023): explores multiple reasoning paths via tree search. 3) TS-SFT (Feng et al., 2023): applies supervised fine-tuning on reasoning paths obtained from ToT. 4) CPO (Zhang et al., 2024): performs preference optimization at the step level, directly aligning intermediate reasoning steps via contrastive pairs. 5) Step-DPO (Lai et al., 2025): step-level preference optimization by explicitly collecting erroneous steps and applying preference optimization to correct them. 6) Full-Step-DPO (Xu et al., 2025): further generalizes step-wise optimization to the entire reasoning trajectory. More implementation details regarding model training and data construction are provided in Appendix § B. 4.2 Main Results As shown in Table 1, DiffCoT can be directly applied to modern instruction-tuned language models, such as Qwen3 and Llama3, via standard fine-tuning. This design enables seamless integration into existing training pipelines and allows DiffCoT to deliver consistent performance improvements across models of varying sizes, demonstrating strong scalability across different base models. From the result, we can observe that: 1) Baselines guided by LLM self-verification signals, such as CPO and ToT, exhibit notable instability across models and datasets. For example, when applied to Qwen3-8B, CPO results in performance degradation on the MATH-1 dataset while achieving competitive performance on SVAMP, indicating sensitivity to both model and task distributions. 2) Step-level preference learning baselines, including Step-DPO and Full-Step-DPO, generally achieve great improvements but still suffer from occasional performance drops under certain settings. This suggests that relying solely on local step-wise optimization may not be sufficient to ensure stable reasoning behavior across diverse evaluation scenarios. 3) Overall, DiffCoT consistently outperforms existing PO approaches across the majority of benchmarks and evaluation settings considered. Although it does not achieve the single best result in every individual setting, such as on SVAMP with the Qwen3-8B backbone, where Full-Step-DPO attains slightly higher accuracy, DiffCoT consistently ranks among the top-performing methods across all settings. Moreover, the improvements achieved by DiffCoT are substantial in magnitude and stable across different model sizes and datasets, highlighting its robustness and reliability. 4.3 Ablation Study To validate the effectiveness of the proposed DiffCoT method, we perform an ablation study on its key components. We select two representative base language models from different model families with varied sizes, Llama3-8B and Qwen3-4B, and evaluate them on two general mathematical reasoning benchmarks, GSM8K and SVAMP. To investigate the impact of incorporating diffusion into CoT reasoning, we evaluate various window sizes to explore how this factor influences the model’s performance. Specifically, when the diffusion window size and stride are set to 1, our approach essentially degenerates into an AR method, albeit with differently constructed prefixes. On the other hand, when the sliding diffusion window and stride are set to the number of steps $K$ , the method reverts to a purely diffusion-based approach. As shown in Table 2, it can be observed that performance degrades when the window size and stride is set to 1 or when it becomes too large. We attribute this phenomenon to a trade-off between causal connectivity and error-correction capability. Strengthening the ability to revise earlier steps often comes at the expense of weakening top-down causal reasoning. When the model operates entirely in AR mode, it exhibits the strongest causal reasoning ability but suffers from exposure bias during testing. In contrast, when the model fully adopts the diffusion mode, an excessively long window introduces noise: it disrupts the structural coherence of reasoning and prolonged denoising fluctuations undermine the causal nature of inference. | Llama3-8B - window size, stride=1 - window size, stride= $K$ | 39.6 36.3 -3.3 30.3 -9.3 | 50.4 48.3 -2.1 42.6 -7.8 | | --- | --- | --- | | - causal noise | 35.5 -4.1 | 48.2 -2.2 | | Qwen3-4B | 65.4 | 83.2 | | - window size, stride=1 | 63.0 -2.4 | 80.2 -3.0 | | - window size, stride= $K$ | 56.7 -8.7 | 73.9 -9.3 | | - causal noise | 61.9 -3.5 | 79.1 -4.1 | Table 2: Ablative results on the general datasets GSM8K and SVAMP, where $K$ is the number of reasoning steps. Numbers shown in the upper-right corner of each cell indicate the relative change in accuracy rate compared to the full DiffCoT model. We further conduct an ablation study to verify the effectiveness of the causal diffusion noise. Specifically, we disrupt our proposed noise scheduling by randomly shuffling the data order used for noising, instead of following the accuracy-based progression. This modification effectively breaks the causal structure of the noise. Results in Table 2 show that this ablation leads to substantial performance degradation across different models and datasets, demonstrating that our causal noise scheduling is a critical component of DiffCoT. 4.4 Analysis Case Study To further illustrate the model’s robustness to accumulated imperfections in intermediate reasoning, we present qualitative case studies that contrast effective and suboptimal reasoning trajectories. In particular, we focus on challenging examples in which the model introduces semantically irrelevant or weakly informative steps at an early stage of reasoning. Although such steps are not necessarily incorrect in isolation, they tend to accumulate and hinder progress toward the correct solution under standard step-by-step reasoning. As shown in Fig. 3, DiffCoT is able to progressively refine earlier reasoning steps through the diffusion process. Instead of rigidly committing to the initial reasoning prefix, the model gradually improves previously generated steps while producing subsequent ones, effectively revising semantically unhelpful content introduced earlier in the trajectory. This allows the model to move away from locally coherent but globally suboptimal reasoning paths and ultimately reach the correct solution. It highlights a key distinction between DiffCoT and conventional autoregressive reasoning. While autoregressive models tend to propagate early semantic noise forward without revision, DiffCoT leverages its denoising dynamics to refine the entire reasoning trajectory, including previously generated steps, resulting in more robust and globally consistent reasoning behavior. <details> <summary>figs/case.png Details</summary>  ### Visual Description \n ## Text Document: Download Time Calculation ### Overview The image presents a problem statement and a step-by-step reasoning process to solve it. The problem involves calculating the total time it takes to download a file given an interruption and a restart. The reasoning is presented in two sections: "Past Reasoning Steps" and "Next Generation". ### Components/Axes The document consists of: * **Question:** The initial problem statement. * **Past Reasoning Steps:** A series of calculations leading up to a certain point. * **Next Generation:** Continuation of the reasoning process. * **Step Numbers:** Each step is numbered sequentially. * **Mathematical Equations:** Calculations are presented as equations. ### Detailed Analysis or Content Details **Question:** "Carla is downloading a 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How long does it take to download the file?" **Past Reasoning Steps:** * **Step 1:** 40% of the way through the download is 0.4 * 200 GB = 80 GB. * **Step 2:** 80 GB / 2 GB per minute = 40 minutes to download the first 80 GB. * **Step 3:** 20 minutes for the restart. * **Step 4:** 200 GB - 80 GB = 120 GB remaining after the restart. **Next Generation:** * **Step 3:** 20 minutes for the restart, so total time so far is 40 + 20 = 60 minutes. * **Step 4:** Then she downloads the full 200 GB from scratch, taking 100 minutes after the restart starts. * **Step 5:** The total time can be calculated by adding the time before and after the interruption. ### Key Observations The "Next Generation" section re-numbers Step 3, which is a potential source of confusion. The final calculation is missing, but the setup is provided. The problem is solved by breaking it down into smaller steps, calculating the time for each part, and then summing them up. ### Interpretation The document demonstrates a problem-solving approach using sequential reasoning. The initial steps correctly calculate the progress before the interruption and the remaining data. The "Next Generation" section correctly accounts for the restart time and the full re-download. The missing final step would be to add the time before the interruption (40 minutes + 20 minutes = 60 minutes) and the time after the interruption (100 minutes) to get the total download time (160 minutes). The document highlights a common scenario in software downloads where interruptions can significantly increase the overall time required. The re-numbering of Step 3 in the "Next Generation" section is an error that could lead to confusion. The document is a clear example of how to approach a quantitative problem by breaking it down into smaller, manageable steps. </details> Figure 3: Example illustrating how DiffCoT modifies early-stage reasoning shift steps. The steps highlighted in blue represent the diffusion sliding window. <details> <summary>x2.png Details</summary>  ### Visual Description \n ## Line Charts: Accuracy vs. Noise Intensity Coefficient for Different Models ### Overview The image presents three separate line charts, each depicting the relationship between Accuracy (Acc(%) on the y-axis) and Noise Intensity Coefficient (ω) on the x-axis. Each chart represents a different model: Llama3-8B, Qwen3-8B, and Qwen3-4B. Within each chart, two lines represent different training methods: "DiffCoT" (blue line) and "Full-Step DPO" (orange line). The charts aim to compare the robustness of these models and training methods to increasing levels of noise. ### Components/Axes * **X-axis Label (all charts):** "Noise Intensity Coefficient ω" with markers at 0.0, 0.1, 0.2, 0.3, and 0.4. * **Y-axis Label (all charts):** "Acc(%)" with a scale ranging from approximately 15% to 70%. * **Chart Titles:** * Left: "Llama3-8B" * Center: "Qwen3-8B" * Right: "Qwen3-4B" * **Legend (bottom-left of each chart):** * Blue Line: "DiffCoT" * Orange Line: "Full-Step DPO" ### Detailed Analysis or Content Details **Llama3-8B (Left Chart):** * The "DiffCoT" line (blue) starts at approximately 38% accuracy at ω = 0.0 and decreases linearly to approximately 19% accuracy at ω = 0.4. * The "Full-Step DPO" line (orange) starts at approximately 36% accuracy at ω = 0.0 and decreases more steeply to approximately 17% accuracy at ω = 0.4. **Qwen3-8B (Center Chart):** * The "DiffCoT" line (blue) starts at approximately 67% accuracy at ω = 0.0 and decreases linearly to approximately 44% accuracy at ω = 0.4. * The "Full-Step DPO" line (orange) starts at approximately 65% accuracy at ω = 0.0 and decreases more steeply to approximately 40% accuracy at ω = 0.4. **Qwen3-4B (Right Chart):** * The "DiffCoT" line (blue) starts at approximately 63% accuracy at ω = 0.0 and decreases linearly to approximately 48% accuracy at ω = 0.4. * The "Full-Step DPO" line (orange) starts at approximately 62% accuracy at ω = 0.0 and decreases more steeply to approximately 38% accuracy at ω = 0.4. ### Key Observations * In all three charts, both training methods show a decrease in accuracy as the noise intensity coefficient increases. * "Full-Step DPO" consistently exhibits a steeper decline in accuracy compared to "DiffCoT" across all models and noise levels. * Qwen3-8B and Qwen3-4B generally achieve higher initial accuracy levels (at ω = 0.0) than Llama3-8B. * The rate of accuracy decline appears relatively consistent for "DiffCoT" across all models. ### Interpretation The data suggests that increasing noise negatively impacts the accuracy of all three models, regardless of the training method used. "Full-Step DPO" appears to be more sensitive to noise than "DiffCoT," as evidenced by the steeper decline in accuracy. This could indicate that "DiffCoT" provides a more robust training approach in the presence of noisy data. The higher initial accuracy of the Qwen models suggests they may have a stronger baseline performance compared to Llama3-8B. The linear relationship between noise intensity and accuracy decline suggests a predictable degradation in performance as noise levels increase. The consistent trends across all three models indicate that the observed behavior is likely not model-specific but rather a general characteristic of these types of language models when exposed to noise. The charts provide a comparative analysis of model robustness and training method effectiveness under varying noise conditions. </details> Figure 4: Correction success rate under stochastic prefix corruption, where noise is injected at the midpoint of the reasoning trajectory with probability $\omega$ . Error Accumulation Analysis We further analyze the model’s ability to recover from accumulated imperfections in intermediate reasoning steps. We consider a correction-oriented setting in which the model is deliberately conditioned on prefixes that contain semantically suboptimal or noisy reasoning steps, and is then required to continue the reasoning process to reach a correct final answer. To this end, we introduce controlled perturbations at an intermediate stage of the reasoning process. Specifically, when the reasoning reaches approximately half of the trajectory, each preceding step is independently perturbed with probability $\omega$ . A perturbation replaces the original step with a low-reward alternative sampled from model-generated trajectories that are semantically plausible but less aligned with the optimal reasoning path. The noise strength $\omega$ therefore governs the degree of accumulated semantic drift in the prefix, enabling a systematic evaluation of robustness under varying levels of intermediate reasoning noise. We compare different training strategies under this protocol using the correction success rate, defined as the proportion of cases in which the model produces a correct final answer despite being conditioned on a perturbed prefix. Experiments are conducted on 300 GSM8K problems with multiple backbone models, comparing our approach against Full-Step-DPO. As shown in Fig. 4, our method consistently achieves substantially higher correction success rates across all settings, demonstrating a stronger ability to compensate for accumulated intermediate noise and to maintain globally coherent reasoning, rather than brittle continuation of imperfect early steps. In other words, our model can mitigate the impact of early-stage reasoning drift, rather than rigidly propagating locally coherent but globally suboptimal trajectories in CoT reasoning. 5 Related Work Preference Learning Preference learning has recently emerged as an effective paradigm for aligning LLMs with human preferences by contrasting desirable and undesirable responses (Ouyang et al., 2022; Liu et al., 2025b), with methods such as DPO showing strong performance on general language tasks (Rafailov et al., 2023). However, extending these gains to mathematical reasoning remains challenging, largely due to the coarse granularity of solution-level supervision, which fails to localize and correct intermediate reasoning errors (Chen et al., 2024b). To address this limitation, recent work incorporates step-level preference signals to align intermediate reasoning processes (Lai et al., 2024; Lu et al., 2024), while Full-Step-DPO further optimizes entire reasoning trajectories using global objectives (Xu et al., 2025). Despite these advances, most existing step-wise methods adopt a teacher-forcing paradigm and reason exclusively over clean prefixes during training, leading to error accumulation and degraded robustness during inference. Accordingly, our work reconceptualizes CoT reasoning as a globally revisable trajectory, in which previously generated steps remain malleable to correction in light of future context. Mathematical Reasoning Mathematical reasoning is widely regarded as a challenging capability for large language models, and prior work has explored several directions to improve it. One line of research strengthens base models through continual pre-training on large-scale mathematical corpora or supervised fine-tuning on synthetic datasets distilled from stronger models (Azerbayev et al., 2023; Zhihong et al., 2024; Xu et al., 2024; Mitra et al., 2024). Another line improves test-time performance by increasing computational budgets, such as generating and reranking multiple solutions using outcome- or process-level rewards, or adopting reward-guided decoding strategies (Guan et al., 2025; Wu et al., 2024; Wang et al., 2024). Recently, reinforcement learning and preference-based optimization have been explored to directly align reasoning behaviors via trajectory- or step-level supervision, aiming to improve robustness beyond supervised objectives alone (Pal et al., 2024; Wang et al., 2024). But they largely operate in a forward-only manner and lack mechanisms for revising corrupted intermediate reasoning, motivating our diffusion-styled reformulation of CoT reasoning to enable effective error correction. 6 Conclusion and Future Work In this work, we proposed a novel DiffCoT framework to enhance mathematical reasoning and alleviate error accumulation by integrating diffusion steps into autoregressive generation. Our method utilizes a sliding diffusion window with causal noise to refine intermediate steps while maintaining consistent reasoning chains. Extensive experiments confirm that our proposed DiffCoT outperforms existing CoT methods, offering a robust solution for multi-step CoT reasoning. Future work will explore the scalability of our paradigm across more backbone models and broader reasoning domains. Limitations Although we have conducted extensive experiments on DiffCoT and observed strong empirical performance, several limitations remain. First, the data construction and training of DiffCoT follow an off-policy paradigm, where preference data are collected using a policy that differs from the one being optimized. Such a mismatch between the behavior policy and the training policy may introduce distribution shift, biased value estimation, and training instability, especially when scaling to more difficult datasets or longer reasoning chains. Second, similar to Diffusion Forcing, DiffCoT breaks the local Markov property of prefix conditioned generation by revisiting and modifying historical reasoning steps. While this violation enables stronger capabilities, it also increases the uncertainty and controllability challenges during generation, typically requiring more training iterations and larger amounts of data to achieve stable convergence. References - Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. - Azerbayev et al. (2023) Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.06786. - Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28. - Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS). - Browne et al. (2012) Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43. - Chen et al. (2024a) Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. 2024a. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125. - Chen et al. (2024b) Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024b. Step-level value preference optimization for mathematical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7889–7903, Miami, Florida, USA. Association for Computational Linguistics. - Chen et al. (2025) Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Ziyang Wang, Tony Quek, Joey Tianyi Zhou, Soujanya Poria, and Zuozhu Liu. 2025. DiffPO: Diffusion-styled preference optimization for inference time alignment of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18910–18925, Vienna, Austria. Association for Computational Linguistics. - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. - Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Quinn Nichol. 2021. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS). - Feng et al. (2023) Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2023. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179. - Guan et al. (2025) Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rStar-math: Small LLMs can master math reasoning with self-evolved deep thinking. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 20640–20661. PMLR. - Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. - Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. - Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS). - Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. - Kojima et al. (2022) Takeshi Kojima, Shixiang Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916. - Lai et al. (2024) Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629. - Lai et al. (2025) Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2025. Step-DPO: Step-wise preference optimization for long-chain reasoning of LLMs. - Li et al. (2022) Xiang Lisa Li, Tianyu Gong, Tian He, Dan Jurafsky, Percy Liang, and Steven CH Zhao. 2022. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems (NeurIPS). - Li et al. (2024) Yanhong Li, Chenghao Yang, and Allyson Ettinger. 2024. When hindsight is not 20/20: Testing limits on reflective thinking in large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3741–3753. - Liu et al. (2025a) Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025a. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. - Liu et al. (2025b) Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, and 1 others. 2025b. Lipo: Listwise preference optimization through learning-to-rank. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2404–2420. - Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. - Lu et al. (2024) Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning. CoRR. - Lyu et al. (2023) Qing Lyu, Tongshuang Wu, Fei Sha, and Graham Neubig. 2023. Faithful chain-of-thought reasoning. Preprint, arXiv:2301.13379. - Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. Preprint, arXiv:2402.14830. - Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. - Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228. - Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191. - Qin et al. (2024) Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and 1 others. 2024. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982. - Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems. - Ross et al. (2011) Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings. - Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. - Wang et al. (2024) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439, Bangkok, Thailand. Association for Computational Linguistics. - Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR). - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. - Wu et al. (2024) Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. 2024. Enhancing mathematical reasoning in llms by stepwise correction. Preprint, arXiv:2410.12934. - Xi et al. (2024) Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, and 1 others. 2024. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579. - Xu et al. (2025) Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, and Anh Tuan Luu. 2025. Full-step-DPO: Self-supervised preference optimization with step-wise rewards for mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24343–24356, Vienna, Austria. Association for Computational Linguistics. - Xu et al. (2024) Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Zhao Wenyi, Jie Tang, and Yuxiao Dong. 2024. ChatGLM-math: Improving math problem-solving in large language models with a self-critique pipeline. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9733–9760, Miami, Florida, USA. Association for Computational Linguistics. - Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. - Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822. - Yoon et al. (2025) Jaesik Yoon, Hyeonseo Cho, Doojin Baek, Yoshua Bengio, and Sungjin Ahn. 2025. Monte carlo tree diffusion for system 2 planning. In Forty-second International Conference on Machine Learning. - Zhang et al. (2024) Xuan Zhang, Chao Dut, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024. Chain of preference optimization: improving chain-of-thought reasoning in llms. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 333–356. - Zhihong et al. (2024) Shao Zhihong, Wang Peiyi, Zhu Qihao, Xu Runxin, Song Junxiao, Zhang Mingchuan, Y. K. Li, Y. Wu, and Guo Daya. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Appendix A Data Statistics We primarily evaluate our DiffCoT on mathematical reasoning benchmarks, specifically SVAMP (Patel et al., 2021), MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021). The MATH is divided into five difficulty levels, as defined in the original dataset. To reduce randomness and better showcase the experimental results, we sample test instances separately from each of five levels. To maintain a reasonable computational budget, particularly given the high cost of Tree-of-Thought (Yao et al., 2023) search, we restrict each dataset to at most 300 test samples through random sampling. For training, we similarly sample 300 instances from each dataset to construct the training set. For a fair comparison, all baseline methods use the same dataset configuration. We provide the detailed data statistics in Table 3. | GSM8K SVAMP MATH-L1 | 5 3 3 | 500 500 300 | 300 300 300 | 5 5 5 | 8 8 8 | | --- | --- | --- | --- | --- | --- | | MATH-L2 | 4 | 300 | 300 | 5 | 8 | | MATH-L3 | 4 | 300 | 300 | 5 | 8 | | MATH-L4 | 5 | 300 | 300 | 5 | 8 | | MATH-L5 | 5 | 300 | 300 | 5 | 8 | Table 3: Dataset-specific configurations for reasoning experiments. $\#\mathrm{Step}\ K$ denotes the maximum number of reasoning steps; $\#\mathrm{Train}$ and $\#\mathrm{Test}$ denote the numbers of training and test samples, respectively; $\#\mathrm{Cand.}$ denotes the number of candidate thoughts sampled at each step; $\#\mathrm{Roll.}$ denotes the number of Monte-Carlo rollouts used for evaluating each candidate. MATH-L1–L5 correspond to increasing difficulty levels in the MATH dataset. Appendix B Implementation Details Data Generation via Rollout-Based Thought Search We generate candidate contexts by conducting an iterative thought search starting from each original problem. Specifically, at reasoning step $t$ , we maintain a prefix that contains all previously selected thoughts. Conditioned on the current prefix, we sample four candidate thoughts from the base model. In parallel, we query DeepSeek-V3.2 Liu et al. (2025a) to produce an additional reference golden candidate thought under the same prefix. This golden candidate is used only during data collection and is not available at inference time. Figure 5 provides a representative example illustrating the constructed candidate thoughts and their corresponding step-wise evaluations under this procedure. To evaluate each candidate thought, we estimate its utility via Monte-Carlo rollouts. Concretely, we perform $R=8$ independent rollouts conditioned on the concatenation of the prefix and the candidate thought. Each rollout is decoded until termination to obtain a final answer. We then compute the empirical success rate as the fraction of rollouts that yield the correct answer. We continue the search by greedily selecting the candidate with the highest success rate, appending it to the prefix, and repeating the above procedure until the reasoning process terminates. We provide a detailed example with step-wise reasoning annotations in Figure 5. Model Training For efficient fine-tuning, we employ Low-Rank Adaptation (LoRA) (Hu et al., 2022) with a rank of 8 and $\alpha=16$ , where LoRA adapters are inserted into the q_proj and v_proj linear projections of every self-attention layer. Model training is conducted using the DPO (Rafailov et al., 2023) loss with a regularization coefficient of $\beta=0.4$ , optimized by AdamW (Loshchilov and Hutter, 2017) with a cosine learning rate schedule. We train for 3 epochs with a learning rate of $1\mathrm{e}{-5}$ , a warm-up ratio of 0.05, and a global batch size of 32. During decoding, the temperature is fixed at $0.4$ . Compared results (p < 0.05 under t-test) are averaged over three random 3 runs. All experiments are conducted on NVIDIA A100 GPUs with 80GB memory. Training and inference on 500 GSM8K samples require approximately 11 GPU-hours in total. Baseline Implementation The backbone models are implemented by adopting the ‘Meta-Llama-3-8B https://huggingface.co/meta-llama/Meta-Llama-3-8B ’, ‘Qwen3-8B https://huggingface.co/Qwen/Qwen3-8B ’, and ‘Qwen3-4B-Instruct-2507 https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507 ’ versions. For Full-Step-DPO, we do not retrain the Process Reward Model (PRM) introduced in Full-Step-DPO. In our implementation, we retain the proposed global step-wise loss formulation, but replace the PRM-based step rewards with empirical success rates estimated from Monte-Carlo rollouts. Specifically, we collect extra complete reasoning trajectories from rollouts and use their final outcomes to compute success-based step scores, which serve as substitutes for PRM outputs in the global loss. Moreover, for CPO and ToT baselines, we follow the implementation described in the CPO paper, where preference pairs are constructed via LLM self-evaluation of intermediate reasoning steps. This design choice highlights the fundamental distinction between CPO-style self-judgment-based supervision and Step-DPO-style supervision derived from execution or outcome feedback. Appendix C Discussion with Reinforcement Learning PO methods generally follow a paradigm: trajectory generation followed by trajectory-level reweighting. The model first expands a full reasoning path via longitudinal generation. Alignment is then applied horizontally across completed trajectories using either reinforcement learning or preference optimization, discouraging low-reward reasoning paths and favoring high-reward ones. While this paradigm of vertical generation followed by horizontal RL-based refinement has achieved broad empirical success, it inevitably suffers from error accumulation, as early mistakes in the generated trajectory can propagate and constrain subsequent optimization. Chen et al. (2025) theoretically showed that the denoising process of diffusion models, which gradually transforms low-quality samples into high-quality ones, is equivalent to preference optimization. Building on this insight, DiffCoT introduces a diffusion sliding window, tightly coupling longitudinal reasoning generation with horizontal preference optimization within a unified framework. From this perspective, we believe that future extensions of our method can naturally integrate reinforcement learning, leveraging RL to directly optimize the denoising dynamics of the diffusion sliding window. Appendix D Generative AI Usage AI assistants were used in a limited and supportive manner during the preparation of this manuscript, primarily for language polishing, formatting suggestions, and improving clarity of presentation. Question. In a truck, there are 26 pink hard hats, 15 green hard hats, and 24 yellow hard hats. If Carl takes away 4 pink hard hats, and John takes away 6 pink hard hats and twice as many green hard hats as the number of pink hard hats that he removed, calculate the total number of hard hats that remained in the truck. Prompt. You are an expert math problem solver. You must reason step by step and avoid logical or arithmetic mistakes. Solve the following math problem. You MUST use exactly 5 reasoning steps. After the reasoning, output the final answer in the last line using the format: Answer: <final_answer>. Step 1. [0] 26 pink hard hats $-$ 4 pink hard hats $=$ 22 pink hard hats success_rate=0.0 [1] 26 $-$ 4 $=$ ? (number of pink hats after Carl’s removal) success_rate=0.125 [2] Initial number of pink hard hats is 26. success_rate=0.5 [3] We have 26 pink hard hats initially. Carl takes away 4 pink hard hats, leaving us with $26-4=22$ pink hard hats. success_rate=0.75 [4] Calculate the initial total number of hard hats. Initial: pink = 26, green = 15, yellow = 24. Total initial = 26 + 15 + 24 = 65. referenced data Step 2. [0] John removed 6 pink hats and 6 green hats, leaving $15-6=9$ green hats. success_rate=0.0 [1] John removes 6 pink hats, so he removes 6 green hats as well. Thus $15-6=9$ green remain. success_rate=0.125 [2] 6 pink hats are removed by John, leaving $22-6=16$ pink hats. success_rate=0.625 [3] John removes 6 pink hats: $22-6=16$ pink left. He removes twice as many green hats: $2× 6=12$ ,so $15-12=3$ green left. success_rate=1.0 [4] Carl removes 4 pink hard hats. Pink after Carl $26-4=22$ . referenced data Step 3. [0] Yellow hard hats remain unchanged, so we still have 24 yellow hard hats. success_rate=0.75 [1] Yellow hard hats remain unchanged, so we still have 24 yellow hard hats. success_rate=0.875 [2] 24 yellow hard hats remain in the truck and are not affected by the removals. success_rate=0.875 [3] 24 yellow hard hats remain unchanged. success_rate=1.0 [4] John removes 6 pink hard hats. Pink after John $22-6=16$ . referenced data Step 4. [0] 16 pink, 3 green, and 24 yellow hard hats remain in the truck. success_rate=0.75 [1] 16 pink and 3 green hard hats remain; $16+3=19$ . success_rate=0.875 [2] Total remaining hats: $16+3+24=43$ . success_rate=0.875 [3] 16 pink and 3 green hard hats remain, so $16+3=19$ hats across these two colors. success_rate=1.0 [4] John removes twice as many green hard hats as the pink hats he removed (he removed 6 pink hats, so twice that is 12 green hats). Green initially $=15$ .Green after John $15-12=3$ . referenced data Step 5. [0] Total: $3+19+24=?$ success_rate=0.0 [1] Total remaining hats: $19+24=43$ . success_rate=0.875 [2] Combining 19 (pink+green) with 24 yellow gives $19+24=43$ . success_rate=1.0 [3] Total remaining hats: $19+24=43$ . success_rate=1.0 [4] Yellow hats remain unchanged at 24. Total remaining = pink $(16)+$ green $(3)+$ yellow $(24)=43$ . . Figure 5: Representative dataset example with step-wise reasoning annotations.