# Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
Abstract
Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) due to the extensive and precise chain of reasoning required for accuracy. Ensuring the correctness of each reasoning step is critical. To address this, we aim to enhance the robustness and factuality of LLMs by learning from human feedback. However, Direct Preference Optimization (DPO) has shown limited benefits for long-chain mathematical reasoning, as models employing DPO struggle to identify detailed errors in incorrect answers. This limitation stems from a lack of fine-grained process supervision. We propose a simple, effective, and data-efficient method called Step-DPO, which treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Additionally, we have developed a data construction pipeline for Step-DPO, enabling the creation of a high-quality dataset containing 10K step-wise preference pairs. We also observe that in DPO, self-generated data is more effective than data generated by humans or GPT-4, due to the latter’s out-of-distribution nature. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at https://github.com/dvlab-research/Step-DPO.
1 Introduction
<details>
<summary>x1.png Details</summary>
### Visual Description
## Scatter Plot: Accuracy on the MATH test set
### Overview
This scatter plot visualizes the accuracy of various language models on the MATH test set, plotted against their model size. The accuracy is measured in percentage (%), and the model size is measured in billions of parameters (B). Different models are represented by different colored markers (red triangles, green circles, and black squares).
### Components/Axes
* **Title:** Accuracy on the MATH test set
* **X-axis:** Model size (B). Scale ranges from approximately 5B to 75B. Markers are placed at 7B, 32B, 47B, 57B, 70B, and 72B.
* **Y-axis:** Accuracy (%). Scale ranges from approximately 42% to 72.5%. Markers are placed at 42.5%, 45%, 47.5%, 50%, 52.5%, 55%, 57.5%, 60%, 62.5%, 65%, 67.5%, 70%, and 72.5%.
* **Legend:** Implicitly defined by marker shape and color.
* Red Triangles: Models ending in "-DPO"
* Green Circles: Models ending in "-Instruct" or "-Instruct-SFT"
* Black Squares: Models without "-DPO" or "-Instruct" in their name.
### Detailed Analysis
The plot shows a general trend of increasing accuracy with increasing model size, but with significant variation between models.
* **Gemini-1.5-Pro:** Approximately 70.0% accuracy at 72B. (Red Triangle)
* **Qwen2-72B-Instruct-Step-DPO:** Approximately 70.5% accuracy at 72B. (Red Triangle)
* **Qwen2-72B-Instruct:** Approximately 69.5% accuracy at 72B. (Red Triangle)
* **Qwen2-72B-Step-DPO:** Approximately 68.5% accuracy at 72B. (Red Triangle)
* **GPT-4-1106:** Approximately 65.0% accuracy at 72B. (Red Triangle)
* **Claude-3-Opus:** Approximately 60.0% accuracy at 72B. (Red Triangle)
* **Llama3-70B-DPO:** Approximately 58.0% accuracy at 70B. (Red Triangle)
* **Qwen2-57B-A14B-Step-DPO:** Approximately 57.5% accuracy at 57B. (Red Triangle)
* **Qwen2-57B-A14B-Instruct:** Approximately 50.0% accuracy at 57B. (Green Circle)
* **Qwen2-32B-Step-DPO:** Approximately 57.5% accuracy at 32B. (Red Triangle)
* **Qwen1.5-32B-SFT:** Approximately 52.5% accuracy at 32B. (Green Circle)
* **Qwen2-7B-Step-DPO:** Approximately 55.0% accuracy at 7B. (Red Triangle)
* **Qwen2-7B-Instruct:** Approximately 50.0% accuracy at 7B. (Green Circle)
* **DeepSeekMath-RL:** Approximately 52.5% accuracy at 7B. (Green Circle)
* **DeepSeekMath-Instruct:** Approximately 47.5% accuracy at 7B. (Green Circle)
* **MathGenieLM-Mistral:** Approximately 45.0% accuracy at 7B. (Green Circle)
* **MAmmoTH2-Mixtral-8x7B:** Approximately 47.5% accuracy at 47B. (Black Square)
* **Llama3-70B-Instruct:** Approximately 50.0% accuracy at 70B. (Green Circle)
### Key Observations
* Models with "-DPO" in their name (red triangles) generally exhibit higher accuracy than those without, especially at larger model sizes.
* Qwen2-72B-Instruct-Step-DPO achieves the highest accuracy among the models shown.
* There's a noticeable gap in accuracy between models around the 32B and 57B size ranges.
* The models ending in "-Instruct" (green circles) show a relatively consistent accuracy across different model sizes.
* MAmmoTH2-Mixtral-8x7B is an outlier, showing lower accuracy than other models of similar size.
### Interpretation
The data suggests that model size is a significant factor in achieving higher accuracy on the MATH test set, but it is not the only one. The training methodology, as indicated by the "-DPO" suffix, appears to play a crucial role. Models trained with Direct Preference Optimization (DPO) consistently outperform others. The variation in accuracy among models of similar size highlights the importance of architectural choices and training data. The outlier, MAmmoTH2-Mixtral-8x7B, may indicate that its architecture or training process is less effective for this specific task. The plot demonstrates the ongoing progress in language model development, with newer models like Gemini-1.5-Pro and Qwen2-72B-Instruct-Step-DPO achieving state-of-the-art results. The consistent performance of the "-Instruct" models suggests that instruction tuning is a valuable technique for improving performance on tasks requiring following instructions.
</details>
Figure 1: Accuracy on the MATH test set across models fine-tuned by Step-DPO and other state-of-the-art models. †: reproduced result using our prompt.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Line Chart: Accuracy of judging preferred and undesirable outputs
### Overview
This line chart displays the accuracy of judging preferred and undesirable outputs for four different models (Qwen2-72B-Step-DPO, Qwen2-7B-Step-DPO, Qwen2-72B-DPO, and Qwen2-7B-DPO) over a range of training steps. The accuracy is measured in percentage (%) and plotted against the number of training steps.
### Components/Axes
* **Title:** Accuracy of judging preferred and undesirable outputs
* **X-axis:** Training steps (ranging from approximately 0 to 275)
* **Y-axis:** Accuracy (%) (ranging from approximately 64 to 84)
* **Data Series:**
* Qwen2-72B-Step-DPO (Purple line with triangle markers)
* Qwen2-7B-Step-DPO (Green line with circle markers)
* Qwen2-72B-DPO (Violet line with square markers)
* Qwen2-7B-DPO (Orange line with diamond markers)
### Detailed Analysis
Here's a breakdown of each data series, with approximate values extracted from the chart:
* **Qwen2-72B-Step-DPO (Purple):** This line generally slopes upward, indicating increasing accuracy with training steps.
* At 0 training steps: ~70% accuracy
* At 50 training steps: ~80% accuracy
* At 100 training steps: ~73% accuracy
* At 150 training steps: ~76% accuracy
* At 200 training steps: ~81% accuracy
* At 250 training steps: ~82% accuracy
* **Qwen2-7B-Step-DPO (Green):** This line shows a relatively stable accuracy with a slight upward trend.
* At 0 training steps: ~76% accuracy
* At 50 training steps: ~78% accuracy
* At 100 training steps: ~76% accuracy
* At 150 training steps: ~76% accuracy
* At 200 training steps: ~75% accuracy
* At 250 training steps: ~70% accuracy
* **Qwen2-72B-DPO (Violet):** This line exhibits a decreasing trend initially, followed by a slight increase.
* At 0 training steps: ~73% accuracy
* At 50 training steps: ~72% accuracy
* At 100 training steps: ~72% accuracy
* At 150 training steps: ~70% accuracy
* At 200 training steps: ~70% accuracy
* At 250 training steps: ~71% accuracy
* **Qwen2-7B-DPO (Orange):** This line shows a gradual increase in accuracy, with some fluctuations.
* At 0 training steps: ~67% accuracy
* At 50 training steps: ~69% accuracy
* At 100 training steps: ~70% accuracy
* At 150 training steps: ~70% accuracy
* At 200 training steps: ~70% accuracy
* At 250 training steps: ~69% accuracy
### Key Observations
* Qwen2-72B-Step-DPO consistently demonstrates the highest accuracy throughout the training process.
* Qwen2-7B-Step-DPO maintains a relatively high and stable accuracy.
* Qwen2-72B-DPO shows a slight decrease in accuracy initially, but stabilizes around 70-72%.
* Qwen2-7B-DPO starts with the lowest accuracy and shows a slow, gradual improvement.
* The accuracy of Qwen2-7B-Step-DPO decreases after 200 training steps.
### Interpretation
The chart suggests that the "Step-DPO" training method generally leads to higher accuracy in judging preferred and undesirable outputs, particularly for the larger Qwen2-72B model. The Qwen2-72B-Step-DPO model consistently outperforms the other models, indicating that increasing model size combined with the Step-DPO training technique is effective. The Qwen2-7B-Step-DPO model also performs well, suggesting that the Step-DPO method is beneficial even for smaller models. The Qwen2-72B-DPO model's initial dip in accuracy could be due to the learning process, where the model initially adjusts to the new training data before improving. The Qwen2-7B-DPO model's slower improvement suggests that it may require more training steps to reach optimal performance. The decrease in accuracy for Qwen2-7B-Step-DPO after 200 steps could indicate overfitting or the need for a different learning rate. Overall, the data highlights the importance of both model size and training methodology in achieving high accuracy in preference judgment tasks.
</details>
<details>
<summary>x3.png Details</summary>
### Visual Description
## Line Chart: Reward Margin Between Preferred and Undesirable Outputs
### Overview
This line chart visualizes the reward margin between preferred and undesirable outputs as a function of training steps for different model configurations. The chart compares the performance of Qwen2-7B-Step-DPO, Qwen2-72B-Step-DPO, Qwen2-72B-DPO, and Qwen2-7B-DPO models.
### Components/Axes
* **Title:** "Reward margin between preferred and undesirable outputs" (Top-center)
* **X-axis:** "Training steps" (Bottom-center), ranging from 0 to 250, with markers at 0, 50, 100, 150, 200, and 250.
* **Y-axis:** "Margin" (Left-center), ranging from 0.1 to 2.1, with markers at 0.1, 0.5, 0.9, 1.3, 1.7, and 2.1.
* **Legend:** Located in the top-right corner, identifying each line with its corresponding model name and color.
* Qwen2-7B-Step-DPO (Orange)
* Qwen2-72B-Step-DPO (Purple)
* Qwen2-72B-DPO (Dark Green)
* Qwen2-7B-DPO (Brown)
### Detailed Analysis
The chart displays four distinct lines, each representing a different model's reward margin over training steps.
* **Qwen2-7B-Step-DPO (Orange):** This line shows a steep upward trend initially, rapidly increasing from approximately 0.1 at 0 training steps to around 1.8 at 100 training steps. The increase slows down after 100 steps, reaching approximately 2.0 at 250 training steps.
* 0 steps: ~0.1
* 50 steps: ~1.0
* 100 steps: ~1.8
* 150 steps: ~1.9
* 200 steps: ~2.0
* 250 steps: ~2.0
* **Qwen2-72B-Step-DPO (Purple):** This line also exhibits an upward trend, but it is less steep than the orange line. It starts at approximately 0.1 at 0 training steps and reaches around 1.7 at 100 training steps, then plateaus around 1.8-1.9 for the remaining training steps.
* 0 steps: ~0.1
* 50 steps: ~1.3
* 100 steps: ~1.7
* 150 steps: ~1.8
* 200 steps: ~1.8
* 250 steps: ~1.9
* **Qwen2-72B-DPO (Dark Green):** This line shows a relatively flat trend. It starts at approximately 0.5 at 0 training steps and remains relatively stable around 0.7-0.8 throughout the training process, reaching approximately 0.8 at 250 training steps.
* 0 steps: ~0.5
* 50 steps: ~0.7
* 100 steps: ~0.7
* 150 steps: ~0.7
* 200 steps: ~0.7
* 250 steps: ~0.8
* **Qwen2-7B-DPO (Brown):** This line shows a moderate upward trend, starting at approximately 0.1 at 0 training steps and reaching around 0.7 at 100 training steps. It then plateaus around 0.7-0.8 for the remaining training steps.
* 0 steps: ~0.1
* 50 steps: ~0.5
* 100 steps: ~0.7
* 150 steps: ~0.7
* 200 steps: ~0.7
* 250 steps: ~0.7
### Key Observations
* Qwen2-7B-Step-DPO consistently outperforms the other models, achieving the highest reward margin.
* Qwen2-72B-Step-DPO performs better than Qwen2-72B-DPO and Qwen2-7B-DPO.
* Qwen2-72B-DPO and Qwen2-7B-DPO exhibit similar performance, with relatively flat reward margins.
* The reward margin for Qwen2-7B-Step-DPO and Qwen2-72B-Step-DPO appears to converge after approximately 150 training steps.
### Interpretation
The data suggests that incorporating "Step-DPO" into the training process significantly improves the reward margin, particularly for the Qwen2-7B model. The higher reward margin indicates that the model is better at distinguishing between preferred and undesirable outputs. The plateauing of the reward margin for Qwen2-72B-DPO and Qwen2-7B-DPO suggests that further training may not yield substantial improvements. The difference in performance between the 7B and 72B models, even with the same training method (Step-DPO), indicates that model size also plays a role in achieving higher reward margins. The convergence of the Step-DPO models suggests a potential saturation point in the learning process, where additional training steps provide diminishing returns. This data is valuable for understanding the effectiveness of different training strategies and model configurations in improving the quality of generated outputs.
</details>
Figure 2: Left: Accuracy of judging preferred or undesirable outputs on the validation set during training. Right: Reward margins between preferred and undesirable outputs on the validation set during training. More details about these experiments are given in the appendix.
Mathematical reasoning is recognized as a critical long-chain reasoning ability in Large Language Models (LLMs). This task is particularly challenging due to the often extensive chain of thought required, which can include numerous reasoning steps. Any error in these steps can lead to an incorrect final answer.
Numerous studies (Yu et al., 2023; Luo et al., 2023; Yue et al., 2023; Liu & Yao, 2024; Lu et al., 2024; Li et al., 2024; Shao et al., 2024; Xin et al., 2024; Yue et al., 2024; Tang et al., 2024) have proposed various data augmentation techniques during the supervised fine-tuning (SFT) stage to enhance alignment. However, models in the SFT process are prone to hallucinations, resulting in saturated performance. A potential reason for this, as highlighted in Hong et al. (2024), is that as the probability of preferred outputs increases, so does the probability of undesirable ones. This phenomenon makes the model more likely to make errors in long-chain reasoning. Therefore, it is essential to develop methods to suppress the likelihood of undesirable outputs.
Recently, Direct Preference Optimization (DPO) (Rafailov et al., 2024) has been proposed for alignment using pair-wise preference data and is popular due to its simplicity. Despite its effectiveness in chat benchmarks (Tunstall et al., 2023; Zheng et al., 2024), DPO offers minimal benefits for long-chain mathematical reasoning. As shown in Fig. 2 (left), models using vanilla DPO perform poorly in distinguishing between preferred and undesirable outputs, failing to identify errors in rejected answers. Additionally, Fig. 2 (right) shows that the reward margin (i.e., the gap between the rewards of preferred and undesirable outputs) is limited for models using vanilla DPO and plateaus with further training. These findings indicate that models fine-tuned with vanilla DPO cannot pinpoint detailed errors in incorrect answers, hindering the improvement of reasoning abilities.
In this work, we introduce Step-DPO, where each intermediate reasoning step is treated as the basic unit for preference optimization. As illustrated in Fig. 3, unlike vanilla DPO, which only considers preference optimization between complete answers (i.e., $p(y_{win}|x)$ and $p(y_{lose}|x)$ ), Step-DPO examines the step-by-step answer (i.e., $y=s_{1},...,s_{n}$ ) and specifically targets the first erroneous reasoning step. Step-DPO aims to select a correct reasoning step and reject an incorrect one, given a math problem and several initial correct reasoning steps (i.e., maximize $p(s_{win}|x;s_{1},s_{2},...,s_{k-1})$ and minimize $p(s_{lose}|x;s_{1},s_{2},...,s_{k-1})$ ). This transition allows the model to easily locate erroneous tokens for effective optimization, significantly enhancing long-chain reasoning.
Moreover, we present an effective and economical pipeline to collect pair-wise preference data, resulting in a high-quality dataset for Step-DPO. This dataset contains approximately 10K samples, each consisting of: 1) a mathematical problem, 2) prior reasoning steps, 3) the chosen step, and 4) the rejected step. Our three-step pipeline for dataset construction includes: 1) Error collection, 2) Step localization, and 3) Rectification. Notably, the chosen reasoning step is generated by the model itself, as we find that in-distribution data (i.e., self-generated data) is more effective than out-of-distribution data (e.g., data written by humans or GPT-4) for Step-DPO, as shown in Table 4.
With this curated dataset, mathematical reasoning performance can be significantly boosted with only hundreds of training steps, as demonstrated in Fig. 6. For instance, fine-tuning Qwen-72B-Instruct with Step-DPO results in a model achieving 70.8% accuracy on MATH and 94.0% on GSM8K, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro.
<details>
<summary>x4.png Details</summary>
### Visual Description
\n
## Diagram: Direct Preference Optimization (DPO) and Step-DPO
### Overview
The image is a diagram illustrating the concepts of Direct Preference Optimization (DPO) and Step-DPO, two methods for training language models based on preference data. It visually represents how preference data is used to refine a language model through a maximum likelihood approach. The diagram is divided into three main sections: preference data input, DPO process, and Step-DPO process.
### Components/Axes
The diagram consists of the following components:
* **Preference Data:** Represented by two speech bubble icons, one with a crown (y<sub>w</sub> - "win") and the other without (y<sub>l</sub> - "lose"). An arrow indicates the preference relationship (y<sub>w</sub> > y<sub>l</sub>).
* **Language Model:** Represented by a cluster of colored circles (yellow, orange, light blue, pink, purple, and red).
* **DPO:** A section showing the transformation of preference data into language model updates via "maximum likelihood".
* **Step-DPO:** A section showing step-wise preference data and its impact on the language model, also via "maximum likelihood".
* **Step-wise Preference Data:** Represented by a series of connected circles (s<sub>1</sub>, s<sub>2</sub>, ..., s<sub>k-1</sub>, s<sub>lose</sub>, s<sub>win</sub>) with arrows indicating the flow of preference. A green checkmark indicates the "win" (s<sub>win</sub>) and a red 'X' indicates the "lose" (s<sub>lose</sub>).
* **Text Labels:** "preference data", "DPO", "Step-DPO", "maximum likelihood", "step-wise preference data".
* **Mathematical Notation:** y<sub>w</sub>, y<sub>l</sub>, s<sub>1</sub>, s<sub>2</sub>, s<sub>k-1</sub>, s<sub>lose</sub>, s<sub>win</sub>.
### Detailed Analysis / Content Details
The diagram illustrates a process flow.
1. **Preference Data:** The process begins with preference data, where one response (y<sub>w</sub>) is preferred over another (y<sub>l</sub>). This is indicated by the arrow pointing from y<sub>l</sub> to y<sub>w</sub>.
2. **DPO:** The preference data is then fed into the DPO process, which uses a "maximum likelihood" approach to update the language model. The language model is represented by a cluster of colored circles.
3. **Step-DPO:** The Step-DPO process takes step-wise preference data (s<sub>1</sub>, s<sub>2</sub>, ..., s<sub>k-1</sub>, s<sub>lose</sub>, s<sub>win</sub>). The preference is indicated by the green checkmark on s<sub>win</sub> and the red 'X' on s<sub>lose</sub>. This data is also used with a "maximum likelihood" approach to update the language model.
The language model in both DPO and Step-DPO appears to be the same, represented by the same color scheme of circles. The Step-DPO process shows a sequential flow of preference data, while DPO appears to handle preference data in a more direct manner.
### Key Observations
* Both DPO and Step-DPO utilize a "maximum likelihood" approach for updating the language model.
* Step-DPO explicitly models step-wise preference data, suggesting a sequential or iterative refinement process.
* The color scheme of the language model circles is consistent across both DPO and Step-DPO, indicating that the underlying model is the same.
* The diagram does not provide any numerical data or specific values. It is a conceptual illustration of the processes.
### Interpretation
The diagram demonstrates two approaches to refining a language model based on human preferences. DPO directly optimizes the model based on pairwise preferences, while Step-DPO refines the model iteratively using step-wise preference data. The use of "maximum likelihood" in both methods suggests that the goal is to maximize the probability of the preferred responses given the preference data.
The Step-DPO process, with its sequential flow, might be useful in scenarios where preferences are revealed incrementally or where the model needs to learn from a series of related choices. The DPO process, on the other hand, might be more efficient when pairwise preferences are readily available.
The diagram highlights the importance of preference data in aligning language models with human values and expectations. It suggests that by incorporating human feedback, these models can be trained to generate more desirable and helpful responses. The lack of numerical data suggests that the diagram is intended to convey the conceptual framework rather than specific performance metrics. The diagram is a high-level overview and does not delve into the technical details of the algorithms.
</details>
Figure 3: Comparison between DPO and Step-DPO.
2 Related Works
2.1 Mathematical Reasoning
Large Language Models (LLMs) have exhibited substantial reasoning capabilities, primarily due to their auto-regressive nature, which allows them to predict the next token based on contextual information. However, these models still struggle with long-chain reasoning tasks, particularly in mathematical contexts. Several prior studies (Yao et al., 2024; Chen et al., 2024; Yoran et al., 2023; Li et al., 2023; Tong et al., 2024; Fu et al., 2022; Zhou et al., 2022) have attempted to enhance the Chain-of-Thought (CoT) inference framework (Wei et al., 2022) to address this issue. While these efforts have led to significant improvements in certain tasks, they have not fully mitigated common hallucinations and have limited generalizability across all reasoning tasks.
Another research direction (Yu et al., 2023; Luo et al., 2023; Yue et al., 2023; Liu & Yao, 2024; Lu et al., 2024; Xu et al., 2024; Li et al., 2024; Shao et al., 2024; Xin et al., 2024; Zhou et al., 2024; Liu et al., 2023; Ying et al., 2024; Yue et al., 2024; Tang et al., 2024; Mitra et al., 2024; Yuan et al., 2023) focuses on various data augmentation techniques, such as rephrasing, extension, and evolution, for supervised fine-tuning (SFT). These methods have significantly enhanced the reasoning abilities of LLMs, but their performance plateaus once the data reaches a certain volume. Additionally, methods like those proposed by Wang et al. (2023a); Liao et al. (2024); Toshniwal et al. (2024); Gou et al. (2023) employ external tools, such as Python, to substantially reduce calculation errors.
Other approaches (Azerbayev et al., 2023; Shao et al., 2024; Lin et al., 2024; Ying et al., 2024; Wang et al., 2023c) involve continued pre-training on extensive, high-quality math-related datasets, which markedly improve mathematical reasoning capabilities. Recent studies (Xu et al., 2024; Ying et al., 2024) have explored reinforcement learning to mitigate hallucinations in mathematical reasoning. Works like Lightman et al. (2023); Shao et al. (2024); Wang et al. (2023b) emphasize the importance of step-by-step verification in reinforcement learning for mathematical problems. However, these methods still rely on the quality of the reward model and require the complex training pipelines of RLHF. Building on this line of research, we propose Step-DPO, a simpler, more effective, and more efficient method.
2.2 Reinforcement Learning from Human Feedback
Supervised fine-tuning (SFT) can align models with human preferences. However, as the probability of preferred outputs increases, so does the likelihood of undesirable ones, leading to hallucinations. To generate more reliable outputs, Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022) has been introduced for LLM alignment. This approach involves training a reward model with comparison data and then using this reward model to optimize the policy model. The final performance heavily depends on the quality of the reward model, and the training pipeline is quite complex.
To simplify this process, Direct Preference Optimization (DPO) (Rafailov et al., 2024) was proposed, which directly uses pair-wise preference data for model optimization. This transition significantly streamlines the training pipeline. While DPO has proven effective in chat benchmarks, it offers only marginal benefits for mathematical reasoning. Inheriting the principles of DPO, Step-DPO is specifically designed for long-chain reasoning and has shown significant performance improvements in solving math word problems.
3 Step-DPO
In this section, we elaborate on the proposed Step-DPO. First, we present step-wise formulation in Sec. 3.1, a novel approach designed to enhance long-chain reasoning abilities by building on DPO. Next, in Sec. 3.2, we illustrate a pipeline for constructing the step-wise preference dataset for Step-DPO. Both components are essential for achieving the desired performance improvements.
3.1 Step-wise Formulation
Preliminary.
Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) is an effective approach for enhancing the robustness, factuality, and safety of LLMs (Ouyang et al., 2022). RLHF consists of two training phases: 1) reward model training, and 2) policy model training. However, the final performance of RLHF is highly sensitive to various hyperparameters in both phases, necessitating meticulous tuning.
To avoid this complex training pipeline, Rafailov et al. (2024) proposed Direct Preference Optimization (DPO), which directly uses pair-wise preference data to optimize the policy model with an equivalent optimization objective. Specifically, given an input prompt $x$ , and a preference data pair $(y_{win},y_{lose})$ , DPO aims to maximize the probability of the preferred output $y_{win}$ and minimize that of the undesirable output $y_{lose}$ . The optimization objective is formulated as:
$$
\displaystyle\begin{aligned} \mathcal{L}_{DPO}(\theta)=-\mathbb{E}_{(x,y_{win}%
,y_{lose})\sim D}[\log\sigma(\beta\log\frac{\pi_{\theta}(y_{win}|x)}{\pi_{ref}%
(y_{win}|x)}-\beta\log\frac{\pi_{\theta}(y_{lose}|x)}{\pi_{ref}(y_{lose}|x)})]%
,\end{aligned} \tag{1}
$$
where $D$ is the pair-wise preference dataset, $\sigma$ is the sigmoid function, $\pi_{\theta}(·|x)$ is the policy model to be optimized, $\pi_{ref}(·|x)$ is the reference model kept unchanged during training, and the hyperparameter $\beta$ controls the distance from the reference model.
Our Solution.
While DPO has proven effective in chat benchmarks, it brings only marginal improvements for long-chain reasoning tasks such as mathematical problems, as shown in Fig. 2 and Table 3. This limitation arises because most undesirable answers in these tasks do not contain errors initially; the first error often appears midway through the reasoning process. Rejecting an entire undesirable answer in DPO may also discard preceding correct reasoning steps, introducing significant noise and negatively impacting training.
Analogous to how teachers correct students by pinpointing specific errors rather than dismissing entire answers, our proposed Step-DPO provides more detailed supervision by identifying the specific erroneous reasoning step. This granular focus allows the model to swiftly locate, rectify, and avoid erroneous steps.
Specifically, the answer $y$ can be decomposed into a sequence of reasoning steps $y=s_{1},...,s_{n}$ , where $s_{i}$ is the $i$ -th reasoning step. As illustrated in Fig. 3, given a prompt $x$ and a series of initial correct reasoning steps $s_{1\sim k-1}=s_{1},...,s_{k-1}$ , Step-DPO aims to maximize the probability of the correct next reasoning step $s_{win}$ and minimize the probability of the incorrect one $s_{lose}$ . This objective can be formulated as:
$$
\displaystyle\begin{aligned} \mathcal{L}(\theta)=-\mathbb{E}_{(x,s_{1\sim k-1}%
,s_{win},s_{lose})\sim D}[\log\sigma(\beta\log\frac{\pi_{\theta}(s_{win}|x;s_{%
1\sim k-1})}{\pi_{ref}(s_{win}|x;s_{1\sim k-1})}-\beta\log\frac{\pi_{\theta}(s%
_{lose}|x;s_{1\sim k-1})}{\pi_{ref}(s_{lose}|x;s_{1\sim k-1})})].\end{aligned} \tag{2}
$$
3.2 In-distribution Data Construction
<details>
<summary>x5.png Details</summary>
### Visual Description
\n
## Diagram: AI Reasoning Process - Error Correction
### Overview
The image depicts a diagram illustrating an AI's reasoning process for solving a mathematical problem, specifically focusing on error detection, localization, and rectification. It's structured in three main sections: Error Collection, Step Localization, and Rectification, each showing a visual representation of the AI's internal state and actions. The diagram uses a combination of text blocks, flowchart-style icons, and connecting lines to represent the flow of information and reasoning.
### Components/Axes
The diagram is divided into three horizontal sections, labeled "1. Error Collection", "2. Step Localization", and "3. Rectification". Each section contains three main elements:
* **Instruction/Problem Statement:** A text block on the left presenting the mathematical problem.
* **Model Completion/Reasoning:** A central visual representation of the AI's reasoning process, depicted as interconnected circles with gears.
* **Response/Output:** A text block on the right showing the AI's generated response.
* **Verification/Correction:** In sections 2 and 3, a checkmark or an "X" is used to indicate the correctness of a step.
### Detailed Analysis or Content Details
**1. Error Collection:**
* **Instruction:** "Determine the least positive integer x for which the sum of x and 4609 gives a remainder of 2104 when divided by 12."
* **Response (Initial):** "Let's think step by step. Step 1:"
* **Wrong Answer Collection:** The AI initially provides an incomplete response and arrives at an incorrect answer of "6". The visual representation shows a crossed-out icon.
**2. Step Localization:**
* **Response (Step-by-Step):**
* Step 1: "We know that x + 4609 ≡ 2104 (mod 12)." (Marked with a checkmark)
* Step 2: "We can rewrite the given congruence as x ≡ 2104 - 4609 (mod 12)."
* Step 3: "Calculating 2104 - 4609 = -2505." (Marked with a checkmark)
* Step 4: "We want to find the positive integer x, so we add 12 until we get a positive number. -2505 + 12 = -2493. -2493 + 12 = -2481..."
* Step 5: "So x = 30 (mod 12)." (Marked with a checkmark)
* Step 6: "The least positive integer x that satisfies this congruence is 6."
* **Step-by-Step Verification:** A visual representation with a checkmark indicates the steps are being verified.
* **Correct Answer Collection:** The AI is shown to be converging towards a correct solution.
**3. Rectification:**
* **Step 4 (Revised):** "Now we need to find the equivalent positive integer for -2505 (mod 12). To do this, we divide -2505 by 12 and find the remainder. -2505 ÷ 12 = -208 with a remainder of -9. However, we want a positive remainder, so we add 12 to -9 to get a positive remainder. -9 + 12 = 3. So, -2505 ≡ 3 (mod 12)."
* **Step 5:** "..."
* **Step 6:** "The answer is: 3"
* **Model Completion (Revised):** The visual representation shows a refined reasoning process.
* **Equation:** "γ~πref(Y|X) Sk=1" is present in the bottom right corner.
### Key Observations
* The AI initially makes an error in the first stage (Error Collection).
* The Step Localization phase involves a detailed, step-by-step verification process.
* The Rectification phase demonstrates the AI's ability to correct its initial error by re-evaluating a specific step and arriving at the correct answer.
* The visual representation of the AI's reasoning process (gears) changes slightly between stages, suggesting refinement.
* The equation "γ~πref(Y|X) Sk=1" appears in the final stage, potentially representing a mathematical formulation of the AI's reasoning or a performance metric.
### Interpretation
This diagram illustrates a crucial aspect of AI development: the ability to detect, localize, and rectify errors in reasoning. The AI doesn't simply provide an answer; it demonstrates a process of self-correction. The diagram highlights the importance of step-by-step reasoning and verification in achieving accurate results. The initial incorrect answer (6) and subsequent correction to (3) demonstrate the AI's iterative refinement process. The equation at the end suggests a formalization of this reasoning process, potentially used for evaluating or improving the AI's performance. The use of visual cues (checkmark, "X", gear icons) effectively communicates the AI's internal state and actions. The diagram suggests a system designed for explainability and debugging, allowing developers to understand *how* the AI arrived at its conclusion and identify areas for improvement.
</details>
Figure 4: Data construction pipeline for Step-DPO.
<details>
<summary>x6.png Details</summary>
### Visual Description
\n
## Textual Document: Problem Solving Steps
### Overview
The image presents a step-by-step reasoning process for solving a mathematical problem. The problem asks to determine the least positive integer *x* such that the sum of *x* and 4609 gives a remainder of 2104 when divided by 12. The document outlines initial reasoning steps, a chosen step, and a rejected step.
### Components/Axes
The document is structured into sections labeled: "prompt", "initial_reasoning_steps", "chosen_step", and "rejected_step". There are no axes or charts present.
### Content Details
**prompt:**
"Determine the least positive integer x for which the sum of x and 4609 gives a remainder of 2104 when divided by 12."
**initial_reasoning_steps:**
"Let's think step by step.
Step 1: We know that x + 4609 ≡ 2104 (mod 12).
Step 2: We can rewrite the given congruence as x ≡ 2104 - 4609 (mod 12).
Step 3: Calculating 2104 - 4609 = -2505.
Step 4:"
**chosen_step:**
"Now we need to find the equivalent positive integer for -2505 (mod 12).
To do this, we divide -2505 by 12 and find the remainder. -2505 ÷ 12 = -208 remainder -9.
However, we want a positive remainder, so we add 12 to -9 to get a positive remainder. -9+12=3.
So, -2505 ≡ 3 (mod 12)."
**rejected_step:**
"We want to find the positive integer x, so we add 12until we get a positive number. -2505 + 12 = -210 + 12 = 30."
### Key Observations
The document demonstrates a modular arithmetic problem-solving approach. The "chosen_step" correctly identifies the positive remainder when -2505 is divided by 12. The "rejected_step" attempts to find a positive integer by repeatedly adding 12, but the calculation is incomplete and leads to an incorrect result.
### Interpretation
The document illustrates a common problem-solving technique: breaking down a complex problem into smaller, manageable steps. The inclusion of both a "chosen_step" and a "rejected_step" suggests an iterative process of reasoning and refinement. The "chosen_step" demonstrates the correct application of modular arithmetic to find the equivalent positive integer, while the "rejected_step" highlights a potential pitfall in the reasoning process. The document is a record of thought, and not a final solution. The final answer is not provided.
</details>
Figure 5: An example of preference data sample for Step-DPO.
According to the optimization target of Step-DPO, we need to create a corresponding high-quality pair-wise preference dataset. Each data sample should comprise four entries: 1) prompt $x$ ; 2) initial reasoning steps $s_{1\sim k-1}=s_{1},...,s_{k-1}$ ; 3) preferred reasoning step $s_{win}$ ; 4) undesirable reasoning step $s_{lose}$ , as shown in Fig. 5. To obtain a high-quality dataset, we propose a data construction pipeline illustrated in Fig. 4, which includes the following three steps.
Error collection.
First, we collect a set $D_{0}=\{(x,\hat{y})\}$ of mathematical problems $x$ with ground-truth answers $\hat{y}$ . Each mathematical problem $x$ is then used as a prompt to infer answers using the initial model $\pi_{ref}$ . Before inference, we add the step-wise Chain-of-Thought (CoT) prefix for prompting, i.e., "Let’s think step by step. Step 1:". This ensures that the model’s inference results are structured into multiple reasoning steps, with each step explicitly starting with "Step i:". Upon completion of inference, we obtain the model answers $y$ for each mathematical problem $x$ . We then select instances where the final answer $y$ differs from the ground truth $\hat{y}$ , resulting in a dataset of erroneous inference results, denoted as $D_{1}=\{(x,\hat{y},y)|x∈ D_{0}\}$ .
Step localization.
Given that each erroneous inference result is explicitly presented as a sequence of reasoning steps $y=s_{1},s_{2},...,s_{n}$ , we proceed to verify the correctness of each reasoning step until we find the first error and record its step number $k$ . This process can be done manually or using GPT-4. We select $s_{k}$ as the erroneous reasoning step $s_{lose}$ , resulting in a dataset that contains the erroneous steps, denoted as $D_{2}=\{(x,\hat{y},s_{1\sim k-1},s_{lose})|x∈ D_{1}\}$ .
Rectification.
To obtain the corresponding correct reasoning step for each sample in $D_{2}$ , we need to sample multiple outputs $y_{cont}$ by inferring the model $\pi_{ref}$ with the prompt $x$ and the preceding correct reasoning steps $s_{1\sim k-1}$ . This process is formulated as:
$$
\displaystyle\begin{aligned} y_{cont}\sim\pi_{ref}(y|x;s_{1\sim k-1}).\end{aligned} \tag{3}
$$
We retain those outputs where the final answer matches the ground truth. Among the remaining outputs, we select the first reasoning step in $y_{cont}$ as $s_{win}$ , resulting in the final dataset $D=\{(x,s_{1\sim k-1},s_{lose},s_{win})|x∈ D_{2}\}$ . An example of a resulting data sample is shown in Fig. 5.
Notably, some cases may have correct final answers but erroneous intermediate reasoning steps. Therefore, we may need to further filter out samples where $s_{win}$ is incorrect, which can be done manually or by GPT-4. We omit this process in the notations for simplicity, and more details are provided in the appendix.
It is important to note that the data pipeline is user-friendly. In this data pipeline, humans or GPT-4 are only required to locate errors and rank answers, and they do not need to write answers or rectifications by themselves.
We also note that the use of in-distribution data is crucial. When selecting $s_{win}$ , we use outputs generated by the model $\pi_{ref}$ rather than answers rectified by humans or GPT-4. Since human or GPT-4 rectified answers $s_{win}^{ood}$ are out-of-distribution (OOD) regarding the model $\pi_{ref}$ , the log-probability of outputting $s_{win}^{ood}$ (i.e., $\log\pi_{ref}(s_{win}^{ood}|x)$ ) is significantly lower than that of an in-distribution (ID) output $\log\pi_{ref}(s_{win}^{id}|x)$ . Moreover, it is challenging for the policy model $\pi_{\theta}$ to learn to increase the probability of $s_{win}^{ood}$ due to gradient decay issues (detailed in the appendix). Consequently, adopting self-generated in-distribution data as the preferred answer proves to be a more effective way of aligning with human preferences.
4 Experiments
In this section, we first introduce the experimental setup in Sec. 4.1. Then, we present the main results in Sec. 4.2, which include an exhaustive performance comparison. Moreover, we conduct an extensive ablation study in Sec. 4.3. Finally, a few demonstrations are shown in Sec. 4.4 to further understand Step-DPO.
4.1 Experimental Setup
Network Architecture.
Our experiments are based on various base models, including the Qwen2 and Qwen1.5 series (Bai et al., 2023), Meta-Llama-3-70B (Touvron et al., 2023), and deepseek-math-7b-base (Shao et al., 2024).
Table 1: Math reasoning performance comparison on MATH and GSM8K across various models. general: general-purpose model. open: open-source.
- $\;\;{}^{\dagger}$ Supervised fine-tuned models with our 299K SFT data based on the open-source base model.
- $\;\;{}^{\ddagger}$ Reproduced using our prompt
Table 2: Math reasoning performance comparison on compitition-level math problems, i.e., AIME 2024 and Odyssey-MATH. Note that the training data for Step-DPO is the same as before.
| Model | size | open | AIME | Odyssey-MATH (%) |
| --- | --- | --- | --- | --- |
| Gemini-1.5-Pro (Reid et al., 2024) | - | ✗ | 2 / 30 | 45.0 |
| Claude-3-Opus | - | ✗ | 2 / 30 | 40.6 |
| GPT-4-1106 (Achiam et al., 2023) | - | ✗ | 1 / 30 | 49.1 |
| GPT-4-Turbo-0409 (Achiam et al., 2023) | - | ✗ | 3 / 30 | 46.8 |
| GPT-4o-0513 | - | ✗ | 2 / 30 | 53.2 |
| DeepSeek-Coder-V2-Lite-Instruct (Zhu et al., 2024) | 16B | ✓ | 0 / 30 | 44.4 |
| Llama-3-70B-Instruct (Touvron et al., 2023) | 70B | ✓ | 1 / 30 | 27.9 |
| DeepSeek-Coder-V2-Instruct (Zhu et al., 2024) | 236B | ✓ | 4 / 30 | 53.7 |
| Qwen2-72B-SFT † | 72B | ✓ | 1 / 30 | 44.2 |
| Qwen2-72B-SFT + Step-DPO | 72B | ✓ | 3 / 30 | 47.0 (+2.8) |
| Qwen2-72B-Instruct (Bai et al., 2023) | 72B | ✓ | 5 / 30 | 47.0 |
| Qwen2-72B-Instruct + Step-DPO | 72B | ✓ | 4 / 30 | 50.1 (+3.1) |
- $\;\;{}^{\dagger}$ Supervised fine-tuned models with our 299K SFT data based on the open-source base model.
Datasets.
In supervised fine-tuning (SFT), we use augmented mathematical problems from MetaMath (Yu et al., 2023) and MMIQC (Liu & Yao, 2024) to infer step-by-step responses with DeepSeekMath, as the SFT data used in DeepSeekMath (Shao et al., 2024) is not publicly available. After filtering out responses with erroneous final answers, we obtain 374K SFT data. Of these, 299K are used for SFT, and the remainder is used for further Step-DPO training.
In the Step-DPO phase, alongside the remaining SFT data, we also incorporate a subset of AQuA (Ling et al., 2017). These datasets are processed as described in Sec. 3.2, resulting in 10K pair-wise preference data for Step-DPO.
For evaluation, we use the widely adopted MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021) datasets. Accuracy in these datasets serves as the evaluation metric. The MATH test set contains 5000 mathematical problems spanning 5 difficulty levels and 7 subjects, including algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus. The GSM8K test set includes 1319 mathematical problems, each with a step-by-step solution and a ground-truth answer. The problems in GSM8K are generally easier than those in MATH. Besides, we also use completition-level problems in American Invitational Mathematics Examination (AIME) (MAA, 2024) and Odyssey-MATH (Netmind.AI, 2024) to evaluate the math reasoning capabilities in solving hard problems.
Implementation Details.
First, we use the 299K SFT data for supervised fine-tuning on the base models, obtaining the SFT models. We train 7B models for 3 epochs and models larger than 30B for 2 epochs. The global batch size is set to 256, and the learning rate is set to 5e-6. We use the AdamW optimizer with a linear decay learning rate scheduler, setting the warmup ratio to 0.03. DeepSpeed ZeRO3 with CPU offload is used to reduce GPU memory usage during training.
Next, we perform Step-DPO based on the SFT models. For Step-DPO, we train 7B models for 8 epochs and models larger than 30B for 4 epochs. The global batch size is set to 128, and the learning rate is set to 5e-7. The hyperparameter $\beta$ is set to 0.5 for the 72B model and 0.4 for others. We use the AdamW optimizer and a cosine learning rate scheduler, with the warmup ratio set to 0.1.
4.2 Results
Table 3: Performance comparison between DPO and Step-DPO. We use only 5K data for training in this ablation study.
| Model MATH (%) Model | Qwen2-7B-SFT 54.8 Qwen2-72B-SFT | Qwen2-7B-SFT + DPO (5K) 55.0 Qwen2-72B-SFT + DPO (5K) | Qwen2-7B-SFT + Step-DPO (5K) 55.8 Qwen2-72B-SFT + Step-DPO (5K) |
| --- | --- | --- | --- |
| MATH (%) | 61.7 | 62.5 | 64.1 |
Table 4: Performance comparison between out-of-distribution and in-distribution data. OOD: out-of-distribution data. ID: in-distribution data.
| MATH (%) | 54.8 | 55.1 | 55.8 |
| --- | --- | --- | --- |
Applying on open-source instruct models.
Table 1 presents a comprehensive comparison of various models, encompassing both open-source and closed-source models. Notably, Step-DPO can be directly integrated into open-source instruction models, such as DeepSeekMath-RL and Qwen2-72B-Instruct, leading to significant performance enhancements even after their prior RLHF training phase. This indicates that Step-DPO complements RLHF effectively. Specifically, when applied to Qwen2-72B-Instruct, Step-DPO achieves scores of 70.8% and 94.0% on the MATH and GSM8K test sets, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro.
Applying on SFT models.
To further substantiate the efficacy of Step-DPO, we applied it to SFT models. Initially, we performed supervised fine-tuning on the 299K SFT dataset mentioned in Sec. 4.1, resulting in models such as DeepSeekMath-Base-SFT, Qwen2-7B-SFT, Qwen1.5-32B-SFT, Llama3-70B-SFT, and Qwen2-72B-SFT. Step-DPO proved highly effective, yielding significant improvements across various model sizes. Particularly, for models exceeding 70B parameters (i.e., Llama-3-70B-SFT and Qwen-2-72B-SFT), Step-DPO achieved approximately a 3% performance boost on the MATH test set.
Interestingly, larger models exhibited greater performance gains from Step-DPO. We hypothesize that larger models have untapped potential that Step-DPO can exploit. If the performance ceiling is not reached through supervised fine-tuning (SFT), Step-DPO can help models approach their optimal performance.
Results on math competition problems.
To further illustrate the superiority of Step-DPO in mathematical reasoning, we evaluated the models on competition-level math problems, specifically AIME 2024 and Odyssey-MATH, as shown in Fig. 2. Despite the increased difficulty of these problems compared to MATH and GSM8K, Step-DPO significantly enhanced performance. On Odyssey-MATH, Step-DPO applied to Qwen2-72B-Instruct achieved 50.1% accuracy, narrowing the performance gap with GPT-4o.
Notably, the models used the same Step-DPO training data for these competition-level problems as for problems of normal difficulty, highlighting Step-DPO’s robust generalization capability.
4.3 Ablation Study
To validate the effectiveness of Step-DPO and its data construction process, we conducted an extensive ablation study.
DPO vs. Step-DPO.
As discussed in Sec. 3.1, models utilizing vanilla DPO struggle to accurately identify errors in incorrect answers, providing only marginal benefits to mathematical reasoning performance. To verify this, we compared vanilla DPO and Step-DPO in terms of both accuracy in judging preferred versus undesirable outputs (left side of Fig. 2) and the reward margin between them (right side of Fig. 2). We also reported the final mathematical reasoning performance on the MATH test set in Table 3. The results indicated that the benefits of DPO are limited and significantly less than those of Step-DPO. In this experiment, we used only 5K Step-DPO training data and 3K for evaluation in Fig. 2.
Out-of-Distribution vs. In-Distribution Data.
The importance of in-distribution data was emphasized in Sec. 3.2. To illustrate its effectiveness, we compared out-of-distribution and in-distribution data in Table 4. Out-of-distribution data was generated by using GPT-4 to correct erroneous reasoning steps in incorrect answers, whereas in-distribution data was generated through the pipeline described in Sec. 3.2. The results in Table 4 underscore the critical role of in-distribution data in enhancing performance.
4.4 Demonstrations
<details>
<summary>x7.png Details</summary>
### Visual Description
\n
## Textual Document: Problem Solving Example
### Overview
The image presents a problem statement followed by two solution approaches, labeled "Owen2-72B-Instruct" and "Owen2-72B-Instruct + Step-DPO". Both approaches aim to solve the same mathematical problem: finding the number of integer values of *t* that satisfy the condition that the square root of *t* is greater than 2 and less than 3.5. The solutions are presented as numbered steps.
### Components/Axes
The document is structured into two main columns, each presenting a solution. Each solution is broken down into numbered steps. The problem statement is at the top. There are no axes or charts in this image.
### Detailed Analysis or Content Details
**Problem Statement:**
"The square root of t is greater than 2 and less than 3.5. How many integer values of t satisfy this condition?"
**Owen2-72B-Instruct Solution:**
1. "Translate the condition into inequalities: 2 < √t < 3.5"
2. "Square all parts of the inequality to eliminate the square root: (2)² < t < (3.5)²"
3. "Calculate the squares: 4 < t < 12.25"
4. "Determine the integer values of t that satisfy the inequality: The integers that fall strictly between 4 and 12.25 are 5, 6, 7, 8, 9, 10, 11, 12. However, since t must be strictly less than 12.25, 12 is not included in the solution set because it is not strictly less than 12.25."
5. "Count the integer values: The integers that satisfy the condition are 5, 6, 7, 8, 9, 10, 11, which gives us a total of 7 integer values of t."
**Owen2-72B-Instruct + Step-DPO Solution:**
1. "Translate the inequality into mathematical terms: The given condition can be written as: 2 < √t < 3.5."
2. "Square all parts of the inequality to eliminate the square root: Squaring all parts of the inequality gives: (2)² < t < (3.5)²"
3. "Calculate the squares: 4 < t < 12.25"
4. "Find the integer values of t that satisfy the inequality: The inequality 4 < t < 12.25 means that t is greater than 4 and less than 12.25. Since t must be an integer, the possible values of t are 5, 6, 7, 8, 9, 10, 11, 12. However, since t must be strictly less than 12.25, the value 12 is included in the range."
5. "Count the integer values: The integers that satisfy the condition are 5, 6, 7, 8, 9, 10, 11, 12. There are 8 integer values of t that satisfy the given condition."
### Key Observations
The two solutions differ in their final count of integer values. The "Owen2-72B-Instruct" solution arrives at 7, while the "Owen2-72B-Instruct + Step-DPO" solution arrives at 8. This discrepancy arises from the interpretation of whether 12 is included in the solution set, given the strict inequality 4 < t < 12.25. The first solution excludes 12, while the second includes it.
### Interpretation
The document demonstrates two approaches to solving a simple mathematical inequality problem. The difference in the final answers highlights the importance of careful interpretation of inequalities, particularly regarding strict versus non-strict inequalities. The "Step-DPO" approach appears to be more careful in its consideration of the range of possible integer values, leading to a different (and arguably more correct) answer. The document serves as a good example of how different reasoning processes can lead to different conclusions, even when starting from the same initial problem statement. The inclusion of both solutions allows for a comparison of the reasoning steps and a discussion of the nuances of mathematical problem-solving. The "Step-DPO" solution is likely an improved version of the original "Owen2-72B-Instruct" solution, incorporating a more precise understanding of the problem's constraints.
</details>
Figure 6: An example of comparison between Qwen2-72B-Instruct and Qwen2-72B-Instruct-Step-DPO.
As shown in Fig. 6, we demonstrate an example of comparison between Qwen2-72B-Instruct and Qwen2-72B-Instruct-Step-DPO. It turns out that Step-DPO does well in correcting minor mistakes in previous models. More comparisons are provided in the appendix.
5 Conclusion
In this work, we proposed a simple, effective, and data-efficient method called Step-DPO. Unlike DPO, which compares preferences between holistic answers, Step-DPO uses a single reasoning step as the fundamental unit for preference comparison. This transition enables fine-grained process supervision for LLMs, facilitating the quick localization of errors within incorrect answers. Additionally, we introduced a data construction pipeline for Step-DPO, creating a dataset with 10K preference data pairs. Our results demonstrate the significant improvements achieved by Step-DPO and the 10K dataset, particularly for large models. We hope that Step-DPO will provide new insights into model alignment for long-chain reasoning problems.
References
- Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023.
- Azerbayev et al. (2023) Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv:2310.10631, 2023.
- Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv:2309.16609, 2023.
- Chen et al. (2024) Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. arXiv:2405.03553, 2024.
- Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, 2017.
- Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.
- Fu et al. (2022) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In ICLR, 2022.
- Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv:2309.17452, 2023.
- Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv:2103.03874, 2021.
- Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv:2403.07691, 2024.
- Li et al. (2024) Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv:2403.04706, 2024.
- Li et al. (2023) Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large language model society. NeurIPS, 2023.
- Liao et al. (2024) Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output–a reproducible pipeline. arXiv:2401.08190, 2024.
- Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv:2305.20050, 2023.
- Lin et al. (2024) Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv:2404.07965, 2024.
- Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv:1705.04146, 2017.
- Liu & Yao (2024) Haoxiong Liu and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing. arXiv:2401.09003, 2024.
- Liu et al. (2023) Yixin Liu, Avi Singh, C Daniel Freeman, John D Co-Reyes, and Peter J Liu. Improving large language model fine-tuning for solving math problems. arXiv:2310.10047, 2023.
- Lu et al. (2024) Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv:2402.16352, 2024.
- Luo et al. (2023) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv:2308.09583, 2023.
- MAA (2024) MAA. American invitational mathematics examination, 2024. URL https://maa.org/math-competitions/american-invitational-mathematics-examination-aime.
- Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv:2402.14830, 2024.
- Netmind.AI (2024) Netmind.AI. Odyssey-math. https://github.com/protagolabs/odyssey-math/tree/main, 2024. Accessed: April 22, 2024.
- Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
- Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2024.
- Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530, 2024.
- Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv:2308.12950, 2023.
- Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024.
- Tang et al. (2024) Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv:2403.02884, 2024.
- Tong et al. (2024) Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. Can llms learn from previous mistakes? investigating llms’ errors to boost for reasoning. arXiv:2403.20046, 2024.
- Toshniwal et al. (2024) Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv:2402.10176, 2024.
- Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023.
- Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv:2310.16944, 2023.
- Wang et al. (2023a) Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv:2310.03731, 2023a.
- Wang et al. (2023b) Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935, 2023b.
- Wang et al. (2023c) Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i–mathpile: A billion-token-scale pretraining corpus for math. arXiv:2312.17120, 2023c.
- Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
- Xin et al. (2024) Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data. arXiv:2405.14333, 2024.
- Xu et al. (2024) Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline. arXiv:2404.02893, 2024.
- Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. NeurIPS, 2024.
- Ying et al. (2024) Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv:2402.06332, 2024.
- Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. Answering questions by meta-reasoning over multiple chains of thought. arXiv:2304.13007, 2023.
- Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv:2309.12284, 2023.
- Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv:2308.01825, 2023.
- Yue et al. (2023) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv:2309.05653, 2023.
- Yue et al. (2024) Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv:2405.03548, 2024.
- Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2024.
- Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv:2205.10625, 2022.
- Zhou et al. (2024) Kun Zhou, Beichen Zhang, Jiapeng Wang, Zhipeng Chen, Wayne Xin Zhao, Jing Sha, Zhichao Sheng, Shijin Wang, and Ji-Rong Wen. Jiuzhang3. 0: Efficiently improving mathematical reasoning by training small data synthesis models. arXiv:2405.14365, 2024.
- Zhu et al. (2024) Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv:2406.11931, 2024.