# Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search
**Authors**: Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, Chaofan Ding
## Abstract
Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMsâ reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its ârelative correctness,â and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence. Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.
## Introduction
Although todayâs large language models (LLMs) can perform excellently on a variety of language tasks (Zhao et al. 2023), even approaching human levels, reasoning remains an unaddressed challenge for them (Huang and Chang 2023; Valmeekam et al. 2022). To enhance their reasoning ability, some studies use Chain-of-Thought (CoT) prompting (Nye et al. 2021; Wei et al. 2022; Kojima et al. 2022) to encourage LLMs to decompose given problems and think step by step, but the elicited reasoning ability is still limited. Nevertheless, these works emphasize the importance of step-by-step reasoning for LLMs.
To enhance LLMsâ reasoning ability, some researchers propose Rejection sampling Fine-Tuning (RFT) (Yuan et al. 2023), which generates reasoning paths using LLMs and filters out those leading to incorrect answers. Moreover, the Self-Taught Reasoner (STaR) (Zelikman et al. 2022) use the ground truth answer as a hint for LLMs to generate reasoning paths that lead to correct answers. However, this approach may also produce incorrect reasoning paths that happen to arrive at the correct answer. Similarly, Iterative Reasoning Preference Optimization (Pang et al. 2024) samples correct and incorrect CoTs to form preference pairs and iteratively applies Direct Preference Optimization (DPO) (Rafailov et al. 2024). These training methods fall into the category of outcome-supervised fine-tuning, as they directly supervise the LLMs to produce correct final answers rather than correct reasoning paths.
Process supervision provides LLMs with more precise and fine-grained feedback. Lightman et al. (Lightman et al. 2023) find that their process reward models (PRM) perform significantly better than outcome reward models (ORM). Using a PRM, reasoning steps can be rewarded, enabling reinforcement learning to train LLMs to generate better CoTs. However, Lightman et al. rely on human annotators to label the reasoning steps, which is highly expensive, particularly for challenging math problems. To automatically label reasoning steps, some works employ (Wang et al. 2024a, b; Luo et al. 2024) Monte Carlo Sampling to estimate the âcorrectnessâ of the steps, showing that PRMs trained on their automatically labeled data can even outperform those trained on PRM800K (Lightman et al. 2023), a human-labeled dataset. Wang et al. (Wang et al. 2024a) further train LLMs with their PRMs using reinforcement learning, and find that it is better than training with ORMs. ReST-MCTS* (Zhang et al. 2024b) leverages Monte Carlo Tree Search (MCTS) (Kocsis and SzepesvĂĄri 2006; Browne et al. 2012) to annotate the process reward of each step for training a PRM, and use the PRM to guide MCTS in turn.
Some studies propose methods that do not require a PRM to provide process supervision to LLMs. Step-DPO (Lai et al. 2024) and Self-Explore (Hwang et al. 2024) identify the first incorrect step in a reasoning path through step-by-step verification, creating a pairwise dataset for subsequent preference learning. MCTS-DPO (Xie et al. 2024) utilizes MCTS to generate pairwise data for preference learning, with training and data generation performed iteratively. However, MCTS-DPO labels reasoning steps only as chosen or rejected, which does not accurately capture the quality of the steps, which can lead to suboptimal performance. Additionally, MCTS-DPO forms preference pairs using only the best and worst steps, discarding other steps that are potentially valuable.
In this paper, we study using MCTS to generate process supervision data. We apply MCTS at each step along the reasoning paths generated by an LLM, assigning a score that captures ârelative correctnessâ to the sampled next steps. Compared to binary preferences, the scores reflect the quality of the steps more accurately. The next steps with scores are integrated into a weighted negative log-likelihood loss function to train the LLM. We also iteratively generate training data and train the LLM, following MCTS-DPO. Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, the models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.
## Related Work
A key technique for enhancing reasoning is Chain-of-Thought (CoT) prompting (Nye et al. 2021; Wei et al. 2022; Kojima et al. 2022; Wang et al. 2022), which encourages LLMs to think about problems step by step and generate reasoning chains. It is also found that reasoning chains with more steps are more likely to lead to correct answers (Fu et al. 2022), further highlighting the importance of step-by-step reasoning. Furthermore, tree search algorithms, such as MCTS, are integrated with LLMs during inference to search for correct reasoning paths, resulting in significant improvements in performance on reasoning tasks (Hao et al. 2023; Yao et al. 2024; Qi et al. 2024; Zhang et al. 2024a), but at the cost of a substantial increase in inference compute.
While the aforementioned studies enhance reasoning during inference, another research direction aims to instill reasoning ability into LLMs through training. Some studies train LLMs using responses generated by the models themselves (Yuan et al. 2023; Zelikman et al. 2022; Pang et al. 2024; Trung et al. 2024), employing supervised fine-tuning or reinforcement learning. Question synthesis has also been shown to be effective for generating training data (Yu et al. 2023; Liu et al. 2024; Lu et al. 2024; Li et al. 2024), where several data augmentation techniques are commonly applied.
Recently, an increasing number of studies have demonstrated that process supervision is more effective than outcome supervision, for training both reward models (Lightman et al. 2023; Wang et al. 2024a, b; Luo et al. 2024) and LLMs (Lai et al. 2024; Hwang et al. 2024; Xie et al. 2024; Wang et al. 2024a; Chen et al. 2024). Learned process reward models are usually used for reinforcement learning and for selecting the best reasoning path from a set of sampled paths. In this paper, we explore training LLMs with process supervision without relying on reward models, thereby avoiding the complexity and instability of reinforcement learning.
## Proposed Methods
<details>
<summary>x1.png Details</summary>
### Visual Description
\n
## Diagram: Iterative Model Training via Monte Carlo Tree Search
### Overview
The image is a technical flowchart illustrating a cyclical process for training or fine-tuning a machine learning model. The core of the process uses a Monte Carlo Tree Search (MCTS) algorithm to explore a problem space, evaluate the correctness of different solution paths, and generate training data to improve the model. The diagram depicts a closed-loop system where the model's outputs feed back into the process.
### Components/Axes
The diagram consists of four primary components connected by directional arrows indicating data flow:
1. **Problem Pool** (Top-Center): A blue cylinder icon representing a repository or source of problems.
2. **Monte Carlo Tree Search (MCTS)** (Left-Center): A dashed rectangular box containing a tree structure. The tree has a root node and multiple levels of child nodes connected by lines. The nodes are colored circles.
3. **Training Data Pool** (Right-Center): A blue cylinder icon representing a repository for collected training examples.
4. **Model** (Top-Right): A blue rounded rectangle representing the machine learning model being trained.
**Legend** (Bottom-Left):
* **Title**: "Relative Correctness"
* **Scale**: A horizontal color bar ranging from **High** (left) to **Low** (right).
* **Colors**: The scale shows a gradient from **Red** (High correctness) through **Pink** and **Purple** to **Blue** (Low correctness).
### Detailed Analysis
**Flow and Relationships:**
1. The process begins with the **Problem Pool**. An arrow points downward from it into the **Monte Carlo Tree Search** block.
2. Inside the MCTS block, a tree structure is shown. The **root node** is colored **red** (indicating High relative correctness). It branches into five first-level nodes. From left to right, their colors are: **blue**, **pink**, **red**, **pink**, **blue**.
3. The central **red** first-level node branches further into three second-level nodes: **purple**, **red**, **pink**.
4. The central **red** second-level node branches into three third-level (leaf) nodes: **blue**, **purple**, **blue**.
5. An arrow exits the right side of the MCTS dashed box and points to the **Training Data Pool**. This indicates that information or data generated/evaluated by the MCTS process is sent to the training pool.
6. An arrow labeled **"Fine-Tune"** points upward from the **Training Data Pool** to the **Model**. This shows the model is updated using the data.
7. An arrow points from the **Model** back to the **Problem Pool**, completing the cycle. This suggests the model may generate new problems or select problems from the pool for the next iteration.
**Node Color Distribution (Relative Correctness):**
* **High Correctness (Red)**: 1 root node, 2 first-level nodes, 1 second-level node. Total: 4 nodes.
* **Medium-High Correctness (Pink)**: 2 first-level nodes, 1 second-level node. Total: 3 nodes.
* **Medium-Low Correctness (Purple)**: 1 second-level node, 1 third-level node. Total: 2 nodes.
* **Low Correctness (Blue)**: 2 first-level nodes, 2 third-level nodes. Total: 4 nodes.
### Key Observations
1. **Central Path Emphasis**: The most developed branch in the tree stems from the central **red** (high correctness) node at the first level, which itself leads to another **red** node. This visually suggests the search algorithm is focusing exploration on promising, high-correctness paths.
2. **Correctness Gradient**: There is a general trend where nodes closer to the root (especially the central path) tend to be red/pink (higher correctness), while leaf nodes at the periphery are more often blue (lower correctness). This aligns with the typical MCTS goal of identifying and refining good solutions.
3. **Closed-Loop System**: The diagram explicitly shows a feedback loop: Model -> Problem Pool -> MCTS -> Training Data -> Fine-Tune -> Model. This represents an iterative, self-improving training paradigm.
4. **Data Generation Source**: The arrow from the MCTS block to the Training Data Pool implies that the *process of search and evaluation itself* generates valuable training data, not just the final solutions.
### Interpretation
This diagram illustrates a sophisticated **self-play or iterative improvement framework** for AI models, likely in domains like game playing, theorem proving, or complex problem-solving.
* **What it demonstrates**: The system uses Monte Carlo Tree Search not just to solve a single problem, but as a **data engine**. By exploring a problem space (from the Problem Pool) and evaluating the relative correctness of different reasoning steps (visualized by node colors), it curates a dataset of high-quality reasoning traces (sent to the Training Data Pool). This dataset is then used to fine-tune the original Model, making it better at solving such problems in the next cycle.
* **Relationships**: The Model is both the **beneficiary** (it gets fine-tuned) and a **participant** (it likely guides the MCTS or selects problems). The Problem Pool is the **input source**, the MCTS is the **exploration and evaluation engine**, and the Training Data Pool is the **curated output** of that engine.
* **Notable Insight**: The color-coding reveals the **internal state of the search algorithm**. The concentration of red nodes along a central path indicates the algorithm has identified a high-quality solution trajectory. The presence of blue nodes shows it also explores less promising paths, which is essential for avoiding local optima and generating diverse training data. The entire process embodies a **Peircean abductive reasoning cycle**: the model generates hypotheses (problems/paths), tests them via search (MCTS), evaluates the results (correctness coloring), and updates its own knowledge (fine-tuning) based on the most explanatory findings.
</details>
Figure 1: An overview of the proposed methods.
Table 1: Performance on MATH and GSM8K. Results are reported as mean $±$ standard error.
| Method Zero-shot-CoT | Llama-3.1-8B-Instruct MATH $47.07± 0.15$ | deepseek-math-7b-instruct GSM8K $80.77± 0.25$ | MATH $41.20± 0.27$ | GSM8K $78.79± 0.20$ |
| --- | --- | --- | --- | --- |
| RFT | $47.96± 0.21$ | $83.33± 0.25$ | $42.04± 0.15$ | $80.90± 0.18$ |
| Step-level DPO - Iteration 1 | $47.12± 0.31$ | $82.43± 0.27$ | $41.32± 0.21$ | $80.67± 0.25$ |
| Step-level DPO - Iteration 2 | $47.31± 0.19$ | $82.55± 0.17$ | $41.14± 0.10$ | $80.52± 0.23$ |
| Step-level DPO - Iteration 3 | $48.29± 0.18$ | / | $41.48± 0.30$ | / |
| Step-level DPO - Iteration 4 | $48.48± 0.27$ | / | / | / |
| Ours - Iteration 1 | $50.04± 0.07$ | $85.35± 0.24$ | $43.15± 0.28$ | $81.77± 0.33$ |
| Ours - Iteration 2 | $50.84± 0.20$ | $85.80± 0.22$ | $44.11± 0.20$ | $82.02± 0.18$ |
| Ours - Iteration 3 | $51.52± 0.18$ | / | $44.57± 0.27$ | / |
| Ours - Iteration 4 | $51.92± 0.20$ | / | / | / |
Table 2: Results of the transfer experiments where models are trained on one dataset and tested on the other.
| Ours - Iteration 1 Ours - Iteration 2 Ours - Iteration 3 | $48.50± 0.31$ $48.74± 0.26$ / | $85.21± 0.15$ $85.72± 0.14$ $85.53± 0.16$ | $41.15± 0.31$ $41.58± 0.25$ / | $80.36± 0.38$ $81.09± 0.18$ $81.10± 0.17$ |
| --- | --- | --- | --- | --- |
| Ours - Iteration 4 | / | $85.65± 0.21$ | / | / |
Assume that we have a dataset of reasoning problems (e.g., mathematical problems) and their corresponding answers $P=\{(x^i,y^i)\}_i=1^N$ . Our goal is to enhance the reasoning ability of the target LLM with $P$ without human annotation. In addition, we assume that we are not accessible to LLMs stronger than the target LLM, so that our methods can be applied to the strongest LLMs.
The proposed methods are illustrated in Figure 1. We train LLMs in a self-training manner (Zhang et al. 2024b; Zelikman et al. 2022), i.e., training LLMs on the data generated by themselves. We leverage MCTS to sample and search for step-by-step reasoning paths and collect training data from the constructed search tree. The generated training data are then used to perform supervised fine-tuning (SFT) on the LLM. This generate-and-fine-tune process is repeated iteratively until it converges.
In the following sections, we will describe in detail how we generate our training data and train the LLM on the generated data.
### Data Generation
We generate a training dataset $D=\{(x^i,p^i_j,s^i_j,c^i_j)\}_i=1^ {N}$ , where $x^i$ denotes the $i$ -th problem in $P$ , $p^i_j$ denotes the $j$ -th partial solution to $x^i$ , $s^i_j$ denotes the next steps of the partial solution $p^i_j$ , and $c^i_j$ denotes the scores assigned to $s^i_j$ .
For each problem in $P$ , We regard each individual step in problem-solving steps as a tree node, and the steps are separated by two newline characters. We perform MCTS at each step along the reasoning paths generated by the LLM, exploring the best next steps. Specifically, for each partial solution $p^i_j$ of the problem $x^i$ , we follow the iterative procedure below:
1. Selection. Starting from the root node (i.e., $p^i_j$ ), we select the child node with the highest Upper Confidence Bound (UCB) (Kocsis and SzepesvĂĄri 2006) value until the current node is either not fully expanded or represents a final step of the solution (e.g., âThe final answer is 3.â).
1. Expansion. If the current node does not represent a final step and has not been fully expanded, we sample the next step as a new child node using the LLM with a non-zero temperature.
1. Simulation. From the newly expanded node, we sample the continuation of the partial solution using the LLM until it produces a final answer.
1. Backpropagation. After the simulation, we compare the produced final answer with the ground truth, and propagate the reward (1.0 for correct and 0.0 for incorrect) back through the visited nodes in the tree, updating their visit counts and cumulative rewards to guide future searches.
After repeating the above procedure multiple times, we have constructed a tree where the child nodes of the root node represent the next steps of $p^i_j$ . For the $k$ -th node $v^i_j,k$ in the child nodes, its score is computed as follows:
$$
r^i_j,k=α· N(v^i_j,k)·â€ft(\frac{Q(v^i_j,k)}{N(v^i
_j,k)}-\frac{â_mQ(v^i_j,m)}{â_mN(v^i_j,m)}\right) \tag{1}
$$
where $Q(·)$ is the cumulative reward of a node, $N(·)$ is the visit counts of a node, and $α$ is a manually set constant for controlling the scale of the scores. The expression in parentheses captures the ârelative correctnessâ of $v_k$ . Finally, we add 4-tuples $\{(x^i,p^i_j,s^i_j,k,r^i_j,k)\}$ into the training dataset, where $s^i_j,k$ is the step corresponding to $v^i_j,k$ , except for the steps whose score is $0 0$ .
Then, we append the next step with the highest UCB value to $p^i_j$ to obtain a new partial solution, and perform the above procedure again, until the next step is a final step or the maximum solution length limit is reached.
### Iterative Training
We iteratively train the LLM after generating training data using the LLM from the last iteration, starting with the pretrained LLM at the first iteration. In each iteration, a certain number of problems are sampled from $P$ for data generation.
At the $i$ -th iteration, the LLM is trained by minimizing the following loss:
$$
\displaystyleL(Ï_Ξ_{i})=-E_(x,p,s,r)âŒ
D_i[r\logÏ_Ξ_{i}(s\mid x,p))]+ \displaystyleD_KL(Ï_Ξ_{i}(s\mid x,p)\parallelÏ_Ξ_{
i-1}(s\mid x,p)) \tag{2}
$$
where $Ï_Ξ_{i}$ denotes the LLM at the $i$ -th iteration. The first term is weighted negative log-likelihood. Inspired by reinforcement learning from human feedback (Ouyang et al. 2022), we incorporate a KL penalty in the second term to mitigate the distribution shift, which is a challenge in offline reinforcement learning. In fact, our training method can be regarded as a form of reinforcement learning.
## Experiments
### Setup
We apply the proposed methods to Llama-3.1-8B-Instruct https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct and deepseek-math-7b-instruct https://huggingface.co/deepseek-ai/deepseek-math-7b-instruct, and evaluate the performance on two popular mathematics datasets, GSM8K (Cobbe et al. 2021) and MATH (Hendrycks et al. 2021).
For comparison, the first baseline is Zero-shot-CoT (Kojima et al. 2022). The second baseline is Rejective Sampling Fine-Tuning (RFT) (Yuan et al. 2023), which generates training data by sampling the correct solutions generated by the LLM itself.
To demonstrate that our data generation method is superior to selecting the best and worst steps (i.e., the steps with the highest and lowest average rewards, respectively, similar to the method in (Xie et al. 2024)) to form preference pairs, we also iteratively train LLMs on the perference data generated in this way. We use DPO for preference optimization, and this baseline is referred to as Step-level DPO.
For efficient training, all models are trained using Low-Rank Adaptation (LoRA) (Hu et al. 2021) .
### Main Results
The experimental results are presented in Table 1. We report the results of iterative training until the accuracies do not increase anymore. It can be observed that our methods consistently outperform the baselines by large margins, with accuracies improving during iterative training. However, the performance converges quickly and fails to continually improve over more iterations. As a result, not all the problems in the training sets of MATH and GSM8K are used for training: only about 2,000 in MATH and 2,000 in GSM8K are used. In addition, the performance converges more quickly on GSM8K than on MATH, which we attribute to the lower difficulty of GSM8K, making it easier for models to learn. It is noteworthy that Step-level DPO achieves only very marginal improvements on MATH, which indicates that our data generation method is significantly superior.
### Transferability Evaluation
The MATH dataset consists of high school math competition problems, while GSM8K comprises grade school math problems. If the LLMsâ mathematical reasoning ability has been improved, they should perform better on both datasets. Therefore, we evaluate the transferability of the models by training it on MATH/GSM8K and testing it on GSM8K/MATH to verify whether their mathematical reasoning ability has indeed been enhanced.
As shown in Table 2, our methods outperform Zero-shot-CoT on unseen datasets, indicating they indeed learn mathematical reasoning ability. As expected, the improvements are less substantial than those observed in non-transfer experiments, since the problems in the two datasets require different mathematical skills and knowledge.
## Conclusion and Limitations
In this work, we propose a process supervision data generation method utilizing MCTS and a training approach, for improving the reasoning ability of LLMs. We evaluate the proposed methods on two well-known mathematical datasets and demonstrating the effectiveness. The trained models also outperform the pre-trained models on datasets they are not trained on, showing the transferability of their learned knowledge and skills.
However, the performance converges quickly during iterative training, failing to continually improve over many iterations, and we do not even use all the problems in the training sets. Training for more iterations or using more problems not only fails to improve the performance but actually degrades it. Future research could study the underlying reasons for this phenomenon and how to achieve more substantial improvements with more iterations.
In addition, the models in our experiments are trained using LoRA, which could have limited the magnitude of the observed improvements.
## References
- Browne et al. (2012) Browne, C. B.; Powley, E.; Whitehouse, D.; Lucas, S. M.; Cowling, P. I.; Rohlfshagen, P.; Tavener, S.; Perez, D.; Samothrakis, S.; and Colton, S. 2012. A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1): 1â43.
- Chen et al. (2024) Chen, G.; Liao, M.; Li, C.; and Fan, K. 2024. AlphaMath Almost Zero: process Supervision without process. arXiv preprint arXiv:2405.03553.
- Cobbe et al. (2021) Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
- Fu et al. (2022) Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations.
- Hao et al. (2023) Hao, S.; Gu, Y.; Ma, H.; Hong, J. J.; Wang, Z.; Wang, D. Z.; and Hu, Z. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992.
- Hendrycks et al. (2021) Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
- Hu et al. (2021) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
- Huang and Chang (2023) Huang, J.; and Chang, K. C.-C. 2023. Towards Reasoning in Large Language Models: A Survey. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Findings of the Association for Computational Linguistics: ACL 2023, 1049â1065. Toronto, Canada: Association for Computational Linguistics.
- Hwang et al. (2024) Hwang, H.; Kim, D.; Kim, S.; Ye, S.; and Seo, M. 2024. Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards. In Findings of the Association for Computational Linguistics: EMNLP 2024, 1444â1466.
- Kocsis and SzepesvĂĄri (2006) Kocsis, L.; and SzepesvĂĄri, C. 2006. Bandit based monte-carlo planning. In European conference on machine learning, 282â293. Springer.
- Kojima et al. (2022) Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199â22213.
- Lai et al. (2024) Lai, X.; Tian, Z.; Chen, Y.; Yang, S.; Peng, X.; and Jia, J. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629.
- Li et al. (2024) Li, C.; Wang, W.; Hu, J.; Wei, Y.; Zheng, N.; Hu, H.; Zhang, Z.; and Peng, H. 2024. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706.
- Lightman et al. (2023) Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Letâs verify step by step. arXiv preprint arXiv:2305.20050.
- Liu et al. (2024) Liu, H.; Zhang, Y.; Luo, Y.; and Yao, A. C.-C. 2024. Augmenting math word problems via iterative question composing. arXiv preprint arXiv:2401.09003.
- Lu et al. (2024) Lu, Z.; Zhou, A.; Ren, H.; Wang, K.; Shi, W.; Pan, J.; Zhan, M.; and Li, H. 2024. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352.
- Luo et al. (2024) Luo, L.; Liu, Y.; Liu, R.; Phatale, S.; Lara, H.; Li, Y.; Shu, L.; Zhu, Y.; Meng, L.; Sun, J.; et al. 2024. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv preprint arXiv:2406.06592.
- Nye et al. (2021) Nye, M.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114.
- Ouyang et al. (2022) Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730â27744.
- Pang et al. (2024) Pang, R. Y.; Yuan, W.; Cho, K.; He, H.; Sukhbaatar, S.; and Weston, J. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733.
- Qi et al. (2024) Qi, Z.; Ma, M.; Xu, J.; Zhang, L. L.; Yang, F.; and Yang, M. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195.
- Rafailov et al. (2024) Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
- Trung et al. (2024) Trung, L.; Zhang, X.; Jie, Z.; Sun, P.; Jin, X.; and Li, H. 2024. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7601â7614.
- Valmeekam et al. (2022) Valmeekam, K.; Olmo, A.; Sreedharan, S.; and Kambhampati, S. 2022. Large language models still canât plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.
- Wang et al. (2024a) Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024a. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9426â9439.
- Wang et al. (2022) Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
- Wang et al. (2024b) Wang, Z.; Li, Y.; Wu, Y.; Luo, L.; Hou, L.; Yu, H.; and Shang, J. 2024b. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658.
- Wei et al. (2022) Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824â24837.
- Xie et al. (2024) Xie, Y.; Goyal, A.; Zheng, W.; Kan, M.-Y.; Lillicrap, T. P.; Kawaguchi, K.; and Shieh, M. 2024. Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning. arXiv preprint arXiv:2405.00451.
- Yao et al. (2024) Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.
- Yu et al. (2023) Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
- Yuan et al. (2023) Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Lu, K.; Tan, C.; Zhou, C.; and Zhou, J. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825.
- Zelikman et al. (2022) Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 15476â15488.
- Zhang et al. (2024a) Zhang, D.; Li, J.; Huang, X.; Zhou, D.; Li, Y.; and Ouyang, W. 2024a. Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B. arXiv preprint arXiv:2406.07394.
- Zhang et al. (2024b) Zhang, D.; Zhoubian, S.; Hu, Z.; Yue, Y.; Dong, Y.; and Tang, J. 2024b. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816.
- Zhao et al. (2023) Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.