# R-PRM: Reasoning-Driven Process Reward Modeling
Abstract
Large language models (LLMs) inevitably make mistakes when performing step-by-step mathematical reasoning. Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step. However, existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy, which is further exacerbated by the scarcity of annotated data. To address these issues, we propose Reasoning-Driven Process Reward Modeling (R-PRM). First, we leverage stronger LLMs to generate seed data from limited annotations, effectively bootstrapping our model’s reasoning capabilities and enabling comprehensive step-by-step evaluation. Second, we further enhance performance through preference optimization, without requiring additional annotated data. Third, we introduce inference-time scaling to fully harness the model’s reasoning potential. Extensive experiments demonstrate R-PRM’s effectiveness: on ProcessBench and PRMBench, it surpasses strong baselines by 11.9 and 8.5 points in F1 scores, respectively. When applied to guide mathematical reasoning, R-PRM achieves consistent accuracy improvements of over 8.5 points across six challenging datasets. Further analysis reveals that R-PRM exhibits more comprehensive evaluation and stronger generalization capabilities, thereby highlighting its significant potential The project is available at https://github.com/NJUNLP/R-PRM.
R-PRM: Reasoning-Driven Process Reward Modeling
Shuaijie She 1 ∗, Junxiao Liu 1 ∗, Yifeng Liu 1, Jiajun Chen 1, Xin Huang 2, Shujian Huang 1 †1 National Key Laboratory for Novel Software Technology, Nanjing University 2 China Mobile Communications Company Limited Research Institute {shesj,junxiaoliu,yfliu}@smail.nju.edu.cn, chenjj@nju.edu.cn, huangxinyjy@chinamobile.com, huangsj@nju.edu.cn ∗ Equal contributions. †Corresponding author.
1 Introduction
Recently, large language models (LLMs) have demonstrated significant progress in solving challenging mathematical problems through chain-of-thought reasoning Wei et al. (2023); Yang et al. (2024); Shao et al. (2024). However, LLMs still tend to make reasoning errors, undermining the reliability of their solutions and hindering their ability to arrive at correct answers.
Therefore, Process Reward Model (PRM) has been proposed to further improve models’ reasoning ability Lightman et al. (2023). Unlike Outcome Reward Models (ORM) that only focus on the final results, PRMs evaluate each reasoning step in a more fine-grained manner and achieve better performance and generalization, such as identifying and mitigating process errors, while also exhibiting stronger generalization capabilities Lightman et al. (2023); Wang et al. (2024b).
A primary challenge in PRM development arises from data scarcity. While human annotation can provide high-quality process-level labels Lightman et al. (2023), it incurs substantial costs. Alternative automated approaches, such as Monte Carlo (MC) methods that estimate step correctness based on the probability of reaching the correct final answer Wang et al. (2024b, a); Luo et al. (2024b), or methods that use stronger language models as judges for data filtering Zhang et al. (2025), have shown some promise. However, these methods either require significant computational resources or still struggle with noise and bias, leaving the challenge of sufficient high-quality training data unresolved.
Moreover, existing process reward models directly provide evaluations based on the given steps. We argue that for challenging process-level evaluation tasks, this direct evaluation approach constrains the model’s learning process and reduces learning efficiency. Moreover, it lacks interpretability, as it fails to identify why specific steps are incorrect, making it difficult to provide constructive feedback for improvement.
To address these issues, we propose a Reasoning-Driven Process Reward Modeling (R-PRM) framework, which utilizes reasoning over each intermediate step to improve process-level evaluation. The framework consists of three key components: First, we construct seed data by prompting stronger LLMs with a small amount of human-annotated step-level labels and subsequently fine-tune Qwen2.5-Math-7B-Instruct. Through this reasoning-centric paradigm, our model develops the capability to perform comprehensive and transparent analyses for evaluating complex solution steps of challenging questions. Second, the generative evaluation paradigm enables us to apply preference optimization, improving model capabilities by encouraging the generation of evaluation processes that lead to correct judgments, without requiring additional data generation. Finally, we exploit the generative nature of our evaluation paradigm at inference time, allowing multiple evaluation processes to be sampled for a more comprehensive and robust assessment perspective.
When evaluated on ProcessBench and PRMBench, our R-PRM achieves F1 score improvements of 11.9 and 8.5 points, respectively, over the strongest baseline trained on the same data. Furthermore, when used to guide policy model reasoning via Best-of-N and Guided Search strategies, our approach improves accuracy by average margins of 8.6 and 8.4 points over the Pass@1 baseline across six challenging math datasets, outperforming both majority voting and all existing PRM baselines. Further analysis reveals our three key additional advantages: (1) comprehensive evaluation coverage through multi-dimension analysis, (2) enhanced generalization capability across diverse datasets, and (3) progressive accuracy improvement with increased reasoning budgets, suggesting significant potential for practical reasoning-system optimization.
2 Related Work
2.1 Mathematical Reasoning
Recent studies have demonstrated that LLMs exhibit enhanced reasoning capabilities when generating step-by-step solutions before providing the final answers Wei et al. (2023). Building on this insight, several pioneering works have focused on developing large-scale mathematical datasets with high-quality reasoning annotations for fine-tuning of LLMs Luo et al. (2025); Wang et al. (2023); Shao et al. (2024); Yang et al. (2024). However, even when models arrive at correct final answers, their intermediate reasoning steps may contain critical errors. This discrepancy undermines the reliability of their problem-solving processes and poses significant obstacles for future model improvements Zheng et al. (2024).
Parallel advancements Snell et al. (2024); O1 (2023); DeepSeek-AI (2025); QwQ (2023) in inference-time have demonstrated that increasing the computational budget to enable multiple reasoning attempts, coupled with majority voting mechanisms for answer selection, can achieve remarkable accuracy improvements.
2.2 Reward Modeling of Reasoning
Reward models are introduced to further improve mathematical reasoning by enhancing training data quality, guiding model learning Lightman et al. (2023); Cobbe et al. (2021); Uesato et al. (2022), and guiding the policy model’s reasoning process through Best-of-N and Guided-Search methods Wang et al. (2024b); Zhang et al. (2025).
Currently, reward models are typically categorized into Outcome Reward Models (ORM) and Process Reward Models (PRM) Lightman et al. (2023). ORM focuses on providing an overall evaluation based on whether the correct answer is ultimately obtained Cobbe et al. (2021).. In contrast, PRM provides a fine-grained evaluation for each reasoning step, and many works have shown that PRM can achieve better results Lightman et al. (2023); Uesato et al. (2022). However, data for PRM is extremely scarce, and the annotation is costly Lightman et al. (2023); Wang et al. (2024b); Luo et al. (2024b). Some studies explore automatic synthesis strategies, such as using Monte Carlo (MC) estimation methods Wang et al. (2024b); Luo et al. (2024b). However, MC methods incur a large computational cost and inevitably introduce bias and noise Zheng et al. (2024). Zhang et al. (2025) propose combining MC with LLM as a judge, helping to reduce noise. However, the quality and quantity of step-level reasoning evaluation data are still limited, and this remains an unsolved challenge.
3 Method
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: R-PRM Training and Reward Generation Pipeline
### Overview
This image is a technical system architecture diagram illustrating a two-part machine learning pipeline. The left side depicts the data collection and training process for a Process Reward Model (R-PRM) using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). The right side, separated by a vertical dashed line, demonstrates the inference phase where the trained model evaluates an input through sampling to calculate a final average reward score.
*Language Declaration:* All text in this image is in English. No other languages are present.
### Components
The diagram utilizes several distinct visual components to convey information:
* **Containers:** Rounded rectangles in various colors (light blue, light green, light red, purple) representing data inputs, outputs, and analytical steps.
* **Icons:**
* A basic white robot head (representing a base Large Language Model).
* A blue robot head with a red siren/light on top (representing the specialized R-PRM model).
* Stacked green coins/disks (representing the SFT Dataset).
* Stacked blue disks (representing the Preference Dataset).
* Green checkmarks ($\checkmark$) and red crosses ($\times$) indicating correctness.
* **Connectors:** Solid black arrows indicating process flow and data movement. Curly brackets grouping data elements.
* **Dividers:** A vertical dashed gray line separating the training phase (left) from the inference phase (right). A dashed rectangular box grouping numerical scores.
### Content Details
#### Left Section: Training Pipeline (Data Collection & Model Training)
**1. Input Generation (Top Left):**
* A large light blue box is labeled **"Evaluation Input"**.
* Inside this box are three smaller light blue boxes labeled: **"Problem"**, **"Previous Steps"**, and **"Now Step"**.
* To the right of the Evaluation Input box is a small purple box labeled **"Label: No"**.
* Below the Evaluation Input box, text reads: **"Collect response from LLM to construct seed data"**. A black arrow points downward from this text.
**2. Seed Data Construction (Middle Left):**
Below the arrow are three horizontal boxes representing generated analyses.
* **Box 1 (Light Green):**
* Text: "Previous Steps Analysis: This step starts by ..."
* Text: "......"
* Text: "Verification: Is the step correct (Yes/No)? No"
* Symbol: A large green checkmark ($\checkmark$) is positioned at the bottom right.
* **Box 2 (Light Green):**
* Text: "......"
* Text: "Now Step Analysis: Now Step checks if 23 is ..."
* Text: "Verification: Is the step correct (Yes/No)? No"
* Symbol: A large green checkmark ($\checkmark$) is positioned at the bottom right.
* **Box 3 (Light Red):**
* Text: "......"
* Text: "Calculation Analysis: The calculation in the..."
* Text: "Verification: Is the step correct (Yes/No)? Yes"
* Symbol: A large red 'X' ($\times$) is positioned at the bottom right.
**3. Dataset Creation and Model Training (Center Column):**
* A black curly bracket groups **Box 1** and **Box 2** (the green boxes). This bracket points to a green stacked-disk icon labeled **"SFT Dataset"**.
* A second black curly bracket groups **Box 2** (green) and **Box 3** (red). This bracket points to a blue stacked-disk icon labeled **"Preference Dataset"**.
* Above the datasets, a white robot icon has a downward arrow pointing to text **"R-PRM SFT"**.
* Below "R-PRM SFT" is a blue robot icon. A downward arrow points from it to text **"R-PRM DPO"**.
* Below "R-PRM DPO" is another instance of the blue robot icon, representing the final trained model.
#### Right Section: Inference Pipeline (Evaluation & Reward)
**1. Input and Model (Top Right):**
* A light blue box labeled **"Evaluation Input"** sits at the top.
* A downward arrow points to the blue robot icon (the trained R-PRM model).
**2. Sampling and Verification (Middle Right):**
* A downward arrow labeled **"Sampling"** points from the blue robot to a stack of three overlapping boxes.
* The front box is light red. The text inside reads:
* "......"
* "Verification: Is the step correct (Yes/No)? No"
* The two boxes behind it are light green. The letters "ct" (likely the end of the word "correct") are visible on the right edge of the middle box.
**3. Reward Calculation (Bottom Right):**
* Three downward arrows point from the stacked boxes into a dashed rectangular outline.
* Inside the dashed outline are three colored circles containing numbers:
* Red circle: **"0.4"**
* Green circle: **"0.9"**
* Green circle: **"0.8"**
* To the right of the dashed outline is the word **"Average"**.
* A downward arrow points from the dashed outline to a purple box labeled **"Reward: 0.7"**.
### Key Observations
* **Color Coding:** Green is consistently used to denote correct verifications or positive outcomes (checkmarks, SFT dataset, high scores). Red is used to denote incorrect verifications or negative outcomes (X marks, low scores). Light blue denotes inputs, and purple denotes final labels/rewards.
* **Mathematical Consistency:** In the reward calculation phase, the three sampled scores are 0.4, 0.9, and 0.8. The sum of these is 2.1. Divided by 3 (to find the average), the result is exactly 0.7, which matches the final "Reward: 0.7" box.
* **Dataset Grouping:** The SFT (Supervised Fine-Tuning) dataset is built exclusively from "correct" (green) examples. The Preference Dataset for DPO (Direct Preference Optimization) is built by contrasting a "correct" (green) example with an "incorrect" (red) example.
### Interpretation
This diagram outlines a sophisticated methodology for training a Process Reward Model (R-PRM) designed to evaluate the step-by-step reasoning of Large Language Models (LLMs).
**The Training Phase (Left):**
The system begins by taking a mathematical or logical problem, its previous steps, and the current step being evaluated. A base LLM generates analyses of these steps.
Crucially, the system evaluates the LLM's own verification of the steps. If the LLM correctly identifies a step as flawed (Green boxes: "Is the step correct? No" -> $\checkmark$), this good behavior is packaged into an **SFT Dataset** to teach the model basic competence.
To make the model highly robust, a **Preference Dataset** is created by pairing a correct verification (Green box) against an incorrect verification (Red box: "Is the step correct? Yes" -> $\times$ when it shouldn't be). This paired data is used for **DPO (Direct Preference Optimization)**, which trains the model to actively prefer the correct analytical reasoning over the flawed reasoning. The evolution of the robot icon from white to blue-with-a-siren visually represents the model gaining this specialized "evaluator/police" capability.
**The Inference Phase (Right):**
Once trained, the R-PRM is put to work. When given a new "Evaluation Input", it doesn't just generate one answer. It uses a **Sampling** technique, generating multiple verification paths (represented by the stacked boxes).
Each sampled path yields a confidence score (0.4, 0.9, 0.8). The red circle for 0.4 correlates with the red box in the sample stack, indicating a lower confidence or a negative verification path, while the green circles (0.9, 0.8) represent higher confidence/positive paths. By averaging these sampled scores, the system produces a final, stabilized **Reward** of 0.7.
*Reading between the lines:* This architecture suggests an attempt to mitigate LLM hallucinations in complex reasoning tasks (like math or coding). By training a model specifically to verify individual steps (Process Reward) rather than just the final answer (Outcome Reward), and by using multi-path sampling to smooth out anomalies, the resulting reward score is likely much more reliable for guiding reinforcement learning or search algorithms (like Monte Carlo Tree Search).
</details>
Figure 1: Illustration of our framework. For brevity, only partial analytical reasoning trajectories are shown. White robots indicate initial models, while colored ones represent models after our training procedure.
3.1 Reasoning for Process Reward Modeling
Given a mathematical problem $Q$ , the policy model generates a sequential chain-of-reasoning process $S=\{s_{1},s_{2},...,s_{n}\}$ , where each reasoning step $s_{i}$ is generated conditioned on both $Q$ and the preceding steps $\{s_{1},...,s_{i-1}\}$ . To evaluate the quality of each reasoning step, current process-level reward models employ a direct prediction mechanism that assigns a score to each step. This evaluation process can be formally expressed as:
$$
R_{i}=M(Q,s_{1},...,s_{i})
$$
where $M(·)$ represents the reward model that outputs a scalar reward $R_{i}$ .
Unlike previous approaches, we propose a reasoning-driven process-level reward model $G$ that operates in a single generation process to perform two phases. First, $G$ generates a comprehensive analysis $A_{i}$ of each reasoning step $s_{i}$ , consisting of multiple analytical dimensions: examining historical reasoning steps, assessing the objective and data sources of the current step, verifying its coherence with preceding steps, and validating the computational transformations involved. Then, following the analysis, model generates a judgment $J_{i}$ which is is a natural language phrase indicating the correctness of the step, expressed as "yes" or "no":
$$
A_{i}=G(Q,s_{1},...,s_{i})
$$
$$
J_{i}=G(Q,s_{1},...,s_{i},A_{i})
$$
We prompt a stronger LLM with samples from PRM800K to generate ( $Q$ , $s_{1:i}$ , $A_{i}$ , $J_{i}$ ) tuples The prompt we used is listed in Appendix E. We retain only those evaluation analyses that produce a judgment consistent with the human label. Subsequently, we concatenate the analysis and score as the target sequence. Let $Y_{i}$ denote the complete output sequence for $s_{i}$ :
$$
Y_{i}=A_{i}\oplus J_{i}=\{y_{1},y_{2},...,y_{t}\}
$$
$$
\mathcal{L}_{\text{SFT}}=-\sum_{j=1}^{t}\log p(y_{j}|Q,s_{1:i},y_{1:j-1})
$$
where $y_{j}$ denotes the $j$ -th token in the output sequence $Y_{i}$ , and $t$ is the total length of the sequence. This is equivalent to standard instruction tuning, where the model learns to generate both the analysis and the judgment in a single forward pass.
3.2 Process Reward Modeling Meta-Optimization
Through supervised fine-tuning (SFT), our model learns to effectively evaluate complex reasoning trajectories in mathematical problems by leveraging its reasoning capabilities. We further enhance the model’s performance by optimizing its reasoning process. More specifically, we leverage preference optimization methods like Direct Preference Optimization (DPO) Rafailov et al. (2024) to encourage the model to generate reasoning processes that yield correct judgments.
DPO involves an input pair $(Y^{w},Y^{l})$ , where $Y^{w}$ is favored over $Y^{l}$ . Correspondingly, we sample multiple evaluation reasoning processes, which are categorized into two groups based on whether their final predictions match the annotated labels. We treat consistent trajectories as $Y^{w}$ and inconsistent ones as $Y^{l}$ to construct preference pairs. We freeze the original supervised fine-tuned LLM as the reference policy $\pi_{\text{ref}}$ and optimize our model using the following loss function:
| | $\displaystyle\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{%
E}_{(x,Y^{w},Y^{l})\sim\mathcal{D}}$ | |
| --- | --- | --- |
3.3 Inference Time Scaling Strategy
Leveraging our PRM’s capability to generate analytical reasoning processes, we propose a scalable inference strategy that enhances evaluation robustness through multiple reasoning trajectories. During inference, for each reasoning step $s_{i}$ , we sample $K$ independent analytical processes as follows:
$$
(A_{i}^{(k)},J_{i}^{(k)})=G(Q,s_{1},...,s_{i}),k\in[1,K]
$$
where each $A^{(k)}$ represents a distinct analytical reasoning process and $R^{(k)}$ is the corresponding judgment ("yes" or "no"). This multi-trajectory approach helps mitigate potential reasoning inconsistencies and stochastic variations inherent in large language models. To aggregate multiple evaluations, we calculate the average probability of "yes" judgments (using softmax with "no" judgments) as the reward:
$$
R_{i}=\frac{1}{K}\sum_{k=1}^{K}P(J_{i}^{(k)}=\text{"yes"}|Q,s_{1},...,s_{i},A_%
{i}^{(k)}).
$$
4 Experiment
| MODEL | GSM8K | MATH | OLYMPIADBENCH | OMNIMATH | Avg. F1 | | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| error | correct | F1 | error | correct | F1 | error | correct | F1 | error | correct | F1 | | |
| LLM-as-judge, Proprietary language models | | | | | | | | | | | | | |
| GPT-4o* | 70.0 | 91.2 | 79.2 | 54.4 | 76.6 | 63.6 | 45.8 | 58.4 | 51.4 | 45.2 | 65.6 | 53.5 | 61.9 |
| o1-mini* | 88.9 | 97.9 | 93.2 | 83.5 | 95.1 | 88.9 | 80.2 | 95.6 | 87.2 | 74.8 | 91.7 | 82.4 | 87.9 |
| LLM-as-judge, Open-source language models | | | | | | | | | | | | | |
| Llama-3.3-70B-Instruct* | 72.5 | 96.9 | 82.9 | 43.3 | 83.2 | 59.4 | 31.0 | 94.1 | 46.7 | 28.2 | 90.5 | 43.0 | 58.0 |
| Qwen2.5-Math-72B-Instruct* | 49.8 | 96.9 | 65.8 | 36.0 | 94.3 | 52.1 | 19.5 | 97.3 | 32.5 | 19.0 | 96.3 | 31.7 | 45.5 |
| Qwen2.5-72B-Instruct* | 62.8 | 96.9 | 76.2 | 46.3 | 93.1 | 61.8 | 38.7 | 92.6 | 54.6 | 36.6 | 90.9 | 52.2 | 61.2 |
| PRMs | | | | | | | | | | | | | |
| Math-Shepherd-7B* | 32.4 | 91.7 | 47.9 | 18.0 | 82.0 | 29.5 | 15.0 | 71.1 | 24.8 | 14.2 | 73.0 | 23.8 | 31.5 |
| Math-PSA-7B | 48.3 | 88.1 | 62.4 | 29.5 | 72.7 | 41.9 | 20.7 | 65.8 | 31.5 | 15.4 | 68.9 | 25.2 | 40.3 |
| RLHFlow-Mistral-8B* | 33.8 | 99.0 | 50.4 | 21.7 | 72.2 | 33.4 | 8.2 | 43.1 | 13.8 | 9.6 | 45.2 | 15.8 | 28.4 |
| RLHFlow-DeepSeek-8B* | 24.2 | 98.4 | 38.8 | 21.4 | 80.0 | 33.8 | 10.1 | 51.0 | 16.9 | 10.9 | 51.9 | 16.9 | 26.6 |
| Llemma-PRM800K-7B | 36.7 | 71.0 | 48.4 | 39.2 | 47.8 | 43.1 | 33.1 | 25.1 | 28.5 | 35.4 | 31.5 | 33.4 | 38.4 |
| Skywork-PRM-7B* | 61.8 | 82.9 | 70.8 | 43.8 | 62.2 | 53.6 | 17.9 | 31.9 | 22.9 | 14.0 | 41.9 | 21.0 | 42.1 |
| ReasonEval-7B | 26.1 | 95.3 | 41.0 | 35.7 | 77.6 | 48.9 | 27.5 | 55.2 | 36.7 | 27.0 | 60.6 | 37.4 | 41.0 |
| Qwen2.5-Math-7B-PRM800K* | 53.1 | 95.3 | 68.2 | 48.0 | 90.1 | 62.6 | 35.7 | 87.3 | 50.7 | 29.8 | 86.1 | 44.3 | 58.5 |
| Qwen2.5-Math-PRM-7B* | 72.0 | 96.4 | 82.4 | 68.0 | 90.4 | 77.6 | 55.7 | 85.5 | 67.5 | 55.2 | 83.0 | 66.3 | 73.5 |
| $\bigstar$ R-PRM-7B-SFT | 66.2 | 92.7 | 77.2 | 60.3 | 88.2 | 71.6 | 48.6 | 77.3 | 59.6 | 40.1 | 75.5 | 52.3 | 65.2 |
| $\bigstar$ R-PRM-7B-DPO | 72.0 | 91.7 | 80.7 | 71.2 | 83.5 | 76.9 | 60.2 | 67.8 | 63.8 | 55.5 | 65.6 | 60.1 | 70.4 |
Table 1: Performance on ProcessBench. $\bigstar$ represents the models we trained. Results marked with * come from Zhang et al. Bold text denotes the best results within PRM.
| Model Name | Simplicity | Soundness | Sensitivity | Overall | | | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NR. | NCL. | Avg. | ES | SC. | DC. | CI | Avg. | PS | DR. | MS. | Avg. | | |
| LLM-as-judge, Proprietary language models | | | | | | | | | | | | | |
| GPT-4o* | 57.0 | 62.4 | 59.7 | 72.0 | 69.7 | 70.7 | 71.1 | 70.9 | 62.5 | 65.7 | 99.2 | 75.8 | 66.8 |
| o1-mini* | 65.6 | 63.7 | 64.6 | 74.5 | 67.7 | 73.8 | 72.3 | 72.1 | 61.8 | 64.8 | 100.0 | 75.5 | 68.8 |
| PRMs | | | | | | | | | | | | | |
| Math-Shepherd-7B* | 44.0 | 50.3 | 47.1 | 49.4 | 44.5 | 41.3 | 47.7 | 45.7 | 47.2 | 48.6 | 86.1 | 60.7 | 47.0 |
| Math-PSA-7B | 47.6 | 55.1 | 51.3 | 56.5 | 49.4 | 47.1 | 54.2 | 51.8 | 51.7 | 54.1 | 88.9 | 64.9 | 52.3 |
| RLHFlow-Mistral-8B* | 46.1 | 47.3 | 46.7 | 56.6 | 55.1 | 54.4 | 63.8 | 57.5 | 51.5 | 56.2 | 97.9 | 68.5 | 54.4 |
| RLHFlow-DeepSeek-8B* | 46.4 | 48.9 | 47.6 | 55.7 | 55.0 | 53.2 | 66.2 | 57.5 | 49.0 | 55.4 | 99.8 | 68.1 | 54.2 |
| Llemma-PRM800k-7B* | 49.3 | 53.4 | 51.4 | 56.4 | 47.1 | 46.7 | 53.3 | 50.9 | 51.0 | 53.5 | 93.6 | 66.0 | 52.0 |
| Skywork-PRM-7B* | 35.7 | 41.2 | 38.4 | 36.7 | 29.1 | 30.6 | 34.4 | 32.7 | 36.8 | 37.4 | 88.8 | 54.3 | 36.2 |
| ReasonEval-7B* | 61.0 | 50.1 | 55.5 | 62.1 | 65.9 | 61.5 | 66.0 | 63.9 | 55.6 | 58.0 | 99.5 | 71.0 | 60.0 |
| Qwen2.5-Math-7B-PRM800K | 48.6 | 47.8 | 48.2 | 62.1 | 59.4 | 58.7 | 68.5 | 62.2 | 52.9 | 64.0 | 99.8 | 72.2 | 58.3 |
| Qwen2.5-Math-PRM-7B | 49.0 | 55.1 | 52.1 | 71.8 | 67.3 | 66.3 | 78.5 | 71.0 | 57.6 | 69.1 | 99.7 | 75.5 | 65.5 |
| $\bigstar$ R-PRM-7B-SFT | 52.7 | 64.7 | 58.7 | 70.1 | 62.7 | 63.4 | 69.5 | 66.4 | 61.4 | 67.4 | 98.3 | 75.7 | 64.9 |
| $\bigstar$ R-PRM-7B-DPO | 52.2 | 58.2 | 55.2 | 72.1 | 69.1 | 68.9 | 75.0 | 71.2 | 61.2 | 69.5 | 99.1 | 76.6 | 66.8 |
Table 2: Performance on PRMBench. $\bigstar$ represents the models we trained. Results marked with * come from Zhang et al. Bold text denotes the best results within PRM.
4.1 Experiment Settings
Tasks and Benchmarks:
To validate the effectiveness of our method for process-level reward modeling, we evaluate it on two challenging benchmarks: ProcessBench Zheng et al. (2024) and PRMBench Song et al. (2025).
- ProcessBench Zheng et al. (2024) is designed to assess the ability to identify erroneous steps in mathematical reasoning processes. The dataset comprises 3,400 test cases covering mathematical problems of varying difficulty levels. Each test case includes a question accompanied by an LLM generated step-by-step solution, with the earliest incorrect step meticulously annotated by human experts. The primary task for the model is to either accurately identify the erroneous step or confirm the correctness of all steps in the solution.
- PRMBench Song et al. (2025) constitutes a comprehensive benchmark for evaluating PRMs, with particular emphasis on granular error diagnosis. It assesses PRM capabilities across three key dimensions: Simplicity, Soundness, and Sensitivity. These dimensions are further subdivided into nine specific aspects. A detailed description of these nine aspects is provided in Appendix A.
Furthermore, we validate the effectiveness of the reward model by applying it to guide the test-time scaling of the policy model Best-of-N, Greedy Guide Search, separately. Specifically, we evaluate on multiple benchmarks, including MATH500 Lightman et al. (2023), Minerva Math Lewkowycz et al. (2022), OlympiadBench He et al. (2024), College Math Tang et al. (2024), AIME24, AMC23 Due to the large size of the Olympiad and College Math test sets, we randomly selected 200 samples from each to form the test subsets.. Following Zhang et al. (2025), we use Qwen2.5-7B-Instruct to sample eight candidate steps with a temperature of 1.
- Best-of-N: selects the response with the highest score among $N$ candidates, as evaluated by a PRM.
- Greedy Guide Search: at each generation step, the model generates $N$ candidate continuations and selects the one with the highest score, as determined by a PRM, to extend the reasoning trajectory. This procedure is repeated until a complete solution is produced.
Baselines:
We selected the following strong process reward models as baselines.
- Math-Shepherd Wang et al. (2024b): Automatically obtaining the probability of reaching the correct solution as step labels based on Monte Carlo Tree Search (MCTS).
- Math-PSA Wang et al. (2024a): combining existing automatic annotation techniques Luo et al. (2024a) and integrating data from Math-Shepherd and PRM800K datasets.
- RLHFlow-DeepSeek/Mistral Dong et al. (2024): Similar to Math-Shepherd, but trained with iterative DPO.
- Skywork-PRM-7B o1 Team (2024): based on Qwen2.5-Math-Instruct and recently released by Skywork.
- ReasonEval-7B Xia et al. (2025): Evaluates mathematical problem-solving step by step, assessing validity and redundancy.
- Llemma-PRM800K-7B Sun et al. (2024): Trained exclusively on PRM800K from levels 1 through 3.
- Qwen2.5-Math-PRM-7B Zhang et al. (2025): Based on Math-Shepherd, additionally employs LLM-as-Judge to perform consistency filtering and result in a 1.8M samples training dataset.
Implement details:
We process each PRM800K case by prompting LLaMA3.3-70B-Instruct to generate four candidate evaluation trajectories, resulting in approximately 289k SFT samples and 269k DPO samples. We train the model on Qwen2.5-Math-7B-Instruct using one epoch of fine-tuning, with a batch size of 128 and separate learning rates: 5e-6 for SFT and 5e-7 for DPO. We set aside 20k samples for validation and retain the checkpoint with the lowest validation loss. By default, we report results using ten evaluation reasoning trajectories per step.
4.2 Experiment Results
| Setting | AIME24 | AMC23 | MATH | Olympiad Bench | College MATH | Minerva MATH | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| pass@1 | 11.2 | 47.8 | 73.0 | 38.0 | 38.6 | 37.2 | 41.0 |
| major@8 | 20.0 | 57.5 | 79.6 | 47.0 | 41.5 | 42.7 | 48.0 |
| pass@8(Upper Bound) | 33.3 | 82.5 | 88.8 | 58.5 | 47.5 | 57.7 | 61.4 |
| Greedy Guide Search@8 | | | | | | | |
| Math-Shepherd-7B | 13.3 | 52.5 | 74.6 | 38.5 | 36.5 | 41.2 | 42.8 |
| Math-PSA-7B | 6.7 | 57.5 | 79.8 | 42.5 | 41.0 | 39.3 | 44.5 |
| RLHFlow-PRM-Mistral-8B | 10.0 | 57.5 | 73.4 | 37.5 | 38.0 | 41.2 | 42.9 |
| RLHFlow-PRM-DeepSeek-8B | 13.3 | 52.5 | 74.8 | 39.5 | 37.0 | 40.8 | 43.0 |
| Llemma-PRM800K-7B | 13.3 | 57.5 | 73.8 | 40.0 | 36.5 | 38.2 | 43.2 |
| Skywork-PRM-7B | 10.0 | 57.5 | 77.8 | 41.5 | 39.0 | 43.4 | 44.9 |
| ReasonEval-7B | 3.3 | 55.0 | 73.0 | 37.5 | 35.5 | 37.9 | 40.4 |
| Qwen2.5-Math-7B-PRM800K | 23.3 | 45.0 | 78.2 | 42.0 | 35.5 | 38.6 | 43.8 |
| Qwen2.5-Math-PRM-7B | 16.7 | 60.0 | 81.0 | 43.5 | 39.0 | 40.4 | 46.8 |
| $\bigstar$ R-PRM-7B-DPO | 16.7 | 70.0 | 80.0 | 46.5 | 39.5 | 43.4 | 49.4 |
Table 3: The performance of PRM guided greedy search with policy model Qwen2.5-7B-Instruct.
R-PRM achieves high evaluation accuracy efficiently.
As shown in Table 1 and Table 2, our SFT approach achieves F1 scores of 65.2 and 64.9 on ProcessBench and PRMBench, respectively, representing significant improvements over state-of-the-art baselines. Notably, when compared with Qwen2.5-Math-7B-PRM800K (the strongest PRM800K-based baseline), R-PRM achieves F1 improvements of 6.7 and 6.6 points on ProcessBench and PRMBench, respectively. The effectiveness of our framework is further demonstrated by its 7.2-point and 3.3-point F1 improvements over LLaMA3.3-70B-Instruct and GPT-4o, respectively, on ProcessBench. These results demonstrate that our reasoning-driven evaluation paradigm successfully enhances model learning efficiency while maximizing the utility of human-annotated data.
The model’s capabilities are significantly enhanced through DPO, achieving remarkable performance scores of 70.4 and 66.8 on two benchmarks. Notably, it surpassed Qwen2.5-Math-PRM (trained with 1.8M data points) on the PRMbench evaluation. These experimental results conclusively validate the efficacy of the DPO approach, highlighting the potential of generative evaluation paradigms to substantially improve assessment capabilities.
R-PRM provides more comprehensive evaluation across multiple dimensions.
In rigorous benchmarking with PRMBench, R-PRM-DPO demonstrates significant advantages over Qwen2.5-Math-7B-PRM800K. It achieves improvements of 7.0, 9.0, and 4.4 points across the three core evaluation dimensions. Notably, it surpasses GPT-4 in both completeness and sensitivity, positioning itself as a more comprehensive assessment paradigm.
More specifically, R-PRM’s strength in soundness is reflected in three dimensions: empirical validity, step consistency, and domain alignment. This structural strength directly enhances the model’s ability to detect logical errors through progressive contextual analysis. Each reasoning step undergoes a rigorous examination of preceding steps to identify inconsistencies. Notably, R-PRM exhibits strong prerequisite sensitivity, with a particular advantage in deception resistance over o1-mini. This highlights its robustness in detecting reasoning steps that appear superficially valid but are logically flawed—errors that conventional systems often fail to identify.
| Setting | AIME24 | AMC23 | MATH | Olympiad Bench | College Math | Minerva MATH | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| pass@1 | 11.2 | 47.8 | 73.0 | 38.0 | 38.6 | 37.2 | 41.0 |
| maj@8 | 20.0 | 57.5 | 79.6 | 47.0 | 41.5 | 42.7 | 48.0 |
| pass@8 | 33.3 | 82.5 | 88.8 | 58.5 | 47.5 | 57.7 | 61.4 |
| 7B+ | | | | | | | |
| Math-Shepherd-7B | 16.7 | 42.5 | 76.0 | 42.0 | 37.0 | 39.3 | 42.3 |
| Math-PSA-7B | 20.0 | 55.0 | 80.8 | 47.5 | 39.5 | 40.1 | 47.2 |
| RLHFlow-Mistral-8B | 10.0 | 55.0 | 76.8 | 42.0 | 39.5 | 37.1 | 43.4 |
| RLHFlow-DeepSeek-8B | 13.3 | 57.5 | 76.2 | 40.0 | 39.0 | 39.7 | 44.3 |
| Llemma-PRM800K-7B | 10.0 | 52.5 | 76.6 | 42.5 | 39.0 | 42.7 | 43.9 |
| Skywork-PRM-7B | 16.7 | 55.0 | 81.2 | 44.0 | 40.5 | 44.5 | 47.0 |
| ReasonEval-7B | 6.7 | 55.0 | 75.2 | 41.0 | 40.0 | 40.4 | 43.1 |
| Qwen2.5-Math-7B-PRM800K | 13.3 | 57.5 | 80.0 | 44.5 | 43.5 | 43.0 | 47.7 |
| Qwen2.5-Math-PRM-7B | 16.7 | 55.0 | 82.0 | 48.0 | 43.5 | 43.0 | 48.0 |
| $\bigstar$ R-PRM-7B-DPO | 20.0 | 62.5 | 82.2 | 48.0 | 41.0 | 44.1 | 49.6 |
Table 4: Performance comparison on the Best-of-8 strategy of the policy model Qwen2.5-7B-Instruct. $\bigstar$ represents the models we trained.
R-PRM demonstrates superior generalization capability.
As shown in Table 1, all listed open-source PRMs, except Qwen2.5-Math-PRM-7B and Skywork-PRM-7B for which the training data sources are unknown, have been trained exclusively on GSM8K and MATH. Among these PRMs, only Math-PSA-7B and Qwen2.5-Math-7B-PRM800K achieve F1 scores above 60 on certain ProcessBench subsets, while others perform relatively poorly, particularly on out-of-domain datasets such as OmniMATH and OlympiadBench. Except for Qwen2.5-Math-7B-PRM800K, which achieved an F1 score of 50.7 on OlympiadBench, the remaining models scored below 50, with most falling between 10 and 40. By contrast, R-PRM not only performs well on the MATH dataset but also achieves F1 scores above 60 on all out-of-domain datasets. This suggests that R-PRM has learned a dataset-independent reasoning pattern, enabling it to perform well across datasets with varying difficulty.
R-PRM guides policy model to reach correct answer effectively.
As shown in Table 3 and Table 4, our method achieves 8.4% and 8.6% average accuracy improvements over the Pass@1 baseline in the Guide Search and Best-of-N settings, respectively. It further achieves state-of-the-art performance by outperforming Qwen2.5-MATH-PRM by 2.6 and 1.6 points, and surpassing Majority Voting in both settings. The experimental results directly demonstrate that our method’s accurate reward evaluation at each reasoning step effectively guides the policy model to arrive at correct solutions. Furthermore, we believe our approach holds greater potential for integration with backtracking-enabled strategies like Monte Carlo Tree Search (MCTS) and multi-candidate strategies such as Beam Search, enabling more comprehensive utilization of our methodology.
5 Analysis
5.1 Effective Data Scaling *[Error downloading image: extracted/6314339/latex/fig/DataSource.png]*
Figure 2: Average F1 score on ProcessBench with different training data scales.
We conduct comprehensive experiments to study the effect of training data scale on model performance. Figure 2 illustrates the performance variation of R-PRM on ProcessBench under varying training set sizes. With only 12.8k training samples, R-PRM achieves an impressive F1 score of 52.6, which already surpasses most existing open-source PRMs. When scaled up to 64k samples, R-PRM demonstrates superior performance, outperforming Qwen2.5-Math-7B-PRM800K, which is trained on 265k samples, by a margin of 1.6 points. Further scaling the training data to the full set of 285k samples yields substantial improvements, reaching an F1 score of 65.2.
Without requiring additional labeled data, we leverage 269K preference pairs from existing sampling results, effectively improving the model’s performance from 65.2 to 70.4 and demonstrating exceptional efficiency in data utilization.
5.2 Inference-Time-Scaling *[Error downloading image: extracted/6314339/latex/fig/processbench-scaling.png]*
Figure 3: Efficient scaling inference-time compute on ProcessBench.
We conducted an investigation into how R-PRM’s performance scales with increasing inference-time budgets. As shown in Figure 3, R-PRM demonstrates consistent performance gains on ProcessBench as the number of evaluation trajectories increases. Notably, scaling from 2 to 4 samples leads to a substantial F1 improvement from 62.8 to 67.6 on ProcessBench. Moreover, increasing the number of evaluation trajectories consistently yields performance improvements across all four datasets, which demonstrates the robustness of our scaling strategy and highlights a unique advantage of our reasoning-driven approach.
5.3 Threshold Robustness in Model Evaluation
<details>
<summary>x2.png Details</summary>
### Visual Description
## Dual Bar Charts: Model F1 Score vs. Threshold on MATH and OlympiadBench Datasets
### Overview
The image consists of two side-by-side grouped bar charts comparing the performance (measured by F1 Score) of two different models across varying threshold levels. The left chart displays results for the "MATH" dataset, while the right chart displays results for the "OlympiadBench" dataset. A single, shared legend is positioned at the top center of the image.
### Components/Axes
**Header / Legend (Top Center):**
* **Teal (Blue-Green) Square:** Labeled `Qwen2.5-Math-7B-PRM800K`
* **Orange (Coral) Square:** Labeled `R-PRM-DPO`
**Left Chart: MATH**
* **Title:** `MATH` (Centered above the chart)
* **Y-Axis:** Labeled `F1 Score` (Rotated 90 degrees, positioned on the far left). Scale ranges from 0 to 80, with major tick marks and horizontal dashed gridlines at 0, 20, 40, 60, and 80.
* **X-Axis:** Labeled `Threshold` (Centered below the axis). Major labeled tick marks are at 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, and 0.8. There are unlabelled minor tick marks halfway between each major tick, indicating increments of 0.05 (e.g., 0.25, 0.35).
* **Data Annotations:** Two specific data points are explicitly labeled with text above their respective bars:
* `76.9` (in Orange text) above the Orange bar at Threshold 0.5.
* `66.9` (in Teal text) above the Teal bar at Threshold 0.75.
**Right Chart: OlympiadBench**
* **Title:** `OlympiadBench` (Centered above the chart)
* **Y-Axis:** Shares the same visual scale and gridlines as the left chart (0 to 80 in increments of 20), though the label "F1 Score" is not repeated.
* **X-Axis:** Labeled `Threshold` (Centered below the axis). Identical scale to the left chart: 0.2 to 0.8 with 0.05 increments.
* **Data Annotations:** Two specific data points are explicitly labeled with text above their respective bars:
* `64.0` (in Orange text) above the Orange bar at Threshold 0.45.
* `58.2` (in Teal text) above the Teal bar at Threshold 0.7.
---
### Detailed Analysis
#### 1. MATH Dataset (Left Chart)
**Trend Verification:**
* **Teal (Qwen2.5-Math-7B-PRM800K):** The trend slopes upward steadily from a low starting point at 0.2, peaking late at 0.75, before slightly dropping at 0.8.
* **Orange (R-PRM-DPO):** The trend starts high, forms a gentle convex curve peaking in the middle at 0.5, and slowly declines toward 0.8. Orange consistently outperforms Teal across all thresholds.
**Extracted Data Points (F1 Score vs. Threshold):**
*(Note: Values without explicit labels are visually approximated based on gridlines)*
* **Threshold 0.20:** Teal ~38 | Orange ~69
* **Threshold 0.25:** Teal ~45 | Orange ~72
* **Threshold 0.30:** Teal ~49 | Orange ~74
* **Threshold 0.35:** Teal ~55 | Orange ~75
* **Threshold 0.40:** Teal ~58 | Orange ~76
* **Threshold 0.45:** Teal ~61 | Orange ~76
* **Threshold 0.50:** Teal ~63 | **Orange 76.9 (Explicit Peak)**
* **Threshold 0.55:** Teal ~64 | Orange ~76
* **Threshold 0.60:** Teal ~65 | Orange ~76
* **Threshold 0.65:** Teal ~66 | Orange ~75
* **Threshold 0.70:** Teal ~66 | Orange ~74
* **Threshold 0.75:** **Teal 66.9 (Explicit Peak)** | Orange ~72
* **Threshold 0.80:** Teal ~65 | Orange ~69
#### 2. OlympiadBench Dataset (Right Chart)
**Trend Verification:**
* **Teal (Qwen2.5-Math-7B-PRM800K):** The trend slopes upward in a near-linear fashion from a very low starting point at 0.2, peaking late at 0.7, and then plateauing/dropping slightly.
* **Orange (R-PRM-DPO):** The trend starts high, rises slightly to a broad plateau between 0.35 and 0.6 (peaking at 0.45), and then declines. Orange significantly outperforms Teal at lower thresholds, but the two models converge and perform identically at thresholds 0.75 and 0.80.
**Extracted Data Points (F1 Score vs. Threshold):**
*(Note: Values without explicit labels are visually approximated based on gridlines)*
* **Threshold 0.20:** Teal ~18 | Orange ~59
* **Threshold 0.25:** Teal ~23 | Orange ~62
* **Threshold 0.30:** Teal ~30 | Orange ~63
* **Threshold 0.35:** Teal ~36 | Orange ~63
* **Threshold 0.40:** Teal ~40 | Orange ~64
* **Threshold 0.45:** Teal ~45 | **Orange 64.0 (Explicit Peak)**
* **Threshold 0.50:** Teal ~50 | Orange ~64
* **Threshold 0.55:** Teal ~53 | Orange ~64
* **Threshold 0.60:** Teal ~55 | Orange ~63
* **Threshold 0.65:** Teal ~56 | Orange ~62
* **Threshold 0.70:** **Teal 58.2 (Explicit Peak)** | Orange ~60
* **Threshold 0.75:** Teal ~58 | Orange ~58 (Convergence)
* **Threshold 0.80:** Teal ~54 | Orange ~54 (Convergence)
---
### Key Observations
1. **Absolute Performance:** Both models achieve higher maximum F1 scores on the MATH dataset compared to the OlympiadBench dataset.
2. **Model Superiority:** The `R-PRM-DPO` model (Orange) demonstrates superior performance over the `Qwen2.5-Math-7B-PRM800K` model (Teal) across almost all tested thresholds on both datasets.
3. **Peak Threshold Discrepancy:**
* `R-PRM-DPO` peaks at lower, mid-range thresholds (0.5 on MATH, 0.45 on OlympiadBench).
* `Qwen2.5-Math-7B-PRM800K` requires much higher thresholds to reach its peak performance (0.75 on MATH, 0.7 on OlympiadBench).
4. **Low-Threshold Robustness:** `R-PRM-DPO` maintains high F1 scores even at the lowest threshold (0.2), whereas the Qwen model's performance collapses at low thresholds (dropping to ~18 on OlympiadBench).
5. **Convergence:** On the OlympiadBench dataset, the performance gap between the two models completely closes at the highest thresholds (0.75 and 0.80), where both models perform equally, albeit sub-optimally compared to their respective peaks.
### Interpretation
The data strongly suggests that the `R-PRM-DPO` training methodology yields a significantly more robust and capable model than the baseline `Qwen2.5-Math-7B-PRM800K`, particularly in its confidence calibration.
Because `R-PRM-DPO` achieves high F1 scores at lower thresholds, it indicates that the model is highly accurate even when its internal confidence (threshold) is relatively low. Conversely, the Qwen model requires a very high threshold to filter out incorrect answers and achieve its peak F1 score, meaning it is likely prone to high-confidence errors (hallucinations) at lower thresholds.
Furthermore, the overall lower scores on the `OlympiadBench` chart compared to the `MATH` chart indicate that OlympiadBench is a fundamentally more difficult dataset for both models. The convergence of the two models at high thresholds on OlympiadBench suggests that when forced to be extremely conservative (threshold $\ge$ 0.75), both models are reduced to answering only the easiest questions, negating the advantages of the `R-PRM-DPO` training.
</details>
Figure 4: Average F1 score on ProcessBench at different thresholds, with the overall performance on the left and performance on the OlympiadBench dataset on the right.
| Col1 |
| --- |
| Problem: An emirp is an integer that, when its digits are written in reverse order, is a prime number. For example, 73 is an emirp because 37 is prime. What is the smallest two-digit prime number that is NOT an emirp? |
| Solution: … Step7: Continue to the next smallest two-digit prime number. 19 is prime. Step 8: Check if 23 is an emirp. Reverse the digits of 23: $32$ . Check if 32 is prime: No, 32 is not prime (divisible by 2). Since 23 is a prime number and its reverse, 32, is not a prime number, 23 is not an emirp. … |
| Qwen2.5_Math-PRM-7B correctness score: 0.9883 |
| Qwen2.5_Math-7B-PRM800K correctness score: 0.8594 |
| R-PRM: Analysis: Previous Steps Analysis: … …… Calculation Analysis: …… Upon reviewing, the mistake in the analysis is assuming 23 is the answer without properly checking if there are smaller primes that are not emirps. …… Given the oversight in the analysis regarding the thorough checking of primes and their reversals, the conclusion that 23 is the smallest such prime without properly validating against all primes in sequence is premature. …… Conclusion: The Now Step incorrectly concludes that 23 is the smallest two-digit prime number …… Verification: Is the step correct (Yes/No)? No |
| R-PRM (Majority Voting) score: 0.0547 |
Figure 5: A case study from ProcessBench MATH dataset. Red text denotes the error step and the scores of other models, and the blue text highlights our model’s critique of the error and our score for that step.
During evaluations of ProcessBench and PRMBench, we adopt a fixed threshold of 0.5 for binary classification to determine whether each step is correct. We further analyze the model’s sensitivity to threshold variations. As shown in Figure 4, R-PRM demonstrates strong robustness to threshold variation, with minimal performance fluctuations on ProcessBench. In contrast, Qwen2.5-Math-7B-PRM800K exhibits greater sensitivity, showing a noticeable rightward shift in its performance curve on ProcessBench.
On the more challenging out-of-domain test subset OlympiadBench, our method maintains its robustness to threshold variations, while Qwen2.5-Math-7B-PRM800K exhibits a larger threshold shift, indicating its tendency to misclassify incorrect solutions as correct. This demonstrates that our method has better generalization capability, maintaining more accurate evaluation even when problem types and domains change. For detailed performance analysis of scores and thresholds on PRMBench, please refer to the Appendix B.
5.4 Case Study
As shown in Figure 5, the solution erroneously skipped verifying the number 19 in Step 7 and directly proceeded to check number 23 in Step 8. Unfortunately, both strong baselines, Qwen2.5-Math-PRM and Qwen2.5-Math-7B-PRM800K, failed to detect the omission, mistakenly assigning high reward scores to Step 8 (0.99 and 0.86, respectively). In contrast, R-PRM first analyzed both the previous and current steps. Based on this analysis, R-PRM concluded that the task required verifying the numbers in ascending order, which showcases its advanced logical reasoning capabilities. Subsequently, R-PRM resumed the reasoning process for Step 7 to verify the correctness of number 19, thus identifying the discrepancy between its own result and the answer in Step 8. Through this reasoning process, R-PRM assigned a reward score of 0.05 to Step 8, successfully detecting the error. Please refer to the Appendix C for more cases.
6 Conclusion
In this paper, we present Reasoning-Driven Process Reward Modeling (R-PRM), a novel framework that advances the process reward modeling of mathematical reasoning. Our framework consists of three components. First, we leverage stronger LLMs to construct seed data, enabling our model to perform a comprehensive evaluation process. Second, we use preference optimization to enhance performance without requiring additional annotated data. Third, we introduce inference-time scaling to fully harness the model’s reasoning capabilities. Extensive experiments demonstrate that our method achieves significant performance improvements on ProcessBench and PRMBench, while also effectively guiding LLM reasoning. Further analysis shows that R-PRM exhibits more comprehensive, robust, and generalizable evaluation capabilities, as its performance continues to improve with increased inference, highlighting its substantial potential.
Limitations
Due to computational resource constraints, we have not yet verified our approach on larger models such as 70B, despite extensive experiments demonstrating its effectiveness on 7B models. We hypothesize that larger models, given their enhanced reasoning capabilities, could achieve higher modeling accuracy when combined with our methodology. Additionally, while we have tested popular inference strategies like Best-of-N and Guided Search, our exploration of advanced search algorithms remains limited. Sophisticated methods such as Monte Carlo Tree Search (MCTS) and Beam Search remain underexplored, although they could potentially better leverage the characteristics of PRM and yield improved generation results.
References
- Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
- DeepSeek-AI (2025) DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
- Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. Preprint, arXiv:2405.07863.
- He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
- Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.
- Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
- Luo et al. (2025) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. Preprint, arXiv:2308.09583.
- Luo et al. (2024a) Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024a. Improve mathematical reasoning in language models by automated process supervision. Preprint, arXiv:2406.06592.
- Luo et al. (2024b) Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. 2024b. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592.
- O1 (2023) OpenAI O1. 2023. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. Accessed: 2025-02-08.
- o1 Team (2024) Skywork o1 Team. 2024. Skywork-o1 open series. https://huggingface.co/Skywork.
- QwQ (2023) QwQ. 2023. Qwq: Reflect deeply on the boundaries of the unknown. https://qwenlm.github.io/blog/qwq-32b-preview//. Accessed: 2025-02-08.
- Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.
- Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
- Snell et al. (2024) Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
- Song et al. (2025) Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. Preprint, arXiv:2501.03124.
- Sun et al. (2024) Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. 2024. Easy-to-hard generalization: Scalable alignment beyond human supervision. Preprint, arXiv:2403.09472.
- Tang et al. (2024) Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884.
- Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. Preprint, arXiv:2211.14275.
- Wang et al. (2024a) Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M. Ni, Linyi Yang, Ying Wen, and Weinan Zhang. 2024a. Openr: An open source framework for advanced reasoning with large language models. Preprint, arXiv:2410.09671.
- Wang et al. (2024b) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024b. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439.
- Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. Preprint, arXiv:2212.10560.
- Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
- Xia et al. (2025) Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2025. Evaluating mathematical reasoning beyond accuracy. Preprint, arXiv:2404.05692.
- Yang et al. (2024) An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. Preprint, arXiv:2409.12122.
- Zhang et al. (2025) Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. Preprint, arXiv:2501.07301.
- Zheng et al. (2024) Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. Preprint, arXiv:2412.06559.
Appendix A Detailed Description of PRMBench Subcategories
- Non-Redundancy (NR): Evaluates the model’s ability to identify and eliminate unnecessary steps within the reasoning process, ensuring efficiency without sacrificing correctness.
- Non-Circular Logic (NCL): Assesses whether the model can detect circular reasoning, where conclusions are reintroduced as premises, leading to logical loops.
- Empirical Soundness (ES): Measures the model’s capability to identify and reject reasoning steps that contradict established facts or real-world knowledge.
- Step Consistency (SC): Evaluates whether the reasoning steps maintain consistency with each other, ensuring that all steps logically flow from one to the next.
- Domain Consistency (DC): Assesses the model’s ability to apply domain-specific knowledge correctly, avoiding the misuse of concepts or theories across different domains.
- Confidence Invariance (CI): Tests whether the model maintains appropriate confidence levels throughout the reasoning process, especially when errors are detected or uncertainties arise.
- Prerequisite Sensitivity (PS): Evaluates whether the model detects missing prerequisites or conditions essential for valid reasoning, ensuring the completeness of the logic.
- Deception Resistance (DR): Measures the model’s ability to detect and reject misleading information that might appear correct but contains subtle errors.
- Multi-Solution Consistency (MS): Assesses the model’s ability to handle multiple valid solutions to the same problem, ensuring consistency across different reasoning paths.
Appendix B Threshold Robustness on PRMBench
As illustrated in Figure 6, the experimental results of PRMBench show that R-PRM has significant robustness advantages, while Qwen2.5-Math-7B-PRM800K exhibits a performance gap of 8.2 points between the 0.5 threshold and its optimal performance. *[Error downloading image: extracted/6314339/latex/fig/prmbench-threshold.png]*
Figure 6: PRMBench Overall Score with Different Threshold
Appendix C Additional Case
As shown in this case, both Qwen2.5-Math-PRM-7B and Qwen2.5-Math-7B-PRM800K fail to detect the calculation error in this step, yet they show high confidence in their judgments with their correctness scores around 0.9. On the other hand, our R-PRM shows its strong ability by explaining the step, checking the calculation, and finding the error, with a final score of around 0.3.
| Col1 |
| --- |
| Problem: Suppose $t$ is a positive integer such that $\mathop{\text{lcm}}[12,t]^{3}=(12t)^{2}$ . What is the smallest possible value for $t$ ? |
| Solution: To solve this problem, we can start by using the properties of the least common multiple (LCM). Given: $\mathop{\text{lcm}}[12,t]^{3}=(12t)^{2}$ . Step 1: Take the cube root of both sides of the equation. $\sqrt[3]{\mathop{\text{lcm}}[12,t]^{3}}=\sqrt[3]{(12t)^{2}}$ . This simplifies to: $\mathop{\text{lcm}}[12,t]=12t^{2/3}$ . …Step 31: We can see that $13824=2304^{2}$ , so $t=48$ satisfies the original equation. Therefore, the smallest possible value for $t$ is $\boxed{48}$ . |
| Qwen2.5-Math-PRM-7B score: 0.9414 |
| Qwen2.5-Math-7B-PRM800K score: 0.8711 |
| R-PRM (Ours): Analysis: Previous Steps Analysis: Step 1: This step states the problem and sets up the equation to be solved, which is $\mathop{\text{lcm}}[12,t]^{3}=(12t)^{2}$ .This step is a correct restatement of the problem …… … Calculation Analysis: The calculation involves taking the cube root of $(12t)^{2}$ . By the properties of exponents, $\sqrt[3]{(12t)^{2}}=\sqrt[3]{12^{2}· t^{2}}=12^{2/3}· t^{2/3}$ . This is because the cube root of a product can be expressed as the product of the cube roots, and the cube root of a power can be expressed as that power divided by 3. Therefore, the correct simplification should be $\mathop{\text{lcm}}[12,t]=12^{2/3}· t^{2/3}$ , not $12t^{2/3}$ as stated in the Now Step. Conclusion: The Now Step incorrectly simplifies the expression $\sqrt[3]{(12t)^{2}}$ to $12t^{2/3}$ . The correct simplification should be $12^{2/3}· t^{2/3}$ , which is different from what is presented. Therefore, the Now Step contains an error. Verification: Is the step correct (Yes/No)? No" |
| R-PRM score: 0.2992 |
Figure 7: A case study from MATH subset of ProcessBench. Red text denotes the error step and the scores of other models, and the blue text highlights our model’s critique of the error and our score for that step.
Appendix D Used Scientific Artifacts
Below are the scientific artifacts used in our work. For the sake of ethics, our use of these artifacts is consistent with their intended use.
- Transformers (Apache-2.0 license), a framework to facilitate downloading and training state-of-the-art pretrained models.
- trl (Apache-2.0 license), a full stack library that provides a set of tools to train transformer language models with Reinforcement Learning. The library is built on top of the Transformers library.
- PRM800K(MIT license), a large-scale dataset proposed by OpenAI, containing 800,000 step-level correctness labels for model-generated solutions to problems from the MATH dataset.
- PRMBench(Apache-2.0 license), a benchmark designed to evaluate the ability of PRMs to identify error-types in the solution process, consisting of 6,216 fine-grained data instances.
Appendix E Prompt for Construction Data
{xltabular}
>X You are an excellent math teacher. Please verify the correctness of the Now Step.
You first need to analyze the Now Step and the Previous Steps and then summarize based on your analysis. Analysis: You need to analyze the following aspects. **Previous Steps Analysis**: You need to analyze the Previous Steps step by step. For each step, you need to first explain what the current step is doing, then you try to find any error in the current step. **Now Step Analysis**: You first need to explain what the Now Step is doing, and then point out which part of the Question it is trying to solve or which part of the information it states. **Data Source Analysis**: First you need to find out what data are used in the Now Step, and then you need to determine whether the source of the data is reasonable and correct. When you judge whether the source of a data is reasonable and correct, you need to specify the specific source of this data: such as which part of the question, or which content of the previous step; and then determine the source and current use is consistent, the Now Step is used correctly. **Consistency Analysis**: You need to check that the Now Step is consistent with the contents of the Previous Steps, and then you need to check that all the information inside the Now Step is consistent. **Calculation Analysis**: If the Now Step involves any calculations, such as addition, subtraction, multiplication, division, equations, modulo operations, etc., you will first need to perform a check on the calculation, such as a reverse operation, to see if the calculation was done correctly, and then analyze the results of your check to see if there was an error in the calculation. Conclusion: Please verify the correctness of the Now Step based on your analysis, if there is any error in the Now Step then the Now Step is wrong and vice versa the Now Step is correct. At the end of the Conclusion, when you give your final answer, write it in the form "Verification: Is the step correct (Yes/No)? X", where X is either Yes or No.
Question: [Math Problem] Previous Steps: [Previous Steps] Now Step: [Current Step] Please carefully analyze the correctness of the Now Step. Reply:
Table 5: The Prompt to Construct Data