2506.09532

# Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models > Work done during the internship at AMD.Corresponding authors. Abstract We present Athena-PRM, a multimodal process reward model (PRM) designed to evaluate the reward score for each step in solving complex reasoning problems. Developing high-performance PRMs typically demands significant time and financial investment, primarily due to the necessity for step-level annotations of reasoning steps. Conventional automated labeling methods, such as Monte Carlo estimation, often produce noisy labels and incur substantial computational costs. To efficiently generate high-quality process-labeled data, we propose leveraging prediction consistency between weak and strong completers as a criterion for identifying reliable process labels. Remarkably, Athena-PRM demonstrates outstanding effectiveness across various scenarios and benchmarks with just 5,000 samples. Furthermore, we also develop two effective strategies to improve the performance of PRMs: ORM initialization and up-sampling for negative data. We validate our approach in three specific scenarios: verification for test time scaling, direct evaluation of reasoning step correctness, and reward ranked fine-tuning. Our Athena-PRM consistently achieves superior performance across multiple benchmarks and scenarios. Notably, when using Qwen2.5-VL-7B as the policy model, Athena-PRM enhances performance by 10.2 points on WeMath and 7.1 points on MathVista for test time scaling. Furthermore, Athena-PRM sets the state-of-the-art (SoTA) results in VisualProcessBench and outperforms the previous SoTA by 3.9 F1-score, showcasing its robust capability to accurately assess the correctness of the reasoning step. Additionally, utilizing Athena-PRM as the reward model, we develop Athena-7B with reward ranked fine-tuning and outperforms baseline with a significant margin on five benchmarks. 1 Introduction In recent years, Large Language Models (LLMs) [5, 1, 52, 30, 61] have achieved remarkable success in natural language processing. Building on this, Multimodal Large Language Models (MLLMs) [31, 26, 3, 8] have made significant strides in various vision-language tasks, such as visual question answering [2, 39] and chart understanding [38]. Despite their impressive performance, solving complex tasks involving mathematical and multi-step reasoning remains challenging. To enhance reasoning capabilities, several approaches have been explored, including fine-tuning on long chain-of-thought (CoT) data [59, 40], offline preference optimization [56, 43], and online reinforcement learning [47, 17, 21, 51]. Another promising avenue is test time scaling (TTS), which involves generating multiple responses from a policy model and selecting the most consistent answer [58] or the solution with the highest reward using reward models [9, 28, 53, 50, 46, 55]. Reward models for TTS mainly include two types: outcome reward models (ORMs) [9] and process reward models (PRMs) [28]. ORMs evaluate the reward score for a given question and its solution, whereas PRMs provide reward scores for each intermediate reasoning step, offering fine-grained feedback. PRMs typically deliver superior performance and better out-of-distribution generalization [28]. However, obtaining high-quality data with process labels poses significant challenges. PRM800K [28] involves collecting 800K labeled steps through human annotations, which is time-consuming and requires skilled annotators, particularly for complex multi-step reasoning and mathematical tasks. Math-Shepherd [55] proposes an automated labeling method using Monte Carlo (MC) estimation. It defines the quality of an intermediate step as its potential to reach the correct answer. MC estimation for step-level labels is typically performed by sampling numerous reasoning trajectories using a language model, referred to as the completer, to estimate the probability of arriving at the correct answer. However, this approach involves significant computational overhead. Besides, another shortcoming of MC-based estimation methods is that estimated step-level labels are inevitably noisy. We aim to address the above challenges: reducing computational costs and mitigating noisy step labels. We first find that the accuracy of step labels estimated by MC-based methods is influenced by the reasoning capability of the completer. A strong completer can arrive at the correct answer despite incorrect intermediate steps as shown in Figure 1, whereas a weak completer may struggle even with correct intermediate steps. The estimation of MC-based methods is biased toward the completers we use. However, the correctness of the intermediate step should not depend on the completer. Based on this insight, we propose using both weak and strong completers to estimate step labels, retaining only those steps where labels generated by both completers are consistent to remove the bias caused by completers. This approach improves the quality of the step label. Empirical results demonstrate that a small set of high-quality labels ( $≈$ 5K) achieves impressive performance compared to large-scale data ( $≈$ 300K) labeled by vanilla MC-based estimation [55]. In addition, our approach requires only 1/45 of the GPU hours for data synthesis and 1/60 of the GPU hours for reward model training, significantly lowering computational costs. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Symbolic Feedback with Textual Annotations ### Overview The image features a minimalist design with a white background. Central to the composition are two hand-drawn symbols: a green checkmark (✓) and a red X (✗), positioned side by side. To the right of these symbols are three vertically stacked colored blocks containing fragmented text. The text appears to reference mathematical equations and logical conditions, though portions are cut off or incomplete. ### Components/Axes - **Central Symbols**: - **Green Checkmark (✓)**: Positioned left of center, slightly above the horizontal midpoint. - **Red X (✗)**: Positioned right of center, aligned horizontally with the checkmark. - **Text Blocks (Right Side)**: 1. **Top Block (Yellow)**: Contains red text: - `"y² = 133. If x > x - y, we 13 × 10 , so"` - Fragmented and incomplete; trailing text cut off. 2. **Middle Block (Blue)**: Contains black text: - `"two perfect"` 3. **Bottom Block (Orange)**: Contains black text: - `"equations there are"` ### Detailed Analysis - **Symbols**: - The green checkmark and red X are classic indicators of correctness/incorrectness, suggesting a binary evaluation (e.g., right/wrong answers). - No explicit legend or labels clarify their meaning, but their placement implies they are the primary focus. - **Text Blocks**: - **Yellow Block**: The text begins with a mathematical equation (`y² = 133`) and a conditional statement (`If x > x - y`), followed by an incomplete arithmetic operation (`13 × 10 , so`). The fragment suggests a step in solving an equation, but the conclusion is missing. - **Blue Block**: `"two perfect"` likely refers to "perfect squares" or "perfect numbers," though the phrasing is truncated. - **Orange Block**: `"equations there are"` is grammatically incomplete, possibly part of a larger phrase like "there are equations..." or "equations there are..." ### Key Observations 1. **Fragmented Text**: All text blocks are cut off, leaving ambiguity about their full context. For example: - The yellow block’s equation (`y² = 133`) could relate to solving for `y`, but the incomplete arithmetic (`13 × 10`) lacks a result. - The blue and orange blocks’ phrases are standalone fragments, requiring external context to interpret. 2. **Color Coding**: - Red text in the yellow block may emphasize critical steps or errors. - Black text in blue/orange blocks suggests neutral or explanatory content. 3. **Spatial Layout**: - Symbols are centered, drawing immediate attention. - Text blocks are aligned vertically on the right, creating a hierarchical structure. ### Interpretation The image likely represents a step-by-step problem-solving process, with the checkmark/X indicating feedback on correctness. The fragmented text suggests: - A mathematical problem involving `y² = 133` and a conditional inequality (`x > x - y`). - A potential error or incomplete reasoning in the arithmetic step (`13 × 10`). - References to "perfect" equations or numbers, possibly hinting at constraints or properties (e.g., perfect squares). The design implies a pedagogical or debugging context, where symbols and text guide the user toward identifying mistakes or validating solutions. However, the incomplete text limits definitive interpretation. </details> Figure 1: Illustration of different completers under the same question and solution steps. Even if given wrong intermediate steps, the strong completer stills reach the final answer while the weak completer fails. We omitted some intermediate steps in the figure for simplicity. After acquiring high-quality datasets, we explore two effective strategies for training PRMs: initialization from ORMs and up-sampling data with negative labels. PRMs are typically fine-tuned from pre-trained foundation models, such as LLaMA-3.1-8B-Instruct [16] in RLHF-workflow [13]. Previous studies indicate that ORMs possess some ability to assess intermediate step correctness through confidence variation [33] or parameterization [10, 63]. Inspired by this, we find that initializing PRMs from ORMs significantly boosts performance, as ORMs trained on large-scale response-level data serve as pre-training with weak supervision. And PRMs can be considered fine-tuning on high-quality, fine-grained step-level data. Additionally, we show that label imbalance exists in process-labeled solutions and we propose to up-sample data with negative step labels to address this. Building on these methodologies, we develop our outcome reward model Athena-ORM and process reward model Athena-PRM. Leveraging Athena-PRM, we introduce Athena-7B with reward ranked fine-tuning. We validate our approach across three scenarios: 1) test time scaling [50]: where Athena-PRM ranks multiple outputs generated by policies under the Best-of-N evaluation; 2) direct evaluation of the correctness of reasoning steps using Athena-PRM; 3) reward ranked fine-tuning [12]: where Athena-PRM ranks outputs sampled from the current policy, utilizing the highest reward response for policy model fine-tuning. In the TTS scenario, we evaluate Athena-PRM on seven multimodal math and reasoning benchmarks with three different policy models ranging from 7B to 72B, demonstrating significant improvements in reasoning abilities. For instance, on the WeMath benchmark [44], Athena-PRM enhances the zero-shot baseline by 10.2 points using Qwen2.5-VL-7B [3] as the policy model. Furthermore, Athena-PRM excels in text-only benchmarks, achieving an 8.9 points improvement on a challenging math benchmark [18] with Mistral-8B. To assess its ability to judge intermediate reasoning step correctness, Athena-PRM is evaluated on the VisualProcessBench [57], showcasing strong performance and outperforming VisualPRM-8B [57], an open-source multimodal PRM, and proprietary models as judge. In the reward ranked fine-tuning scenario, our fine-tuned model Athena-7B, based on Qwen2.5-VL-7B [3], significantly enhances the policy model’s reasoning capabilities across seven math and reasoning benchmarks. Our contributions are summarized as follows: - We propose using prediction consistency between weak and strong completers to filter noisy process labels, enhancing the quality of the automated-labeled process data. The high-quality process labeled data shows surprising data efficiency for training PRMs. - We introduce two effective strategies to improve the performance of PRMs: ORM initialization and negative data up-sampling. - We develop our reward models Athena-ORM and Athena-PRM, and evaluate them under the Best-of-N setting on nine math and reasoning benchmarks across various model sizes and families, demonstrating the effectiveness of our approach. Moreover, Athena-PRM achieves state-of-the-art (SoTA) performance in assessing the correctness of intermediate steps directly on the VisualProcessBench [57]. - Leveraging Athena-PRM, we introduce Athena-7B, a MLLM with exceptional capabilities fine-tuned using reward ranked fine-tuning. Extensive experiments highlight the superiority of Athena-7B across diverse benchmarks. 2 Method In this section, we first introduce basic concepts, including ORMs and PRMs in Sec. 2.1. Next, we present our method for constructing a high-quality PRM training set, along with practical training strategies to improve PRM performance in Sec. 2.2. The data collection process for training is detailed in Sec. 2.3. Finally, we discuss three application scenarios of PRMs in Sec. 2.4. 2.1 Reward Models for Mathematical Problem ORMs. Given a mathematical problem $x$ with golden answer $y^{*}$ and its solution $a$ generated by policy $\pi$ , i.e., $a\sim\pi\left(·\mid x\right)$ , ORMs assign a reward $r\left(x,y,y^{*}\right)$ to the final answer $y$ of $a$ to reflect the correctness of the solution. To train ORMs, we take inputs as $(x,y)$ and predict correctness $\delta(y,y^{*})$ of prediction with following loss function: $$ \mathcal{L}_{ORM}=\delta\left(y,y^{*}\right)\cdot\log r\left(x,y,y^{*}\right)+% \left(1-\delta\left(y,y^{*}\right)\right)\cdot\log\left(1-r\left(x,y,y^{*}% \right)\right), \tag{1} $$ where $\delta\left(y,y^{*}\right)=1$ if $y=y^{*}$ , otherwise $\delta\left(y,y^{*}\right)=0$ . PRMs. Different from ORMs, PRMs aim to predict the correctness of each step in the solution. Specifically, we sample response $a\sim\pi\left(·\mid x\right)$ . A response $a$ usually consists of multiple reasoning steps separated by a delimiter (e.g. \n\n), a.k.a $a=(a_{1},a_{2},...,a_{K})$ when $K$ is the number of reasoning steps. To predict the correctness of each step, we train PRMs with the cross-entropy loss for each step in the solution. Formally, we get the following loss function: $$ \mathcal{L}_{PRM}=\sum_{i=1}^{K}\delta_{i}\cdot\log r\left(x,a_{i}\right)+% \left(1-\delta_{i}\right)\cdot\log\left(1-r\left(x,a_{i}\right)\right), \tag{2} $$ where $\delta_{i}=1$ if the step $a_{i}$ is correct, otherwise $\delta_{i}=0$ . 2.2 Automated Data Scaling of PRMs Compared with ORMs, PRMs usually achieve better performance and exhibit stronger out-of-distribution generalization [28]. However, collecting high-quality data with process labels is challenging. Human annotation provides accurate supervision signals [28], but it requires expensive human labeling and difficult to scale up. Math-Shepherd [55] proposes to use a Monte Carlo (MC) sampling method to estimate the correctness of each step without human supervision. Specifically, to estimate the correctness of step $a_{i}$ , we use a completer $\phi$ to finalize the reasoning process from step $a_{i}$ , and get the final answer $y$ . We repeat sampling $T$ times and get corresponding answers $\mathcal{Y}=\{y_{j}\}_{j=1}^{T}$ . There are two ways to estimate the correctness of $a_{i}$ : soft estimation and hard estimation. For soft estimation, we assume that the frequency of getting the correct answer could dictate the quality of a step: $$ \delta_{i}^{soft}=\frac{\sum_{j=1}^{T}\mathbb{I}(y_{j}=y^{*})}{T}. \tag{3} $$ For hard estimation, we assume that the correct answer is right when the step could reach the answer in $T$ samples: $$ \delta_{i}^{hard}=\begin{cases}1&\text{if }\exists y_{j}\in\mathcal{Y},y_{j}=y% ^{*},\\ 0&\text{otherwise.}\end{cases} \tag{4} $$ In this paper, we mainly discuss the hard estimation for PRMs because it allows us to use standard language modeling loss to train PRMs and eliminates the adjustments of the training pipeline. MC-based estimation methods provide an automated and scalable labeling strategy for intermediate reasoning steps. However, automatically labeled data often contain incorrect labels, and MC-based methods typically incur substantial computational costs due to extensive sampling requirements. For instance, when processing a solution $a$ with $K$ steps, the completer $\phi$ must generate $T× K$ solutions, which produces $T× K$ times computation cost compared with ORMs. To enhance the accuracy of the labeling process and reduce the computational burden of data synthesis, our central approach is to utilize a smaller set of high-quality data with process labels. We improve the quality of automatically labeled process labels by introducing consistency filtering between weak and strong completers. The rationale is straightforward: the results of hard estimation depend on the number of samples $T$ , the difficulty of the problem $x$ , and the ability of the completer $\phi$ . A strong completer can arrive at the correct answer even when provided with incorrect steps, as shown in Figure 1, whereas a weak completer struggles to find the final answer even when given correct steps. We aim to enhance the quality of process labels through prediction consistency between weak and strong completers. Specifically, we only use steps whose assigned labels are consistent between different completers. To validate our method, we conducted a simple experiment using the PRM800K dataset [28]. We employed Mistral-7B-Instruct [22] as the weak completer ( $\phi_{w}$ ) and Qwen2.5-72B-Instruct [61] as the strong completer ( $\phi_{s}$ ), setting the number of samples $T=8$ for MC-based estimation. We randomly sampled 50 queries and reported the accuracy of estimated process labels: | | Weak Completer $\phi_{w}$ | Strong Completer $\phi_{s}$ | Consistency Filter | | --- | --- | --- | --- | | Accuracy (%) | 78.2 | 83.4 | 94.1 | Our experiments show that label quality improves significantly after applying the consistency filter. Empirically, we demonstrate that a small number of high-quality labels is sufficient to achieve superior performance, thereby reducing computational costs compared to the vanilla MC-based estimation [55]. Results of more strategies and combination of completers are provided in Appendix A.1. Additional strategies for PRMs. We offer two strategies for enhancing the performance of PRMs: initialization from ORMs and negative data up-sampling. First, initializing PRMs with ORMs trained on large-scale sample-level annotated data improves performance. It is shown that initialization from ORMs will improve performance with different datasets, see verification in Table 5. Previous work indicates that ORMs possess a certain capacity to assess the correctness of intermediate steps [10, 33, 63]. We empirically show that PRMs benefit from a few high-quality examples when initialized with ORMs. Large-scale sample-level annotation data provides weaker supervision, with outcome supervision acting as a simplified form of process supervision. Training ORMs on large-scale sample-level annotations serves as a “pre-training”, while training PRMs from pre-trained ORMs acts as“fine-tuning” with high-quality process-level annotated data. Secondly, our findings reveal that label imbalance is prevalent in most process-labeled datasets, such as PRM800K [28] and our synthetic data, where correct steps are more common than errors. We provide a detailed distribution of process labels in Table 9. Empirically, slightly up-sampling data with negative labels improves performance with minimal additional computation. 2.3 Data Collection To construct a diverse and high-quality dataset, we first gather data from various public multimodal and text-only datasets. For multimodal datasets, we collect from MathV360k [48], UniGeo [7], Geometry3k [34], CLEVR-Math [29], ScienceQA [35], GeomVerse [24], GeoQA-plus [6], and DocVQA [39]. For text-only datasets, we utilize GSM8K [9], MATH [18], and NuminaMath-1.5. https://huggingface.co/datasets/AI-MO/NuminaMath-1.5 If the original dataset includes different splits, we use only the training portion to prevent test set contamination. We remove all judgment questions and convert multiple-choice questions to open-ended ones to deter policy models from “guessing” answers. Then, for each query, we sample 8 responses from the policy model. To ensure dataset diversity, we employ multiple models as policies, including InternVL2.5-8B/78B [8], Qwen2.5-VL-7/72B-Instruct [3], LLaVA-OneVision-7B/72B [26] for multimodal datasets, and Qwen2.5-7/72B-Instruct [61] for text-only datasets. Next, we apply carefully designed filtering rules to all responses. We exclude responses with too many or too few tokens and eliminate those exhibiting repetitive patterns. We employ $n$ -gram deduplication to remove similar responses and enhance diversity. Ultimately, we obtain approximately 600K queries with corresponding responses. We parse answers from responses and assign correctness labels of 0 or 1 to each query-response pair. We refer to this dataset as Athena-600K and employ it to train our outcome reward model Athena-ORM. For training PRMs, we generate process labels using the proposed method introduced in Sec. 2.2. We use Qwen2.5-VL-7B [3] as the weak completer ( $\phi_{w}$ ) and Qwen2.5-VL-72B [3] as the strong completer ( $\phi_{s}$ ). Finally, we collect approximately 5K samples for training our process reward model Athena-PRM. We also compare with PRMs trained using approximately 300K samples estimated by the vanilla MC-based method outlined in [55]. We denote these datasets as Athena-5K and MC-300K, respectively, and compare their performance and computational cost in Sec. 3.4. 2.4 Application Following reward model training, we assess the performance of ORMs and PRMs in three scenarios: test time scaling [50], direct judgment, and reward ranked fine-tuning [12]. Verification for test time scaling. We adopt a Best-of-N evaluation paradigm. Specifically, given a problem $x$ in the test set, we sample $N$ solutions from the policy $\pi$ . All solutions are scored using a reward model, and we choose the solution with the highest score. For ORMs, we directly use the outputs from ORMs as the reward of solutions. For PRMs, the minimum reward across all steps is used to rank all solutions. We test different choices, including the reward from the last step, the minimum reward across all steps, or the product of the rewards from all steps, as the reward for solutions in Appendix A.3. Direct judgment for reasoning steps. Besides verification at test time under the Best-of-N setting, Athena-PRM can also be used to directly identify erroneous steps in the mathematical reasoning process. Given a solution $a$ with $K$ steps as input, Athena-PRM outputs the correctness score $\{\delta_{i}\}_{i=1}^{K}$ for each step $\{a_{i}\}_{i=1}^{K}$ in a single forward pass. Response ranking for reward ranked fine-tuning. We explore the use of PRMs for data synthesis in reward ranked fine-tuning [12, 64, 49, 66]. Using the current policy $\pi$ , we generate $M=8$ solutions for an input problem $x$ . Subsequently, we filter out solutions with incorrect answers and apply deduplication to remove highly similar responses, enhancing the diversity of the remaining solutions. We retain queries where 2 to 6 out of 8 responses are correct, excluding those with too few or too many correct answers, as such cases are either too easy or too challenging for the current policy $\pi$ . Following this filtering step, we use Athena-PRM to score all solutions for each query and select the solution with the highest reward as the corresponding label. The policy $\pi$ is fine-tuned using the synthetic data generated as described above. 3 Experiments Table 1: Results on seven multimodal reasoning benchmarks under Best-of-N (N=8) evaluation. $\dagger$ denotes that the results are from [57]. We mark our results and highlight the best result. | | WeMath | MathVista | MathVision | MathVerse | DynaMath | MMMU | LogicVista | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen2.5-VL-7B [3] | 36.2 | 68.1 | 25.4 | 41.1 | 21.8 | 58.0 | 47.9 | | Self-consistency [58] | 44.7 | 71.6 | 28.6 | 43.7 | 22.9 | 60.1 | 49.5 | | $\text{VisualPRM-8B}^{\dagger}$ [57] | 39.8 | 70.3 | 31.3 | 44.3 | 23.0 | 58.6 | 48.3 | | Athena-ORM | 45.1 | 72.8 | 29.8 | 44.1 | 23.1 | 62.7 | 51.3 | | Athena-PRM | 46.4 | 75.2 | 32.5 | 46.3 | 23.4 | 63.8 | 53.0 | | InternVL2.5-8B [8] | 23.5 | 64.5 | 17.0 | 22.8 | 9.4 | 56.2 | 36.0 | | Self-consistency [58] | 28.4 | 66.1 | 21.1 | 24.7 | 13.8 | 57.8 | 40.2 | | $\text{VisualPRM-8B}^{\dagger}$ [57] | 36.5 | 68.5 | 25.7 | 35.8 | 18.0 | 60.2 | 43.8 | | Athena-ORM | 28.6 | 66.9 | 22.0 | 25.8 | 15.2 | 59.1 | 40.8 | | Athena-PRM | 30.1 | 71.4 | 23.4 | 26.1 | 18.7 | 60.3 | 44.4 | | Qwen2.5-VL-72B [3] | 49.1 | 74.2 | 39.3 | 47.3 | 35.9 | 70.2 | 55.7 | | Self-consistency [58] | 54.8 | 77.0 | 43.1 | 50.8 | 37.6 | 71.1 | 59.6 | | Athena-ORM | 55.6 | 77.8 | 43.0 | 51.2 | 39.6 | 72.3 | 60.1 | | Athena-PRM | 58.7 | 79.1 | 44.8 | 54.6 | 42.5 | 75.8 | 60.9 | Table 2: Results on text-only math benchmarks under Best-of-N (N=8) evaluation. $\dagger$ denotes that the results are from [57]. We mark our results and highlight the best result. (a) Qwen2.5-7B | | GSM8K | MATH | | --- | --- | --- | | Qwen2.5-7B [61] | 91.6 | 75.5 | | Self-consistency [58] | 93.4 | 79.9 | | $\text{VisualPRM-8B}^{\dagger}$ [57] | 94.5 | 81.6 | | Athena-ORM | 94.0 | 81.1 | | Athena-PRM | 94.8 | 82.4 | (b) Ministral-8B | | GSM8K | MATH | | --- | --- | --- | | Ministral-8B | 85.6 | 54.5 | | Self-consistency [58] | 90.7 | 62.0 | | Athena-ORM | 91.4 | 63.2 | | Athena-PRM | 93.1 | 65.4 | (c) Qwen2.5-72B | | GSM8K | MATH | | --- | --- | --- | | Qwen2.5-72B [61] | 95.8 | 83.1 | | Self-consistency [58] | 96.0 | 86.0 | | $\text{VisualPRM-8B}^{\dagger}$ [57] | 96.5 | 85.2 | | Athena-ORM | 96.0 | 86.2 | | Athena-PRM | 97.0 | 87.4 | To comprehensively evaluate the proposed method, we assess Athena-ORM and Athena-PRM under the Best-of-N setting on seven multimodal benchmarks and two text-only benchmarks in Sec. 3.1. Furthermore, we evaluate Athena-PRM on VisualProcessBench, comparing its performance with MLLMs as judges and a recent open-source multimodal PRM, VisualPRM-8B [57], as described in Sec. 3.2. We assess the capability of Athena-7B on multimodal math and reasoning tasks in Sec. 3.3. To validate the effectiveness of our designs for Athena-PRM, an in-depth analysis is provided in Sec. 3.4. 3.1 Best-of-N Evaluation Benchmarks. We evaluate Athena-ORM and Athena-PRM across multiple multimodal mathematical and logical reasoning benchmarks, including MathVista [36], MathVision [54], MathVerse [68], WeMath [44], DynaMath [70], LogicVista [60], and MMMU [65]. To validate the effectiveness of Athena-ORM and Athena-PRM in the text-only scenario, we conduct experiments on the GSM8K [9] and MATH [18] under the BoN settings. Further details can be found in Appendix A.2. Settings. We employ Athena-ORM and Athena-PRM as the reward model for BoN evaluation, sampling $N=8$ solutions for every problem by default. For decoding, we set the temperature to 0.8 and use nucleus sampling [20] with top- $p$ set to 0.9, and report the average of five runtimes. For multimodal benchmarks, we choose Qwen2.5-VL-7B/72B [3] and InternVL2.5-8B [8] as policy models. All models used in this paper are “instruct” version. For simplicity, we ignore “instruct” in the text. For text-only datasets, we choose Qwen2.5-7/72B [61] and Ministral-8B https://huggingface.co/mistralai/Ministral-8B-Instruct-2410 as policy models. Zero-shot and self-consistency [58] serve as our baselines, alongside a recent public multimodal process reward model, VisualPRM-8B [57]. Training details. To train ORMs and PRMs, we fine-tune Qwen2.5-VL-7B on our dataset as introduced in Sec. 2.3. We use the AdamW optimizer [32] with the weight decay as 0 and set the learning rate as $1e^{-6}$ with cosine decay. We train all models with one epoch and set the global batch size to 64. We use DeepSpeed with zero2 [45] and flash-attention-2 [11] to improve training efficiency. To train ORMs, a new special token <step> is added at the end of solutions for ORMs training. This makes a consistent formulation for ORMs and PRMs. For PRMs, <step> is added to separate all reasoning steps and assign labels to corresponding tokens. We use “ $+$ ” and “ $-$ ” as labels to denote the correctness of each step. We use $8×$ AMD-MI250 GPUs for training and data synthesis. Results. Table 1 demonstrates that Athena-ORM and Athena-PRM generally enhance the reasoning performance of MLLMs across policy models of different sizes and benchmarks. Notably, Athena-PRM achieves a +10.2 points improvement on the WeMath dataset with Qwen2.5-VL-7B [3] as the policy model, compared to the zero-shot baseline. ORMs also exhibit general improvement across all benchmarks and policy models, although some gains are limited, such as +0.4 points on MathVerse with Qwen2.5-VL-72B [3]. In contrast, PRMs consistently show substantial performance gains over ORMs, illustrating the benefits of fine-grained rewards in test-time scaling. Table 2 highlights that Athena-PRM enhances reasoning abilities for text-only inputs in both the Qwen2.5 series and Ministral-8B models. Specifically, Athena-PRM achieves +9.9 points improvement for Ministral-8B on the challenging MATH [18] benchmark, demonstrating its effectiveness in text-only scenarios. Table 3: Results on VisualProcessBench. We report the F1-score on different dataset and macro F1 in overall. We highlight the best result across all models. | | MMMU | MathVision | MathVerse | DynaMath | WeMath | Overall | | --- | --- | --- | --- | --- | --- | --- | | Random Guessing | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | 50.0 | | Proprietary Models as Judge | | | | | | | | GPT-4o-mini [41] | 53.6 | 58.9 | 57.1 | 56.7 | 58.5 | 57.9 | | GPT-4o [41] | 56.3 | 60.2 | 59.7 | 59.0 | 63.3 | 60.3 | | Gemini-2.0-Flash [15] | 58.5 | 60.1 | 62.8 | 66.7 | 58.7 | 62.3 | | Open-source Models as Judge | | | | | | | | MiniCPM-V2.6-8B [62] | 44.9 | 50.9 | 58.9 | 46.7 | 57.4 | 50.4 | | LLaVA-OV-7B [26] | 45.7 | 43.0 | 42.2 | 44.7 | 52.5 | 44.4 | | LLaVA-OV-72B [26] | 46.1 | 48.4 | 53.0 | 57.0 | 57.3 | 52.3 | | Qwen2.5-VL-7B [3] | 53.1 | 51.8 | 47.8 | 51.3 | 54.2 | 51.0 | | Qwen2.5-VL-72B [3] | 59.2 | 59.0 | 59.7 | 62.9 | 62.3 | 60.5 | | InternVL2.5-8B [8] | 47.1 | 45.5 | 47.8 | 50.3 | 50.8 | 48.0 | | InternVL2.5-26B [8] | 48.8 | 47.4 | 49.2 | 50.4 | 51.4 | 49.2 | | InternVL2.5-38B [8] | 51.5 | 48.4 | 50.9 | 51.8 | 52.5 | 50.8 | | InternVL2.5-78B [8] | 52.0 | 51.7 | 53.7 | 50.8 | 52.5 | 52.6 | | Multimodal PRMs | | | | | | | | VisualPRM-8B [57] | 58.5 | 62.1 | 61.0 | 62.7 | 61.8 | 62.0 | | Athena-PRM | 74.1 | 61.2 | 60.7 | 72.7 | 73.8 | 65.9 | 3.2 Evaluation on Direct Judgment of Steps Setup. In addition to Best-of-N evaluation, we assess Athena-PRM on VisualProcessBench [57], comparing it with general MLLMs and VisualPRM-8B [57]. VisualProcessBench [57] aims to evaluate the ability to directly judge the correctness of each step. For MLLMs, we prompt the model to analyze and judge the correctness of each step. We report the F1-score for every subset and the micro-F1 score overall. Results. The results of VisualProcessBench are presented in Table 3. It is evident that current general open-source MLLMs struggle to judge the correctness of reasoning steps, with most $\scriptstyle\sim$ 7B models performing no better than the random guessing baseline. Conversely, multimodal process reward models achieve superior performance, rivaling proprietary models. Notably, our process reward model Athena-PRM, with only 7B parameters, achieves a +3.9 points improvement compared to VisualPRM-8B [57], even outperforming proprietary models as judges. 3.3 Evaluation on the Fine-tuned Model Table 4: Comparison of multimodal reasoning and mathematical performance across different models. $\Delta$ denotes improvements compared to Qwen2.5-VL-7B [61]. We highlight improvements at least +0.5 points with green. | | MMMU | WeMath | MathVista | MathVision | MathVerse | DynaMath | LogicVista | | --- | --- | --- | --- | --- | --- | --- | --- | | Proprietary Models | | | | | | | | | GPT-4o [41] | 72.9 | 50.6 | 71.6 | 43.8 | 49.9 | 48.5 | 64.4 | | Claude-3.7-Sonnet [1] | 71.0 | 49.3 | 66.8 | 41.9 | 46.7 | 39.7 | 58.2 | | Gemini-2.0-Pro [15] | 72.6 | 56.5 | 71.3 | 48.1 | 67.3 | 43.3 | 53.2 | | Open-source Models (>70B) | | | | | | | | | InternVL2.5-78B-MPO [56] | 68.2 | 37.6 | 76.6 | 36.2 | 43.7 | 21.2 | 50.8 | | InternVL2.5-78B [8] | 70.1 | 39.8 | 70.6 | 32.2 | 39.2 | 19.2 | 49.0 | | Qwen2.5-VL-72B [3] | 68.2 | 49.1 | 74.2 | 39.3 | 47.3 | 35.9 | 55.7 | | Open-source Models (~7B) | | | | | | | | | InternVL2.5-8B [8] | 56.2 | 20.2 | 58.3 | 20.0 | 20.4 | 9.2 | 33.6 | | MiniCPM-o-2.6-8B [62] | 50.9 | 25.2 | 73.3 | 21.7 | 35.0 | 10.4 | 36.0 | | Qwen2.5-VL-7B [3] | 58.0 | 36.2 | 68.1 | 25.4 | 41.1 | 21.8 | 47.9 | | Athena-7B | 61.1 | 43.0 | 71.4 | 25.7 | 45.7 | 21.9 | 51.3 | | $\Delta$ | +3.1 | +6.8 | +3.3 | +0.3 | +4.6 | +0.1 | +3.4 | Setup. We select Qwen2.5-VL-7B [3] as the initial policy model, obtaining training data as outlined in Sec. 2.4. We use AdamW [32] optimizer and set the learning rate as $1e^{-6}$ with cosine decay. The batch size is set to 128, and the number of epochs is set to one. We denote our fine-tuned model as Athena-7B and evaluate it on the same benchmarks as Sec. 3.1 in the zero-shot setting. For evaluation, we use VLMEvalKit [14] to evaluate Athena-7B and report the results of all baselines from the open leaderboard. https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard Results. As shown in Table 4, Athena-7B consistently improves upon Qwen2.5-VL-7B [3] across all benchmarks, with significant enhancements of at least $+$ 0.5 points on five benchmarks. Athena-7B outperforms large open-source models (e.g. InternVL2.5-78B [8]) on five benchmarks. On the MathVista [36], Athena-7B even achieves the comparable performance with proprietary models such as Gemini-2.0-Pro [15], Claude-3.7-Sonnet [1], and GPT-4o [41]. 3.4 Analysis Test time scaling. To evaluate the scalability of Athena-ORM and Athena-PRM, we compare their performance using different numbers of samples ranging from 4 to 64 on the MathVista [36]. As shown in Figure 2, reward ranking for solutions at test time improves reasoning performance across different policy models and number of samples. Specifically, ORMs and PRMs improve the self-consistency [58] baseline by 0.8 and 2.1 points with Qwen2.5-VL-72B [3] using eight samples, respectively. As the sample size increases, the performance improves steadily. However, the performance gain of ORMs is markedly lower than that of PRMs. For instance, Athena-PRM achieves 82.4% with Qwen2.5-VL-72B [3] using 64 samples, while Athena-ORM achieves similar performance with self-consistency [58] baseline. These findings underscore the advancements and scalability of PRMs in test-time scaling scenarios. <details> <summary>x2.png Details</summary>  ### Visual Description ## Line Chart: MathVista Accuracy (%) vs. # Solutions per Problem ### Overview The chart compares the accuracy of four methods (PRM, ORM, Self-consistency, Zero-shot) across varying numbers of solutions per problem (4, 8, 16, 32, 64). Accuracy is measured in percentage, with PRM achieving the highest values and Zero-shot remaining constant. ### Components/Axes - **X-axis**: "# Solutions per problem" (logarithmic scale: 4, 8, 16, 32, 64). - **Y-axis**: "MathVista Accuracy (%)" (68% to 76%). - **Legend**: - **PRM**: Teal diamond line (highest accuracy). - **ORM**: Orange triangle line. - **Self-consistency**: Red square line. - **Zero-shot**: Dashed blue cross line (baseline at 68%). ### Detailed Analysis 1. **PRM (Teal)**: - Starts at ~72.5% (4 solutions), peaks at ~76.5% (64 solutions). - Steady upward trend with minor fluctuations (e.g., slight dip at 16 solutions). 2. **ORM (Orange)**: - Begins at ~70% (4 solutions), rises to ~73.5% (64 solutions). - Gradual increase with minor plateaus (e.g., stable at 16 and 32 solutions). 3. **Self-consistency (Red)**: - Starts at ~69.5% (4 solutions), ends at ~73% (64 solutions). - Consistent upward trajectory with no dips. 4. **Zero-shot (Blue)**: - Flat line at ~68% across all solution counts. ### Key Observations - **PRM dominates** in accuracy, outperforming all methods by ~3–4% at 64 solutions. - **ORM and Self-consistency** show similar improvement patterns but lag behind PRM. - **Zero-shot** remains unchanged, serving as a static baseline. - **Divergence grows** with more solutions: PRM’s lead over others widens significantly (e.g., ~3.5% gap at 64 solutions vs. ~2% at 4 solutions). ### Interpretation The data suggests that **increasing the number of solutions per problem improves accuracy** for PRM, ORM, and Self-consistency, with PRM being the most scalable method. Zero-shot’s stagnation implies it lacks adaptive mechanisms to leverage additional solutions. The logarithmic x-axis hints at exponential scaling benefits for PRM, potentially due to its architecture (e.g., iterative refinement). ORM and Self-consistency may rely on simpler heuristics, limiting their gains. This trend underscores the importance of solution quantity in optimizing performance for complex tasks. </details> (a) Qwen2.5-VL-7B <details> <summary>x3.png Details</summary>  ### Visual Description ## Line Chart: MathVista Accuracy vs. Number of Solutions per Problem ### Overview The chart illustrates the relationship between the number of solutions per problem and MathVista accuracy (%), with three distinct data series represented by different markers and colors. The x-axis (logarithmic scale) shows the number of solutions per problem (4, 8, 16, 32, 64), while the y-axis (linear scale) represents accuracy from 65% to 75%. A constant blue dashed line at 65% serves as a baseline. ### Components/Axes - **X-axis**: "# Solutions per problem" (logarithmic scale: 4, 8, 16, 32, 64). - **Y-axis**: "MathVista Accuracy (%)" (linear scale: 65% to 75%). - **Legend**: - **Teal diamonds**: Line 1 (highest accuracy). - **Orange triangles**: Line 2 (middle accuracy). - **Red squares**: Line 3 (lowest accuracy). - **Blue dashed line**: Constant baseline at 65%. ### Detailed Analysis 1. **Teal Diamonds (Line 1)**: - Starts at **68%** for 4 solutions. - Increases to **71.5%** at 8 solutions. - Rises to **73.5%** at 16 solutions. - Peaks at **74%** for 32 solutions. - Reaches **75%** at 64 solutions. - **Trend**: Steep initial growth, then plateaus. 2. **Orange Triangles (Line 2)**: - Begins at **65%** for 4 solutions. - Increases to **67%** at 8 solutions. - Rises to **68.5%** at 16 solutions. - Peaks at **69.5%** for 32 solutions. - Slightly declines to **69.5%** at 64 solutions. - **Trend**: Gradual, steady increase with a minor plateau. 3. **Red Squares (Line 3)**: - Starts at **64%** for 4 solutions. - Increases to **66%** at 8 solutions. - Rises to **68%** at 16 solutions. - Peaks at **68.5%** for 32 solutions. - Declines to **67.5%** at 64 solutions. - **Trend**: Rapid initial growth, then a slight drop at 64 solutions. 4. **Blue Dashed Line**: - Constant at **65%** across all solution counts. - **Trend**: Flat, no variation. ### Key Observations - **Teal Line (Line 1)** consistently outperforms other series, showing the highest accuracy. - **Orange Line (Line 2)** demonstrates a moderate improvement, with a slight plateau at 64 solutions. - **Red Line (Line 3)** exhibits the most variability, with a notable drop at 64 solutions. - The **blue baseline (65%)** acts as a reference point, indicating that all series exceed this minimum accuracy. ### Interpretation The data suggests that increasing the number of solutions per problem generally improves MathVista accuracy, but the rate of improvement varies by series. - **Teal Line (Line 1)**: Likely represents the most effective method, with a strong correlation between solution count and accuracy. - **Orange Line (Line 2)**: Indicates a balanced approach, with consistent gains but limited by a plateau. - **Red Line (Line 3)**: Shows initial promise but may suffer from diminishing returns or inefficiencies at higher solution counts (e.g., the drop at 64 solutions). - The **blue baseline** highlights that even the lowest-performing series (red) exceeds the minimum threshold, suggesting room for optimization across all methods. **Notable Anomaly**: The red line’s drop at 64 solutions (67.5% vs. 68.5% at 32) could indicate a regression or a limitation in scaling the method for larger problem sets. Further investigation into the methodology or data collection for this series is warranted. </details> (b) InternVL2.5-8B <details> <summary>x4.png Details</summary>  ### Visual Description ## Line Chart: MathVista Accuracy vs. Solutions per Problem ### Overview The chart illustrates the relationship between the number of solutions per problem and MathVista accuracy percentages. Three data series (teal, orange, red) show increasing accuracy trends as solution counts grow from 4 to 64. A blue dashed baseline (74.5%) and gray dashed target (74%) provide reference points. ### Components/Axes - **X-axis**: "# Solutions per problem" (logarithmic scale: 4, 8, 16, 32, 64) - **Y-axis**: "MathVista Accuracy (%)" (74–82%) - **Legend**: - Teal (diamond markers): "Series A" - Orange (triangle markers): "Series B" - Red (square markers): "Series C" - **Dashed Lines**: - Blue: "Baseline Accuracy (74.5%)" - Gray: "Target Accuracy (74%)" ### Detailed Analysis 1. **Teal Line (Series A)**: - Starts at ~76% accuracy at 4 solutions. - Increases steadily to ~82% at 64 solutions. - Slope: ~0.09% per solution increment (approximate). 2. **Orange Line (Series B)**: - Begins at ~75% at 4 solutions. - Rises to ~79% at 64 solutions. - Slope: ~0.06% per solution increment. 3. **Red Line (Series C)**: - Starts at ~74.5% at 4 solutions. - Reaches ~79% at 64 solutions. - Slope: ~0.07% per solution increment. 4. **Dashed Lines**: - Blue line remains constant at 74.5% across all solution counts. - Gray line remains constant at 74% across all solution counts. ### Key Observations - All three data series show upward trends, with teal (Series A) demonstrating the steepest improvement. - At 64 solutions, all series exceed the gray "Target Accuracy" line (74%). - The blue "Baseline Accuracy" (74.5%) is consistently surpassed by all series after 8 solutions. - Red (Series C) starts closest to the blue baseline but diverges significantly by 64 solutions. ### Interpretation The data suggests a strong positive correlation between solution quantity and accuracy improvement. Teal (Series A) outperforms other series across all solution counts, potentially indicating superior algorithmic efficiency or data quality. The convergence of orange and red lines at higher solution counts implies diminishing returns for additional solutions beyond a certain threshold. The persistent gap between the blue baseline and all series highlights that even minimal solution counts (4) already exceed the "Baseline Accuracy" metric, suggesting robust foundational performance. The gray target line (74%) appears overly conservative, as all series surpass it within the first data point (4 solutions). </details> (c) Qwen2.5-VL-72B Figure 2: Best-of-N results on the MathVista [36] across different policies. The number of solutions we sample is from 4 to 64. Table 5: Ablation study of our design choices on the MathVista [54] and WeMath [44] with Qwen2.5-VL-7B [3] as the policy model. | Method | Data | ORM init | Up-sample | MathVista | WeMath | | --- | --- | --- | --- | --- | --- | | Zero-shot | - | - | - | 68.1 | 36.2 | | Self-consistency | - | - | - | 71.6 | 44.7 | | Athena-ORM | Athena-600K | - | - | 72.8 | 45.1 | | Athena-PRM | Random 5K | ✘ | ✘ | 73.1 | 45.2 | | Athena-5K | ✘ | ✘ | 74.1 | 45.8 | | | Random 5K | ✔ | ✘ | 73.6 | 45.4 | | | Athena-5K | ✔ | ✘ | 74.8 | 46.2 | | | Athena-5K | ✔ | ✔ | 75.2 | 46.4 | | | MC-300K | ✔ | ✔ | 75.5 | 46.4 | | The effectiveness of our design choices. Table 5 presents results under various design choices. Our findings indicate that PRMs trained with selectively chosen 5K samples significantly outperform those trained with randomly selected samples, achieving scores 74.1 vs. 73.1 and 74.8 vs. 73.6 when initialized from ORMs. Moreover, a large-scale dataset of 300K samples labeled using vanilla MC-based methods yields similar performance to our selected 5K samples. With these selected samples, ORM initialization enhances performance by 0.7 points (74.8 vs. 74.1), and up-sampling negative labels further boosts performance by 0.4 points on the MathVista [36]. Computation cost analysis. Vanilla MC-based estimation [55] incurs substantial computational demands to estimate the correctness of intermediate steps. Our prediction consistency filter strategy significantly reduces this burden, requiring only 5K samples for training, thereby lowering the computational cost by a factor of 60. For data synthesis, we employ vLLM primarily [25] to accelerate MLLM inference. Compared to the MC-300K dataset, our Athena-5K dataset reduces GPU hours by approximately 45-fold while achieving similar performance. 4 Related Work Test time scaling. While scaling data and model parameters during training has been extensively studied [23, 19], scaling computation at test time has garnered interest more recently [50, 46]. TTS enables language models to leverage additional computational resources when faced with challenging questions. Our study primarily focuses on parallel scaling [67], wherein language models generate multiple outputs simultaneously and aggregate them into a final answer. This aggregation can involve selecting the most common outcome [58] or using reward models [50] to rank solutions, selecting the one with the highest reward, as demonstrated in our Best-of-N settings. Process reward models. Reward models are crucial in reinforcement learning [17, 47, 55, 42, 4, 13] and test time scaling [46, 28, 55, 9, 50]. Outcome reward models [9] assign a scalar value to question-response pairs, whereas process reward models evaluate each step, typically achieving superior performance and generalization [28, 37, 46, 27]. While better performance and generalization are achieved, collecting data to train PRMs is challenging. PRM800K [28] is the first open-source process supervision dataset annotated by humans, but scaling it is challenging due to the time and skill required for annotating reasoning steps. Math-Shepherd [55] proposes an automated method for estimating intermediate step correctness using Monte Carlo estimation, though it generates some incorrect labels and demands extensive computational resources. Our work explores strategies to reduce computational costs while improving process label accuracy. 5 Conclusion In this paper, we introduce Athena-PRM, trained on 5K high-quality data with process labels. To automatically obtain accurate process labels, we employ a data filtering method based on consistency between weak and strong completers. This strategy significantly reduces computational costs for training and data synthesis while ensuring precise process labels for training PRMs. Additionally, we present two effective strategies for training PRMs: ORM initialization and negative data up-sampling. Athena-PRM demonstrates substantial performance improvements in Best-of-N evaluations, achieving state-of-the-art results on VisualProcessBench. Leveraging Athena-PRM, we train Athena-7B, fine-tuned from Qwen2.5-VL-7B using a reward ranked fine-tuning approach, markedly enhancing the reasoning capabilities of Qwen2.5-VL-7B. Extensive experiments confirm the effectiveness of our proposed methods. References - Anthropic [2025] Anthropic. Claude 3.7 sonnet and claude code, 2025. URL https://www.anthropic.com/news/claude-3-7-sonnet. Accessed on Feb 25, 2025. - Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, 2015. - Bai et al. [2025] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. - Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. - Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020. - Cao and Xiao [2022] Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th international conference on computational linguistics, pages 1511–1520, 2022. - Chen et al. [2022] Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746, 2022. - Chen et al. [2024] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024. - Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. - Cui et al. [2025] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025. - Dao [2024] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In ICLR, 2024. - Dong et al. [2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. - Dong et al. [2024] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. Transactions on Machine Learning Research, 2024. - Duan et al. [2024] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. VLMEvalKit: An open-source toolkit for evaluating large multi-modality models. In ACM MM, 2024. - Gemini Team [2023] Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. - Grattafiori et al. [2024] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. - Guo et al. [2025] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. - Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021. - Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. - Holtzman et al. [2020] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. - Hu [2025] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025. - Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. - Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. - Kazemi et al. [2023] Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023. - Kwon et al. [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. - Li et al. [2025] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research, 2025. - Li and Li [2025] Wendi Li and Yixuan Li. Process reward model with q-value rankings. In ICLR, 2025. - Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In ICLR, 2024. - Lindström and Abraham [2022] Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022. - Liu et al. [2024] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. - Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. - Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. - Lu et al. [2024a] Jianqiao Lu, Zhiyang Dou, Hongru WANG, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. AutoPSV: Automated process-supervised verifier. In NeurIPS, 2024a. - Lu et al. [2021] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165, 2021. - Lu et al. [2022] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022. - Lu et al. [2024b] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024b. - Luo et al. [2024] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024. - Masry et al. [2022] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022. - Mathew et al. [2021] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021. - Muennighoff et al. [2025] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025. - OpenAI [2024] OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024. Accessed: 2024-02-09, 2024-02-11, 2024-02-12. - Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022. - Pang et al. [2024] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. Iterative reasoning preference optimization. In NeurIPS, 2024. - Qiao et al. [2024] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024. - Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020. - Setlur et al. [2025] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In ICLR, 2025. - Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. - Shi et al. [2024] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See Kiong Ng, Lidong Bing, and Roy Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. In EMNLP, 2024. - Singh et al. [2024] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024. - Snell et al. [2025] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In ICLR, 2025. - Team et al. [2025] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. - Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. - Uesato et al. [2022] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. - Wang et al. [2024a] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In NeurIPS, 2024a. - Wang et al. [2024b] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In ACL, 2024b. - Wang et al. [2024c] Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024c. - Wang et al. [2025] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. VisualPRM: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291, 2025. - Wang et al. [2023] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. - Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022. - Xiao et al. [2024] Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973, 2024. - Yang et al. [2024] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. - Yao et al. [2024] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. - Yuan et al. [2024] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01981, 2024. - Yuan et al. [2023] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. - Yue et al. [2024] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. - Zeng et al. [2025] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B-STar: Monitoring and balancing exploration and exploitation in self-taught reasoners. In ICLR, 2025. - Zhang et al. [2025a] Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models. arXiv preprint arXiv:2503.24235, 2025a. - Zhang et al. [2024] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In ECCV, 2024. - Zhang et al. [2025b] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025b. - Zou et al. [2025] Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In ICLR, 2025. Appendix A More Experiments and Details A.1 Choices of Completers Here we verify the effect of combining different completers on the accuracy of estimated labels. Results are listed in Table 6. From Table 6, it is noted that the consistency filter for two completers usually improves the accuracy of estimated labels. For example, combining two weak or strong completers improves accuracy by almost two points in general. Furthermore, the accuracy of estimated labels is improved by combining one weak and one strong completer, achieving the most competitive accuracy, e.g., 95.2% with weak completer $\phi_{w}^{2}$ and strong completer $\phi_{s}^{1}$ . This shows that the proposed consistency filter strategy is effective across all combinations of different completers and achieves the best performance when we combine a weak completer and a strong completer. Table 6: The accuracy of estimated process labels with Different completers combination. In Two Completers, we use consistency filter introduced in Sec. 2.2. $\phi_{w}^{1}$ , $\phi_{w}^{2}$ , $\phi_{s}^{1}$ , and $\phi_{w}^{2}$ denote Mistral-7B-Instruct, Qwen2-7B-Instruct, Qwen2.5-72B-Instruct and Llama-3-70B-Instruct, respectively. | | Weak Completer | Strong Completer | Accuracy(%) | | | | --- | --- | --- | --- | --- | --- | | $\phi_{w}^{1}$ | $\phi_{w}^{2}$ | $\phi_{s}^{1}$ | $\phi_{s}^{2}$ | | | | Single Completer | ✔ | | | | 78.2 | | ✔ | | | 80.1 | | | | ✔ | | 83.4 | | | | | ✔ | 84.7 | | | | | | Two Completers | ✔ | ✔ | | | 82.3 | | ✔ | ✔ | 85.6 | | | | | ✔ | | ✔ | | 94.1 | | | ✔ | | | ✔ | 93.8 | | | ✔ | ✔ | | 95.2 | | | | ✔ | | ✔ | 94.7 | | | A.2 Benchmarks We provide more details about used benchmarks in Table 7. Table 7: Details of all benchmarks used and corresponding metrics. | | Split | # Samples | Metric | | --- | --- | --- | --- | | MathVista [36] | Testmini | 1000 | Accuracy | | MathVision [54] | Full | 3040 | Accuracy | | MathVerse [68] | Vision-Only | 788 | Accuracy | | DynaMath [70] | Full | 5050 | Accuracy | | WeMath [44] | Testmini | 1740 | Score (Strict) | | LogicVista [60] | Full | 448 | Accuracy | | MMMU [65] | Validation | 900 | Accuracy | | GSM8K [9] | Test | 1319 | Accuracy | | MATH [18] | Test | 5000 | Accuracy | | VisualProcessBench [57] | - | 2866 | F1-score | A.3 Reward Choice In this paper, we consider three reward choices of Athena-PRM for aggregating step scores into a final score [69], including PRM-minium, PRM-last, and PRM-product. For all reasoning steps $\{a_{i}\}_{i=1}^{K}$ , its corresponding reward for each step is $\{r_{i}\}_{i=1}^{K}$ , the reward $r$ for each solution with different scoring methods are calculated as follows: 1) PRM-minium uses the minium reward across all steps, i.e., $r=\min\{r_{i}\}_{i=1}^{K}$ ; 2) PRM-last uses the reward of last step, i.e., $r=r_{K}$ ; 3) PRM-product uses the reward product of all steps, i.e. $r=\prod_{i=1}^{K}r_{i}$ . We list results in Table 8. It is shown that different choices of selecting a reward for PRMs could produce different performance. In all other experiments, we use the minimum reward across all steps as the reward for the solution unless otherwise specified. Table 8: Best-of-8 performance comparison on the MathVista [36] with Qwen2.5-VL-7B [3] with different reward choices: last step reward, minium reward across all steps and product of all rewards. Default settings are marked in gray. | Method | Accuracy(%) | | --- | --- | | Zero-shot | 68.1 | | Self-consistency | 71.6 | | ORM | 72.8 | | PRM-product | 74.5 | | PRM-last | 75.1 | | PRM-minium | 75.2 | A.4 Label Distribution We present the distribution of process labels in Table 9. It is shown that label imbalance exists in PRM800K and Athena-300K. We try different up-sample rates (e.g. $× 2$ ) for samples with error steps and find that slightly up-sampling negative data is beneficial for training PRMs, as shown in Table 10. We find that increasing the up-sample rate will not improve performance. We set the up-sample rate to 2 for other experiments. Table 9: Distribution of labels across different datasets. Good, neutral, and bad denote different process labels. | Dataset | Good | Neutral | Bad | | --- | --- | --- | --- | | PRM800K [28] | 76% | 12% | 12% | | Athena-300K | 81% | - | 19% | | Athena-5K | 79% | - | 21% | Table 10: Results on MathVista under Best-of-8 setting with different up-sampling rate. We use Athena-5K as training dataset and Qwen-2.5-VL-7B as the policy model. Default settings are marked in gray. | Method | Up-sample rate | Accuracy(%) | | --- | --- | --- | | Zero-shot | - | 68.1 | | Self-consistency | - | 71.6 | | Athena-ORM | - | 72.8 | | Athena-PRM | ✘ | 74.8 | | Athena-PRM | $× 2$ | 75.2 | | Athena-PRM | $× 5$ | 75.2 | | Athena-PRM | $× 10$ | 74.9 | Appendix B Limitations and Social Impacts Limitations. Athena-PRM achieves impressive performance across various scenarios and benchmarks. However, Athena-PRM is specifically designed to enhance the reasoning capabilities of LLMs and MLLMs. Its application may be limited to solving problems with clear answers, such as multi-step reasoning and math problems. Effectively assigning rewards for intermediate steps in open-ended text generation tasks remains an open problem, and we leave this for future research. In addition, how to define a step for complex multi-step reasoning is still underexplored. Existing PRMs typically split solutions into multiple steps using predefined symbols (e.g., “\n\n”) or fixed numbers of tokens or lines. This approach may result in steps that lack sufficient information. To prevent steps from containing too little information, we design a simple strategy that merges steps with too few tokens or without any mathematical symbols. However, better splitting strategies without human-designed rules still need to be explored. Social Impacts. The paper introduces an effective method for generating high-quality, process-labeled data for reasoning problems. Our approach enhances the efficiency and effectiveness of training PRMs. We hope that our research will have a broader impact on improving the reasoning abilities of foundation models. Since our method focuses on general improvements in the reasoning capabilities of foundation models, it does not directly produce negative societal impacts. Appendix C Case Study We provide some examples from VisualProcessBench and MathVista to help readers understand how PRMs work, see Figure 3, 4, 5, 6. <details> <summary>extracted/6532028/fig/example_fig/images_MathVision_MINI_15.png Details</summary>  ### Visual Description ## Diagram: Grid-Based Creature Movement Puzzle ### Overview The image depicts a grid-based puzzle with labeled sections (<image1> to <image7>). Key elements include: - **<image1>**: Red ant with white flag (standing) - **<image2>**: Simple house outline - **<image3>**: Red ladybug - **<image4>**: Red ant with white flag (walking) - **<image5>**: Simple house outline - **<image6>**: 5x5 grid with creatures and directional arrows - **<image7>**: Multiple-choice options (A-E) with creature icons ### Components/Axes 1. **Grid Structure** (<image6>): - 5x5 checkerboard pattern (alternating gray/white cells) - No explicit axis labels, but cells are referenced by position - Creatures occupy specific cells with directional arrows indicating movement paths 2. **Creature Labels**: - Ant (red with white flag) - Ladybug (red with black spots) - Butterfly (orange/black wings) - Snail (brown shell) - Frog (green with red tongue) - Worm (green segmented body) 3. **Directional Arrows**: - Black arrows show movement between cells - Arrows connect: - Ant → Ladybug → Butterfly - Ant → Snail → Worm - Ant → Frog 4. **Multiple-Choice Options** (<image7>): - (A) Butterfly - (B) Bee (yellow/black stripes) - (C) Worm - (D) Snail - (E) Frog ### Detailed Analysis - **Grid Placement**: - Ant: Top-left cell (row 1, column 1) - Ladybug: Top-middle cell (row 1, column 3) - Butterfly: Top-right cell (row 1, column 5) - Snail: Middle row, middle column (row 3, column 3) - Worm: Middle row, left column (row 3, column 1) - Frog: Middle row, right column (row 3, column 5) - **Arrow Connections**: - Ant (1,1) → Ladybug (1,3) → Butterfly (1,5) - Ant (1,1) → Snail (3,3) → Worm (3,1) - Ant (1,1) → Frog (3,5) - **Creature Repetition**: - Ant appears in <image1> (standing) and <image4> (walking) - House outline appears in <image2> and <image5> ### Key Observations 1. **Movement Constraints**: - Arrows suggest sequential movement from ant to other creatures - No direct path from ladybug/butterfly to other creatures 2. **Creature Distribution**: - Ant is the only creature appearing in multiple images - House outline is repeated but unlabeled 3. **Option Ambiguity**: - Option (B) "Bee" does not appear in the grid - All other options (A, C, D, E) correspond to grid creatures ### Interpretation This appears to be a logic puzzle or game interface where: 1. The ant must navigate the grid to reach specific creatures 2. Arrows represent allowed movement paths 3. <image7> options likely represent choices for the next move 4. Repeated house outlines may indicate "home" positions or level markers The puzzle emphasizes spatial reasoning and pathfinding, with the ant as the primary agent. The absence of a bee in the grid despite its inclusion in options suggests potential red herrings or incomplete information. The directional arrows create a dependency chain, requiring players to plan sequential moves carefully. </details> | Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end. Question: When the ant <image1> walks from home <image2> along the arrows $→ 3,\uparrow 3,→ 3,\uparrow 1$ , he gets to the ladybird <image3>. Which animal does the ant <image1> get to when he walks from home <image2> along the following arrows: $→ 2,\downarrow 2,→ 3,\uparrow 3,→ 2,\uparrow 2$ ? <image6> <image7> Choices:(A) A (B) B (C) C (D) D (E) E (Answer: A) | | | | --- | --- | --- | | Step | Reward | Label | | To determine which animal the ant reaches, we will follow the given movements step-by-step on the grid: | 0.95 | ✔ | | 1. Start at the home position (‘<image2>‘), which is at the bottom-left corner of the grid. | 0.99 | ✔ | | 2. Follow the arrows: - $→ 2$ : Move 2 squares to the right. - $\downarrow 2$ : Move 2 squares down. - $→ 3$ : Move 3 squares to the right. - $\uparrow 3$ : Move 3 squares up. - $→ 2$ : Move 2 squares to the right. - $\uparrow 2$ : Move 2 squares up. | 0.99 | ✔ | | 3. Trace the movement: - Start at the bottom-left corner. - Move 2 squares right: You are now in the third column of the first row. - Move 2 squares down: You are now in the third column of the third row. - Move 3 squares right: You are now in the sixth column of the third row. - Move 3 squares up: You are now in the sixth column of the top row. - Move 2 squares right: You are now in the eighth column of the top row. - Move 2 squares up: You are now in the eighth column of the top row (still at the butterfly position). | 0.12 | ✘ | | The ant reaches the butterfly, which corresponds to option A. | 0.87 | ✔ | | ### Final Answer: A | 0.75 | ✔ | Figure 3: A case study from VisualProcessBench [57]. <details> <summary>extracted/6532028/fig/example_fig/images_MathVerse_MINI_Vision_Only_35.png Details</summary>  ### Visual Description ## Geometry Diagram: Circle with Tangent and Inscribed Angle ### Overview The image depicts a circle labeled **O** with center **O**. A tangent line **CD** touches the circle at point **C**, and point **D** lies outside the circle. Points **A** and **B** are on the circle, connected to **O** via radii **OA** and **OB**. The problem asks for the measure of **angle A**, with options provided. A labeled angle of **50°** is marked at point **D**. --- ### Components/Axes - **Circle**: Centered at **O**, labeled as "circle O." - **Tangent Line**: **CD** touches the circle at **C** (tangent property: **OC ⊥ CD**). - **Points**: - **A**, **B**: On the circle, connected to **O** (radii **OA**, **OB**). - **C**: Point of tangency. - **D**: External point where **CD** extends. - **Angles**: - **∠D = 50°** (marked at point **D**). - **∠A**: To be determined (options: 20°, 25°, 40°, 50°). --- ### Detailed Analysis 1. **Tangent Property**: Since **CD** is tangent to the circle at **C**, **OC ⊥ CD** (angle **∠OCD = 90°**). 2. **Triangle OCD**: - **∠OCD = 90°** (tangent-radius property). - **∠D = 50°** (given). - **∠COD = 180° - 90° - 50° = 40°** (sum of angles in a triangle). 3. **Central Angle ∠COD = 40°**: This angle subtends arc **CB**. 4. **Inscribed Angle Theorem**: The inscribed angle subtended by the same arc **CB** is half the central angle: - **∠CAB = ½ × 40° = 20°**. 5. **Angle A**: If **angle A** refers to **∠CAB**, it equals **20°**. --- ### Key Observations - The tangent **CD** and radius **OC** form a right angle (**90°**). - The central angle **∠COD** (40°) directly relates to the inscribed angle **∠CAB** via the inscribed angle theorem. - The given **50°** at **D** is critical for calculating **∠COD**. --- ### Interpretation The problem leverages the **tangent-radius perpendicularity** and **inscribed angle theorem**. The **50°** at **D** allows calculation of the central angle **∠COD = 40°**, which is halved to find the inscribed angle **∠A = 20°**. This aligns with option **A: 20°**. The diagram emphasizes the relationship between external angles (e.g., **∠D**) and internal circle properties (central/inscribed angles). </details> | According to the question shown in the image, please directly answer the question and provide the correct option letter, e.g., A, B, C, D. (Answer: A) | | | | --- | --- | --- | | Step | Reward | Label | | To find the degree of angle A, we can use the property that the angle subtended by an arc at the center of the circle is twice the angle subtended at any point on the circle. | 0.82 | ✔ | | 1. Identify the key points: - The arcs are AC and BC. - Angle A is subtended by these arcs at point O (the center of the circle). | 0.85 | ✔ | | 2. Understand the given information: - CD is a tangent at point C. - The angle between the tangent and the chord is equal to the angle subtended by the chord at the circumference. Therefore, angle CDB = 50°. | 0.85 | ✔ | | 3. Angle properties: - The angle A is subtended by arc BC at the center. - Since CD is a tangent, angle A = 50° (as the inscribed angle is half the central angle). | 0.23 | ✘ | | Thus, the degree of angle A is 20°. | 0.67 | ✔ | | Final answer: A | 0.96 | ✔ | Figure 4: A case study from VisualProcessBench [57]. <details> <summary>extracted/6532028/fig/example_fig/mathvista_559.jpg Details</summary>  ### Visual Description ## Photograph: Omega Wristwatch ### Overview The image depicts a silver Omega wristwatch with a black dial, positioned on a white surface against a neutral background. The watch features a metallic bracelet, a tachymeter bezel, and a transparent case back revealing the movement. The dial includes hour markers, hands, and text, with a date display at the 3 o'clock position. ### Components/Axes - **Brand**: "OMEGA" (center of the dial, gold text). - **Movement Type**: "Co-Axial" (text near the 6 o'clock position, gold). - **Bezel**: Black with white numerals (10, 20, 30, 40, 50, 60) in 10-minute increments. - **Hour Markers**: White Arabic numerals (1–12) on the dial. - **Hands**: Yellow hour and minute hands; red second hand. - **Date Display**: "28" (visible at 3 o'clock, white text on black background). - **Case Back**: Transparent, exposing the internal movement. - **Crown**: Silver, located on the right side of the case. ### Detailed Analysis - **Bezel**: The tachymeter scale ranges from 10 to 60, with 10-minute intervals. - **Dial Text**: - "OMEGA" (center, gold). - "Co-Axial" (near 6 o'clock, gold). - **Time Display**: - Hour hand: ~10. - Minute hand: ~10. - Second hand: ~20. - **Date**: "28" (visible at 3 o'clock). - **Case**: Silver metal bracelet with polished links. ### Key Observations - The watch is a luxury chronograph, indicated by the tachymeter bezel and "Co-Axial" movement. - The date display is functional but lacks a month indicator. - The transparent case back highlights the mechanical complexity of the movement. - The time shown (10:10) is a common default for watch photography, emphasizing symmetry. ### Interpretation This Omega watch combines precision engineering (Co-Axial movement) with practical features (tachymeter, date). The black dial and yellow hands enhance readability, while the transparent case back appeals to enthusiasts of mechanical design. The absence of a month indicator on the date display suggests a focus on simplicity over additional functionality. The watch’s design aligns with Omega’s reputation for high-quality, functional timepieces. </details> | Col1 | Col2 | Col3 | | --- | --- | --- | | Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: what is the highest number shown on the black outer part of the watch? (Answer: 55) | | | | Step | Reward | Label | | To determine the highest number shown on the black outer part of the watch, let’s analyze the image step by step | 0.90 | ✔ | | 1. The black outer part of the watch appears to be a bezel with numbers printed on it. | 0.87 | ✔ | | 2. The numbers on the bezel are arranged in a circular pattern. | 0.96 | ✔ | | 3. Observing the numbers, they range from 0 to 60, increasing in increments of 5. | 0.83 | ✔ | | 4. The highest number visible on the bezel is 60. | 0.34 | ✘ | | Therefore, the highest number shown on the black outer part of the watch is 60. | 0.15 | ✘ | | Answer: $\boxed{60}$ | 0.04 | ✘ | Figure 5: A case study from MathVista [36] with Qwen2.5-VL-72B [3]. <details> <summary>extracted/6532028/fig/example_fig/mathvista_605.jpg Details</summary>  ### Visual Description ## 3D Rendering: Geometric Shapes on Neutral Surface ### Overview The image depicts a 3D-rendered scene of seven geometric objects arranged on a flat, neutral gray surface. The composition includes spheres, cubes, and cylinders with varying materials (metallic, matte) and colors (gold, green, cyan, bronze, gray). No textual labels, legends, or annotations are visible in the image. ### Components/Axes - **Objects**: 1. **Golden Sphere**: Large, reflective, positioned left-center. 2. **Green Cube**: Matte, located in the background (top-center). 3. **Cyan Cylinder**: Large, matte, centered-right. 4. **Bronze Cube**: Reflective, positioned far-right. 5. **Small Gray Sphere**: Matte, near the green cube. 6. **Small Cyan Sphere**: Matte, near the cyan cylinder. 7. **Small Gray Cylinder**: Reflective, near the golden sphere. - **Spatial Relationships**: - Objects are evenly spaced but vary in size and material. - Lighting creates subtle shadows, emphasizing material properties (e.g., reflections on metallic objects). ### Detailed Analysis - **Material Properties**: - Metallic objects (golden sphere, bronze cube, gray cylinder) exhibit high reflectivity. - Matte objects (green cube, cyan cylinder, small spheres) lack reflections. - **Color Distribution**: - Dominant colors: Gold, green, cyan, bronze. - Neutral grays used for smaller objects and surface. - **Size Hierarchy**: - Largest: Golden sphere, cyan cylinder. - Medium: Green cube, bronze cube. - Smallest: Gray sphere, cyan sphere, gray cylinder. ### Key Observations 1. **Symmetry and Balance**: The arrangement lacks strict symmetry but achieves visual balance through color and size contrasts. 2. **Material Contrast**: Reflective vs. matte surfaces create a tactile distinction. 3. **Lighting Effects**: Shadows and highlights suggest a single light source from the upper-left. ### Interpretation This image likely serves as a study in 3D rendering techniques, focusing on material realism, lighting, and spatial composition. The absence of labels or annotations suggests it is not data-driven but rather a demonstration of rendering capabilities. The use of contrasting materials and colors may highlight how light interacts with different surfaces, useful for applications in computer graphics, product visualization, or educational contexts. ## No textual data, charts, or diagrams present. </details> | Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: Subtract all large green matte cubes. Subtract all big green blocks. How many objects are left? (Answer: 6) | | | | --- | --- | --- | | Step | Reward | Label | | To solve this problem, let’s identify and count the objects in the image: | 0.95 | ✔ | | 1. There is one large green matte cube. | 0.99 | ✔ | | 2. There are no other big green blocks since the only green block is already counted. | 0.90 | ✔ | | Now, we subtract these objects as per the instructions | 0.92 | ✔ | | - Subtract all large green matte cubes: This removes the large green matte cube from the count. | 0.99 | ✔ | | - Subtract all big green blocks: Since there are no other big green blocks, nothing additional is removed. | 0.99 | ✔ | | After removing the large green matte cube, the remaining objects are: - A small gray metallic cylinder. - A large gold metallic sphere. - A small gray matte sphere. - A large cyan matte cylinder. - A small cyan matte sphere. - A large gold metallic cube. | 0.84 | ✔ | | Counting the remaining objects, we have 6 objects left. | 0.92 | ✔ | | Answer: $\boxed{6}$ | 0.99 | ✔ | Figure 6: A case study from MathVista [36] with Qwen2.5-VL-72B [3].