2508.15202

# Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in Large Language Models > These authors contributed equally to this work and conducted the research during their internship at Qwen DianJin Team.Corresponding Author ## Abstract Process Reward Models (PRMs) have emerged as a promising framework for supervising intermediate reasoning in large language models (LLMs), yet existing PRMs are primarily trained on general or Science, Technology, Engineering, and Mathematics (STEM) domains and fall short in domain-specific contexts such as finance, where reasoning is more structured, symbolic, and sensitive to factual and regulatory correctness. We introduce Fin-PRM, a domain-specialized, trajectory-aware PRM tailored to evaluate intermediate reasoning steps in financial tasks. Fin-PRM integrates step-level and trajectory-level reward supervision, enabling fine-grained evaluation of reasoning traces aligned with financial logic. We apply Fin-PRM in both offline and online reward learning settings, supporting three key applications: (i) selecting high-quality reasoning trajectories for distillation-based supervised fine-tuning, (ii) providing dense process-level rewards for reinforcement learning, and (iii) guiding reward-informed Best-of-N inference at test time. Experimental results on financial reasoning benchmarks, including CFLUE and FinQA, demonstrate that Fin-PRM consistently outperforms general-purpose PRMs and strong domain baselines in trajectory selection quality. Downstream models trained with Fin-PRM yield substantial improvements with baselines, with gains of 12.9% in supervised learning, 5.2% in reinforcement learning, and 5.1% in test-time performance. These findings highlight the value of domain-specialized reward modeling for aligning LLMs with expert-level financial reasoning. Our project resources will be available at https://github.com/aliyun/qwen-dianjin. ## 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, leading to their increasing application in specialized domains such as finance (Yang, Liu, and Wang 2023; Zhu et al. 2025). However, financial reasoning tasks like financial statement analysis, investment strategy formulation, and regulatory compliance assessment demand a level of precision, factuality, and logical coherence that pushes the limits of current models. Therefore, a critical research direction is to align LLM reasoning pathways with expert cognitive processes monitoring tools, such as PRMs (Lightman et al. 2023a; Zhang et al. 2025; Setlur et al. 2024). PRMs selects the best one from multiple responses, often as part of test-time scaling strategies like Best-of-N (Khalifa et al. 2025; Liu et al. 2025), and gives scalarized reward signals to reinforcement learning progress (Zou et al. 2025; Cui et al. 2025). The central challenge for the successful implementation of PRM lies in the creation and curation of high-fidelity datasets, which must feature not only verified outcomes but also granular, step-by-step demonstrations of expert financial reasoning analysis (Lightman et al. 2023b; Zhang et al. 2025). We acknowledge the foundational work of OpenThoughts (Guha et al. 2025), which established a systematic methodology for reasoning data synthesis and validated its principles through comprehensive ablation studies. Following their successful data synthesis framework, we generate reasoning trajectories from a faithful data source(Placing our full trust in OpenThoughts’ recommendation, we have moved to a single data source), Cflue, a knowledge-based Chinese financial benchmark. Leveraging the advanced reasoning model, Deepseek-R1 (DeepSeek-AI et al. 2025), we obtained pairs of trace and solution and constructed a reasoning dataset in finance. Regarding the challenge of obtaining trustworthy reward signals, recent work has been greatly inspired by the effectiveness of LLM-as-a-Judge (Gu et al. 2025). While accept the advancement of LLM-as-a-Judge, we contend that relying solely on this method for the financial domain is often opaque and uninterpretable. We introduce a knowledge verification and verifiable regularization signal to aggregate reward labels, ensuring trained PRM is knowledge aware and interpretable. To enable PRM to effectively learn from the multi-dimensional scoring signals encompassing teacher model trajectory reasoning, large model assessment, and knowledge construction, we propose a novel dual-level training paradigm that simultaneously optimizes step-wise and trajectory-wise evaluation capabilities. Within this framework, we trained our model, named Fin-PRM, which provides reward signals at both the step and trajectory levels. Fin-PRM used a newly constructed, high-quality 3k financial reasoning dataset for training and show great awareness of step correctness and trajectory logic. We demonstrate the effectiveness of Fin-PRM by performing experimental validation on three methods: Offline data selection for SFT (Muennighoff et al. 2025; Xia et al. 2024), Best-of-N selection (Liu et al. 2025; Snell et al. 2024) and online reinforcement learning (Uesato et al. 2022; Cui et al. 2025). In summary, our primary contributions include: A High-Quality Financial Reasoning Dataset: We constructed and curated a new dataset of 3,000 samples, which provides granular, step-by-step reasoning traces with trustful reward label in the financial domain. A Novel Dual-Level Training Framework: We developed a training paradigm that fuses reward signals at both the step and trajectory levels, enabling the PRM to learn from multi-dimensional feedback and validate its key components through ablation studies. Comprehensive Experimental Validation: We demonstrated the effectiveness of Fin-PRM by successfully applying it to three distinct tasks—offline data selection, Best-of-N selection, and online reinforcement learning—proving its capability to enhance financial reasoning models ## 2 Related Work ### 2.1 Process Reward Models Process Reward Models (PRMs) have emerged as a crucial framework for providing step-level supervision and interpretable reward signals in complex reasoning tasks. State-of-the-art PRMs, exemplified by MathShepherd (Wang et al. 2024), Skywork-PRM (He et al. 2024), and Qwen2.5-Math-PRM (Zhang et al. 2025), employ human-annotated supervision with synthetic reward generation to deliver evaluation capabilities across diverse reasoning domains including mathematical problem solving, scientific analysis, and programming. Recent exploratory works such as ReasonFlux-PRM (Zou et al. 2025) combines both step-level and template-guided trace-level reward signals, Open-PRM (Zhang et al. 2024) leverages authoritative ORMs to reverse-engineer process-level supervision signals. In application, PRMs successfully integrated into Best-of-N sampling (Liu et al. 2025), offline data selection (Xie et al. 2023), and online reinforcement learning for model optimization (Bai et al. 2022). However, effective PRM evaluation should derive its reasoning assessment capabilities from concrete thinking trajectories rather than merely final solution correctness, and real-world vertical domain applications of PRMs impose critical requirements for deep domain knowledge mastery. Guided by these thoughts, we design a domain-spcialized framework that integrates trajectory-aware evaluation with expert knowledge validation, enabling more reliable process-level assessment for financal domain. ### 2.2 Data Synthesis for Reasoning Tasks. High-quality data has proven fundamental to developing effective reasoning models (Gunasekar et al. 2023). Early approaches focused on expanded existing datasets through rule-based transformations and template-driven generation (Wei and Zou 2019). These methods improving data coverage, but lacked the sophistication required for complex reasoning task. LLMs has enabled more advanced synthesis paradigms with distillation-based approaches leveraging powerful teacher models to generate high-quality reasoning traces for training efficient student models (Mukherjee et al. 2023). Notable contributions include WizardMath (Luo et al. 2025) synthesizes reasoning data through instruction evolution, MetaMath (Yu et al. 2024) generates diverse problems with step-by-step solutions, and OpenThoughts (Guha et al. 2025) establishes systematic methodological foundations for reasoning data synthesis through comprehensive ablation studies, providing empirical evidence for data composition principles. Considered that domain-specific applications demand specialized knowledge while maintaining reasoning quality standards, we adopt CFLUE (Zhu et al. 2024) as data source and Deepseek-R1 as teacher model to obtain reasoning trace. Recent advanced approaches employ LLM-as-a-judge for automated reward labeling, which always challenged by inherent limitations including black-box evaluation processes and insufficient reproducibility (Zheng et al. 2023). Building upon these observations, we enhance our reward annotation methodology by integrating expert-based knowledge verification mechanisms, significantly improves the observability and trustworthiness of reward signals, thereby ensuring more reliable evaluation capabilities for complex financial reasoning tasks. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Technical Framework for Fin-PRM Model ### Overview The image presents a technical architecture diagram for a fine-tuning process of a language model (Fin-PRM). It is divided into three interconnected sections: **Data Construction**, **Reward Design**, and **Training Integration**. The diagram uses color-coded components, arrows, and labels to illustrate data flow, evaluation criteria, and model training. --- ### Components/Axes #### Data Construction Section - **Dataset**: Contains three elements: - `question` - `option answer` - `analysis` - **Deepseek-R1**: A model processing the dataset. - **Qwen3**: Another model processing the dataset. - **Our dataset**: Output of the pipeline, containing: - `question` - `option answer` - `analysis` - `trace` (highlighted in orange) - `solution` (highlighted in orange) - `knowledge` (highlighted in blue) #### Reward Design Section - **Trace + Solution**: Combined input for trace labeling. - **Trace label**: Contains step-by-step labels (`step_label1` to `step_label5`). - **Reward Design**: Includes: - `MC evaluate` → `importance reward` - `LLM-as-a-judge` → `quality reward` - `knowledge check` → `accuracy reward` - `correctness check` → `correctness reward` - `knowledge coverage` → `knowledge reward` - **Expert Check**: Human evaluation step. - **Train**: Combines rewards into `Fin-PRM` using the formula: `L = λ · Lstep + μ · Ltraj` #### Spatial Layout - **Top-left**: Data Construction flowchart. - **Top-right**: Reward Design flowchart with step labels. - **Bottom**: Reward Design details with color-coded rewards and checks. --- ### Detailed Analysis 1. **Data Construction**: - Raw data (`question`, `option answer`, `analysis`) is processed by two models (Deepseek-R1 and Qwen3). - Outputs are merged into a structured dataset with additional fields: `trace`, `solution`, and `knowledge`. 2. **Reward Design**: - **Trace Labeling**: Steps 1–5 are annotated with labels (`step_label1`–`step_label5`). - **Reward Metrics**: - `importance reward`: Derived from Monte Carlo (MC) evaluation. - `quality reward`: Assigned by LLM-as-a-judge. - `accuracy reward`: Based on knowledge checks. - `correctness reward`: From correctness checks. - `knowledge reward`: From knowledge coverage analysis. - **Expert Check**: Human validation step before training. 3. **Training Integration**: - Rewards (`Lstep`) and trace labels (`Ltraj`) are combined using weighted coefficients (`λ` and `μ`) to train the Fin-PRM model. --- ### Key Observations - **Color Coding**: - Orange (`trace`, `solution`) and blue (`knowledge`) highlight critical components. - Green (`importance`, `quality`, `accuracy`, `knowledge`) and orange (`correctness`) rewards are distinct. - **Flow Direction**: - Data flows left-to-right (Dataset → Models → Our dataset). - Reward design flows top-to-bottom (Trace → Rewards → Training). - **Redundancy**: - `correctness check` appears in both step-level and trace-level reward paths. --- ### Interpretation This diagram illustrates a multi-stage pipeline for training a language model (Fin-PRM) with explicit reward signals. Key insights: 1. **Data Quality**: The integration of `trace` and `solution` into the dataset suggests a focus on process transparency and correctness. 2. **Reward Engineering**: Multiple reward signals (importance, quality, accuracy, etc.) ensure the model is evaluated holistically, balancing correctness, knowledge coverage, and human judgment. 3. **Human-in-the-Loop**: The `Expert Check` step implies manual validation to refine automated rewards. 4. **Training Objective**: The linear combination of `Lstep` (step-level rewards) and `Ltraj` (trace-level rewards) indicates a focus on both granular step quality and overall solution coherence. The framework emphasizes iterative refinement through expert feedback and multi-criteria evaluation, likely aiming to reduce hallucinations and improve factual accuracy in generated solutions. </details> Figure 1: Total process from data construction to model training. Fin-PRM used our dataset with both step-level and trajectory-level reward signals. ## 3 Financial Reasoning Dataset Construction Capability of the PRM fundamentally rely on the quality of its training data (Ye et al. 2025). In the financial domain, training data must not only feature correct outcomes but also capture the nuances of expert-level reasoning to process-based supervision (Lightman et al. 2023b). This section details our construction of Chain-of-thought dataset for financial reasoning. ### 3.1 Synthesizing Reasoning Trajectories We select Cflue (Zhu et al. 2024), an authoritative Chinese financial benchmark, as our basic data source. Cflue consists of a wide array of complex questions, accompanied by detailed analyses written by financial experts, which serves as a valuable source of ground-truth knowledge. Inspired by the systematic data synthesis framework proposed by OpenThoughts (Guha et al. 2025), we leverage Deepseek-R1 to generate structured reasoning traces. The model’s output is structured as a pair $(s,a)$ , where: - $s=(s_{1},s_{2},\dots,s_{T})$ represents the reasoning trace, a sequence of intermediate steps. - $a$ is the final, consolidated solution derived from the trace. Each triplet $(x,s,a)$ serves as a candidate for which we will later generate reward signals. ### 3.2 Financial Knowledge Extraction Financial reasoning is intensely knowledge-driven. To mitigate reward-hacking (Khalifa et al. 2025), we extract these knowledge as a new base, $\mathcal{K}$ , directly from the analysis of Cflue. We employ Qwen3-235b-a22b extract key financial terms and their corresponding explanations. For example, from an analysis discussing company valuation, the model might extract: - Term: Price-to-Book Ratio - Explanation: A financial ratio used to compare a company’s current market price to its book value. This knowledge base acts as the trusted external reference for the knowledge verification module during the reward annotation process, ensuring our reward signals are factually grounded. Based on Cflue dataset, our final dataset $\mathcal{D}$ consists of $(x,s,a,\mathcal{K},y,y_{analysis})$ , where $y$ is gold truth answer, $y_{analysis}$ is the expert analysis of the question. We treat $a$ as a sliver truth because the answer teacher model gives maybe wrong. ## 4 Fin-PRM: Domain-Specialized Process Reward Model In this section, we present the detailed architecture and training methodology of Fin-PRM. Our work is positioned to address the critical need for precise, factual, and interpretable evaluation of LLM reasoning in finance. We begin with a high-level overview of the Fin-PRM framework. Then we detail our method for handling step-level reward signals and trajectory-level signals. Subsequently, we describe our systematic process for reward data construction. Finally, we formulate the training objective that integrates these components. ### 4.1 Overview of Fin-PRM Fin-PRM is designed to serve as an expert evaluator for reasoning processes generated by LLMs in financial domain. It assesses the quality of a model’s thinking process, rather than just the final answer. To formalize this, we first define the core object of evaluation. Problem Formulation. The input for our reward model is a triplet $(x,s,a)$ . Its scoring function, $R_{\phi}$ , can be applied to evaluate either individual reasoning steps or the entire trajectory. When applied at the step level, Fin-PRM assesses the local correctness and utility of an individual reasoning step $s_{t}$ . The model produces a step-level reward conditioned on the full context: $$ R_{step}=R_{\phi}(s_{t}\mid x,s_{<t},a) \tag{1} $$ where $s_{<t}$ is the preceding reasoning history. This score addresses questions like, ”Is this calculation correct?” or ”Is this a logical inference?”. When applied at the trajectory level, Fin-PRM assesses the global coherence, logical flow, and strategic soundness of the entire reasoning trace $s$ . It produces a holistic score for the complete process: $$ R_{\mathrm{traj}}=R_{\phi}(s\mid x,a) \tag{2} $$ This score addresses higher-level questions, such as ”Is this the right overall approach to solve the problem?”. Training Objective. The goal of training is to learn the parameters $\phi$ of $R_{\phi}$ such that its predictions align with ground-truth reward signals derived from expert knowledge and verification. The objective is to minimize the discrepancy between the predicted and target rewards: $$ \min_{\phi}\mathrm{E}_{(x,s,a,\{r_{t}^{\mathrm{{}^{\prime}}}\})\sim\mathcal{D}}\left[\sum_{t=1}^{T}\mathcal{L}\left(R_{\phi}(s_{t}\mid x,s_{<t},a),r_{t}^{\mathrm{{}^{\prime}}}\right)\right] \tag{3} $$ where $\mathcal{D}$ is our training dataset, $r_{t}^{\mathrm{{}^{\prime}}}$ is the ground-truth reward, and $\mathcal{L}$ is loss function. ### 4.2 Step-level Reward Modeling To capture the multifaceted nature of a good reasoning step, we decompose our step-level reward into three distinct components: Monte Carlo estimation score $r_{\mathrm{importance}}$ , LLM-as-a-judge score $r_{\mathrm{qual}}$ , and an accuracy score $r_{\mathrm{acc}}$ that verifies its factual correctness. Importance Score $r_{\mathrm{importance}}$ : $r_{\mathrm{importance}}$ quantifies the utility of a step by evaluating its likelihood of being on a correct reasoning path. For each step $s_{t}$ in a trace, we prompt Qwen2.5-7b-math to generate $N$ (in our case, $N=8$ ) continuous rollouts until a final answer is reached. $r_{\mathrm{importance}}$ defined as the proportion of these rollouts that yield a correct final answer. This provides a soft-label score reflecting the potential of the current step, if soft-label is not $0$ , hard-label defined as $1$ . $$ r_{\mathrm{importance}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{I}(\xi(\mathrm{R}_{\mu,i}(s_{t}\mid x,s_{<t},y))) \tag{4} $$ where $\mathrm{R}_{\mu,i}(s_{t})$ is the $i$ -th generated completion starting from step $s_{t}$ , $\xi$ is the answer check progress and $\mathbf{I}(\cdot)$ is the indicator function, which returns 1 if the final answer of the rollout is correct and 0 otherwise. Qualitative Score $r_{\mathrm{qual}}$ : $r_{\mathrm{qual}}$ captures the abstract quality of a reasoning step. We leverage a powerful LLM, Qwen3-235b-a22b(we also considered chat-gpt-4.1, but observed almost the same score as Qwen3 gives), to evaluate each step $s_{t}$ from semantic coherence, logic soundness, and answer orientation, details of prompt can be found in appendix. $$ r_{\mathrm{qual}}=\mathrm{R}_{\mathrm{\theta}}(s_{t}\mid x,s_{<t},a) \tag{5} $$ where the score is prompt to be a scalar between 0 and 1. Against to prior works like Openthoughts and Reasonflux-PRM treat $a$ as a golden truth, we consider that reasoning data constructed for SFT is not suitable for PRM training. Accuracy Score $r_{\mathrm{acc}}$ : $r_{\mathrm{acc}}$ provides a robust, quantitative measure of a step’s factual and procedural correctness. It is organized into two parts as following exaplained, specifically designed to anchor the reward signal in ground truth $y$ and knowledge base $\mathcal{K}$ , aims to mitigate issues like LLM hallucination and reward hacking: Procedural Correctness ( $r_{\mathrm{step\_correctness}}$ ): This sub-score assesses the procedural validity of a given step $s_{t}$ . We employ a powerful LLM as a verifier, prompting it to make a binary assertion (1 for correct, 0 for incorrect) on whether the step $s_{t}$ constitutes a logically sound and relevant action towards reaching the known gold truth, $y$ . Here, the difference between $r_{\mathrm{step\_correctness}}$ and $r_{\mathrm{qual}}$ is that they use different base as their ground truth. Factual Accuracy ( $r_{\mathrm{knowledge\_acc}}$ ): This sub-score measures the factual accuracy of the content within $s_{t}$ . It systematically validates all identifiable claims and financial terms within the step against our knowledge base $\mathcal{K}$ . This directly counteracts model hallucination by ensuring that the reasoning is built upon verified facts from the trusted expert analysis, $y_{\mathrm{analysis}}$ . The final accuracy score combines these two facets in a weighted sum: $$ r_{\mathrm{acc}}=0.5(r_{\mathrm{step\_correctness}}(s_{t},y)+\omega_{k}\cdot r_{\mathrm{knowledge\_acc}}(s_{t},\mathcal{K}_{x})) \tag{6} $$ Here, the hyperparameter $\omega_{k}$ allows us to adjust the relative importance of factual grounding versus procedural correctness. A higher $\omega_{k}$ would more heavily penalize factual inaccuracies, aligning the model with a stricter standard of verifiability. In our experiments, we set $\omega_{k}=1.0$ , treating both types of correctness as equally critical. This composite score thereby ensures that highly-rated steps are both logically sound and factually impeccable. ### 4.3 Trajectory-level Reward Modeling Notice that a trajectory consists of correct steps sometimes lead to wrong answer, and PRMs can easily fall into reward hacking. We introduce trajectory-level reward signal combines two parts: an outcome-based correctness score $r_{\mathrm{out}}$ and a knowledge coverage score $r_{\mathrm{cover}}$ . Outcome correctness score $r_{\mathrm{out}}$ . $r_{\mathrm{out}}$ provides an assessment of the final answer’s correctness. For the tasks in our dataset typically require selecting a final option (e.g., ’A’, ’B’, ’ACD’), we extra model’s chosen option compared directly to the ground-truth correct option, yields a strict binary signal, $r_{\mathrm{out}}\in\{0,1\}$ . Knowledge coverage score $r_{\mathrm{cover}}$ . A high-quality reasoning process should be comprehensive and well-supported by relevant domain knowledge. $r_{\mathrm{cover}}$ measures the extent to which the reasoning trace $s$ and the final answer $a$ utilize the necessary knowledge terms, calculated as the ratio of relevant knowledge concepts mentioned in generation to the total number of concepts required. $$ r_{\mathrm{cover}}=\frac{|\phi_{\mathrm{ext}}(s\oplus a)\cap\mathcal{K}_{x}|}{|\mathcal{K}_{x}|} \tag{7} $$ Here, $\mathcal{K}_{x}\subseteq\mathcal{K}$ is the subset of our knowledge base containing all terms deemed relevant to the prompt $x$ . The function $\phi_{\mathrm{ext}}(\cdot)$ represents the extraction process, implemented through LLMs. $\oplus$ denotes string concatenation. ### 4.4 Reward Data Construction We construct ground-truth labels by aggregating the multiple signals into a single score for each granularity, which is then binarized. Step-level Label Construction. To form a single supervisory signal for each step $s_{t}$ , we aggregate its three distinct reward components—importance, quality, and accuracy—using a dynamic weighting scheme. This approach, based on the softmax function, adaptively emphasizes the score providing the strongest signal. The final continuous score for step $t$ , denoted $r_{t}^{\mathrm{step}}$ , is calculated as: $$ r_{t}^{\mathrm{step}}=\sum_{k\in\{\mathrm{imp},\mathrm{qual},\mathrm{acc}\}}\mathrm{softmax}\big{(}r_{t}^{\mathrm{imp}},r_{t}^{\mathrm{qual}},r_{t}^{\mathrm{acc}}\big{)}_{k}\cdot r_{t}^{k} \tag{8} $$ In this formulation, $r_{t}^{k}$ is the raw score for component $k$ at step $t$ . The $\mathrm{softmax}$ function converts the vector of raw scores into a probability distribution, which serves as a set of dynamic weights. The $k$ -th element of this distribution, indicated by the subscript $(\cdot)_{k}$ , is then multiplied by its corresponding raw score $r_{t}^{k}$ . This method functions as an attention mechanism: a score that is significantly higher than the others will receive a proportionally larger weight in the final sum, allowing its signal to dominate. For instance, a step with exceptionally high factual accuracy ( $r_{t}^{\mathrm{acc}}$ ) will have its contribution amplified, even if other scores are moderate. This is more robust than a fixed-weight average. Finally, this aggregated score $r_{t}^{\mathrm{step}}$ is binarized using a 0.5 threshold to produce the final step label, $L_{t}^{\mathrm{step}}$ . Trajectory-level Label Construction. For each trajectory, we combine its outcome and coverage scores into a single score, $S_{\mathrm{traj}}$ : $$ R_{\mathrm{traj}}(s,a)=r_{\mathrm{out}}(a)+\eta\cdot r_{\mathrm{cover}}(s,a) \tag{9} $$ Here, $\eta$ are the weights for outcome correctness and knowledge coverage. We set $\eta$ to 1.5. This weighting ensures that the knowledge coverage score has a meaningful impact on the final label. The trajectory score is also converted to a binary label, $L_{\mathrm{traj}}$ , using a 1.25 threshold, we select these weights to balance the contribution of these two signals, by using the mean value of their weights as the threshold value, we give the ability to change the hard label to each reward signal. ### 4.5 Training Objective To train Fin-PRM effectively, we formulate a joint objective to train model through binary cross-entropy (BCE), learning to predict the correctness of both individual steps and entire trajectories. The total loss $\mathcal{L}_{\mathrm{total}}$ : $$ \mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{step}}+\lambda\cdot\mathcal{L}_{\mathrm{traj}} \tag{10} $$ where $\lambda$ are hyperparameters that balance the contribution of each supervision signal. The step-level loss, $\mathcal{L}_{\mathrm{step}}$ , is the average loss over all steps in a reasoning trace. It measures the discrepancy between the model’s prediction and the ground-truth step label, $L_{\mathrm{step}}(s_{t})$ : $$ \mathcal{L}_{\mathrm{step}}=\frac{1}{T}\sum_{t=1}^{T}\mathcal{L}_{\mathrm{BCE}}\Big{(}R_{\phi}(s_{t}\mid x,s_{<t},a),R_{step}\Big{)} \tag{11} $$ The trajectory-level loss, $\mathcal{L}_{\mathrm{traj}}$ , follows the same principle. It compares the model’s prediction for the entire trajectory against the ground-truth trajectory label, $L_{\mathrm{traj}}(s,a)$ : $$ \mathcal{L}_{\mathrm{traj}}=\mathcal{L}_{\mathrm{BCE}}\Big{(}R_{\phi}(s,a\mid x),R_{traj}\Big{)} \tag{12} $$ where $\sigma(\cdot)$ denotes the sigmoid function, which converts the model’s raw logit outputs into probabilities. $\mathcal{L}_{\mathrm{BCE}}(\cdot,\cdot)$ denotes the BCE loss function. For a ground-truth label $L\in\{0,1\}$ and a model logit output $R_{\phi}$ , it is defined as $\mathcal{L}_{\mathrm{BCE}}(R_{\phi},L)=-[L\log\sigma(R_{\phi})+(1-L)\log(1-\sigma(R_{\phi}))]$ , By jointly optimizing this objective, Fin-PRM is trained to make judgments. ## 5 Applications of Fin-PRM To validate the effectiveness of our framework and the capability of Fin-PRM, we apply three critical use cases and compare its performance against relevant baselines. Supervised Fine-tuning with Data Selection: Using Fin-PRM as an offline filter to curate a high-quality dataset for more efficient and effective SFT. Reward-guided Test-Time Scaling: Employing Fin-PRM at inference time to select the best response from multiple candidates in a Best-of-N (BoN) setting. Online Reward Modeling: Applying Fin-PRM as a reward function to guide the policy optimization through reinforcement learning. | Method Accuracy (%) | Qwen2.5-7b-instruct 45.3 | Random Selection 43.8 | Math-PRM-7B 56.5 | Math-PRM-72B 57.1 | Fin-PRM (Ours) 58.2 | | --- | --- | --- | --- | --- | --- | Table 1: Offline data selection comparison on the CFLUE benchmark. All SFT methods use 1,000 selected samples to fine-tune the Qwen2.5-7B-Instruct base model. The highest performance is in bold. ### 5.1 Supervised Fine-tuning with Data Selection PRM can be used to identify and select only the most coherent and correct examples from the whole dataset according to the score it gives. Through comparing the capability of student model trained by different selected dataset examples, advancement of Fin-PRM be proved. We use Qwen3-8B to produce multiple distinct reasoning trajectories for each question, scoring each trajectory through Fin-PRM by the following function: $$ \hat{R}=\frac{1}{T}\sum_{t=1}^{T}\hat{R}_{\mathrm{step}}(s_{t}\mid x,s_{<t},a)+\zeta\cdot\hat{R}_{\mathrm{traj}}(s,a\mid x) \tag{13} $$ Here, $\hat{R}_{\mathrm{step}}$ and $\hat{R}_{\mathrm{traj}}$ are the reward scores generated by Fin-PRM or other PRMs. The hyperparameter $\zeta$ balances the contribution of fine-grained step correctness and overall trajectory quality; we found $\zeta=1.0$ to work well in practice. As shown in Table 1, the selection strategy has a profound impact on model performance. Fine-tuning on randomly selected data degrades performance to 43.8%, highlighting the risk of using noisy synthetic data. In contrast, all PRM-based selection methods provide a substantial boost over the 45.3% base model accuracy. Our domain-specialized Fin-PRM achieves the highest accuracy of 58.2%. This result not only demonstrates a 12.9-point improvement over the base model but also confirms that a finance-aware reward model is superior for curating high-quality reasoning data in this domain. <details> <summary>AAAI_BoN.png Details</summary>  ### Visual Description ## Line Graphs: Accuracy vs. Number of Samples (N) for CFLUE and MATH500 ### Overview The image contains two line graphs comparing the accuracy of different models (PASS@N, Fin-PRM, Maj@N, Qwen2.5-MATH-PRM-7B, Qwen2.5-MATH-PRM-72B) across two datasets: **CFLUE** (top graph) and **MATH500** (bottom graph). The x-axis represents the number of samples (N) on a logarithmic scale (2⁰ to 2⁴), and the y-axis represents accuracy in percentage. Each model is represented by a distinct line style and color, as defined in the legend. --- ### Components/Axes - **X-axis**: "Number of Samples (N)" with ticks at 2⁰ (1), 2¹ (2), 2² (4), 2³ (8), 2⁴ (16). - **Y-axis**: "Accuracy (%)" with ranges: - **CFLUE**: 55% to 80% - **MATH500**: 55% to 70% - **Legends**: - **CFLUE**: - Red dashed line: PASS@N - Black solid line: Fin-PRM - Brown dashed line: Maj@N - Blue dashed line: Qwen2.5-MATH-PRM-7B - Green dashed line: Qwen2.5-MATH-PRM-72B - **MATH500**: Same legend as CFLUE. --- ### Detailed Analysis #### **CFLUE Graph** 1. **PASS@N (Red Dashed Line)**: - Starts at ~58% (2⁰) and increases steadily to ~83% (2⁴). - **Trend**: Strong upward slope, indicating consistent improvement with more samples. 2. **Fin-PRM (Black Solid Line)**: - Starts at ~58% (2⁰), rises to ~68% (2¹), then plateaus with minor fluctuations (~66–68%) at 2²–2⁴. - **Trend**: Initial sharp increase, followed by stabilization. 3. **Maj@N (Brown Dashed Line)**: - Starts at ~58% (2⁰), drops to ~54% (2¹), then rises to ~63% (2⁴). - **Trend**: Initial dip, followed by gradual recovery. 4. **Qwen2.5-MATH-PRM-7B (Blue Dashed Line)**: - Starts at ~58% (2⁰), peaks at ~62% (2¹), then declines to ~58% (2⁴). - **Trend**: Initial rise, followed by a decline. 5. **Qwen2.5-MATH-PRM-72B (Green Dashed Line)**: - Starts at ~58% (2⁰), rises to ~57% (2¹), then fluctuates between ~57–60% (2²–2⁴). - **Trend**: Slight improvement but no significant growth. #### **MATH500 Graph** 1. **PASS@N (Red Dashed Line)**: - Starts at ~60% (2⁰), increases to ~68% (2⁴). - **Trend**: Steady upward trajectory. 2. **Fin-PRM (Black Solid Line)**: - Starts at ~60% (2⁰), rises to ~63% (2³), then plateaus at ~63% (2⁴). - **Trend**: Gradual improvement with stabilization. 3. **Maj@N (Brown Dashed Line)**: - Starts at ~60% (2⁰), drops to ~57% (2¹), then rises to ~64% (2⁴). - **Trend**: Initial dip, followed by recovery. 4. **Qwen2.5-MATH-PRM-7B (Blue Dashed Line)**: - Starts at ~60% (2⁰), rises to ~64% (2⁴). - **Trend**: Consistent upward trend. 5. **Qwen2.5-MATH-PRM-72B (Green Dashed Line)**: - Starts at ~60% (2⁰), rises to ~66% (2³), then plateaus at ~66% (2⁴). - **Trend**: Strong improvement, followed by stabilization. --- ### Key Observations 1. **PASS@N** consistently outperforms other models in both datasets, showing the most significant improvement with increased samples. 2. **Maj@N** exhibits a temporary dip at 2¹ in both graphs but recovers by 2⁴. 3. **Qwen2.5-MATH-PRM-72B** (green dashed line) performs best in **MATH500**, surpassing other models at higher sample sizes. 4. **Qwen2.5-MATH-PRM-7B** (blue dashed line) shows mixed performance: a decline in CFLUE but steady growth in MATH500. 5. **Fin-PRM** (black solid line) demonstrates stability in both datasets but lacks the growth seen in PASS@N. --- ### Interpretation - **PASS@N** is the most effective model for improving accuracy with more samples, suggesting it is optimized for scalability. - **Qwen2.5-MATH-PRM-72B** excels in **MATH500**, indicating its architecture is particularly suited for mathematical reasoning tasks. - The **Maj@N** dip at 2¹ may reflect overfitting or inefficiencies at smaller sample sizes, which are mitigated at larger scales. - **Qwen2.5-MATH-PRM-7B** underperforms in CFLUE but shows promise in MATH500, highlighting dataset-specific model effectiveness. - All models improve with more samples, but the rate of improvement varies, emphasizing the importance of model selection based on task requirements. --- ### Spatial Grounding & Color Verification - **Legends**: Positioned at the top of each graph, with colors matching the corresponding lines (e.g., red dashed = PASS@N). - **Data Points**: Confirmed alignment with legend colors (e.g., green dashed line = Qwen2.5-MATH-PRM-72B). --- ### Final Notes The graphs demonstrate that **PASS@N** and **Qwen2.5-MATH-PRM-72B** are the most scalable models, with performance gains directly tied to the number of samples. The data underscores the need for task-specific model selection, as performance varies significantly across datasets. </details> Figure 2: BoN test on Cflue dataset. Fin-PRM is the black line, outperforms all baselines in financial domain. Qwen2.5-Math-PRM-7B shows performance below major-voting when the number of N becomes larger, points crucial needs for domain-specific PRM. The figure below shows that Fin-PRM performs almost identically to proprietary mathematical PRM of the same size in the field of mathematics. ### 5.2 Test-Time Scaling Another key application is to enhance model performance at inference time through Best-of-N (BoN) selection. This method computes the score from Equation 13 for N candidate answers generated by a policy model to find the best one. We evaluate performance in our target financial domain. Using Qwen2.5-7B-Instruct as the generator, we perform BoN selection on a 1,000-sample subset of the CFLUE test set for N values of 4, 8, and 16. Fin-PRM is compared against two baselines: a strong, general-domain Qwen2.5-Math-PRM-7B and major voting method. As presented in Figure 2, in the Cflue BoN test, Fin-PRM consistently leads to greater accuracy gains as N increases, outperforming the majority-voting baseline by more than $5.1\$ at N=16. This result highlights its effectiveness for financial reasoning. To validate the generalization of Fin-PRMs, we conduct the same BoN experiment on an out-of-domain benchmark, Math500. | Majority Voting Fin-PRM (Ours) Qwen2.5-Math-PRM-7B | 60.0 60.0 60.0 | 56.6 60.5 60.4 | 61.0 61.6 62.0 | 63.0 63.1 63.2 | 63.3 63.0 63.8 | | --- | --- | --- | --- | --- | --- | | Qwen2.5-Math-PRM-72B | 60.0 | 62.5 | 63.1 | 65.6 | 65.8 | Table 2: Best-of-N performance on the out-of-domain Math500 benchmark. Fin-PRM shows better performance than major voting. Best selector performance is in bold. The results on Math500, shown in Table 2, Fin-PRM demonstrates a respectable baseline capability, proving it does not completely fail on unfamiliar tasks. Its performance closely tracks that of the 7B-scale math-specialized PRM and remains competitive with majority voting. This indicates that while its expert knowledge is sharply honed for finance, Fin-PRM retains a foundational ability to assess logical structure, allowing it to function as a competent generalist evaluator in other domains and demonstrating robust generalization. ### 5.3 Reward signals for RL training Beyond offline data curation and test-time selection, Fin-PRM’s most dynamic application is providing a composite reward that guides policy optimization through step-aware supervision. We integrate Fin-PRM into the Group Relative Policy Optimization (Shao et al. 2024) (GRPO) framework. By default, GRPO optimizes for the outcome-level reward, which in our case is $r_{\mathrm{out}}$ . To incorporate the nuanced, process-level supervision from Fin-PRM, we augment this reward with our holistic score $\hat{R}$ (from Equation 13). The new composite reward for a given trace $(s,a)$ is defined as: $$ r_{\mathrm{rl}}=(1-\delta)\cdot r_{\mathrm{out}}+\delta\cdot\hat{R} \tag{14} $$ where the hyperparameter $\delta$ controls the relative weight of the process-level reward. For a group of $N$ responses, the advantage is: $$ A_{\mathrm{rl}}=\frac{r_{\mathrm{rl}}-\mathrm{mean}(\{r_{\mathrm{rl}}\}_{j=1}^{N})}{\mathrm{std}(\{r_{\mathrm{rl}}\}_{j=1}^{N})} \tag{15} $$ With the Fin-PRM derived advantage term, $A_{\mathrm{comp}}$ , the GRPO objective is updated to: $$ \begin{array}[]{l}\mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathrm{E}_{x,\{s_{i}\}\sim\pi_{\theta_{\mathrm{old}}}}\Bigg{[}\frac{1}{N}\sum_{i=1}^{N}\frac{1}{T_{i}}\sum_{t=1}^{T_{i}}\Big{(}\\ \qquad\min\Big{\{}\frac{\pi_{\theta}(s_{i,t}|x,s_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(s_{i,t}|x,s_{i,<t})}A_{rl,i},\\ \qquad\quad\mathrm{clip}\big{(}\frac{\pi_{\theta}(s_{i,t}|x,s_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(s_{i,t}|x,s_{i,<t})},1-\epsilon,1+\epsilon\big{)}A_{rl,i}\Big{\}}\\ \qquad-\beta_{\mathrm{KL}}\mathcal{D}_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})\Big{)}\Bigg{]}\end{array} \tag{16} $$ where $A_{rl,i}$ is the advantage for the $i$ -th sample, $\epsilon$ is the clipping hyperparameter, and the term weighted by $\beta_{\mathrm{KL}}$ is a KL-divergence penalty against a reference policy $\pi_{\mathrm{ref}}$ . <details> <summary>AAAI_online.png Details</summary>  ### Visual Description ## Line Graph: Algorithm Accuracy Comparison Over Steps ### Overview The image is a line graph comparing the accuracy of three algorithms—FIN-PRM, GRPO(Rule-Based), and Gwen2.5-Math-PRM-7B—across 180 incremental steps. The y-axis represents accuracy (0.35–0.60), and the x-axis represents steps (0–180). The legend is positioned in the top-left corner, with distinct colors for each algorithm: blue (FIN-PRM), orange (GRPO), and green (Gwen2.5). ### Components/Axes - **X-axis (Step)**: Labeled "Step," ranging from 0 to 180 in increments of 20. - **Y-axis (Accuracy)**: Labeled "Accuracy," ranging from 0.35 to 0.60 in increments of 0.05. - **Legend**: Top-left corner, with color-coded labels: - Blue circle: FIN-PRM - Orange square: GRPO(Rule-Based) - Green triangle: Gwen2.5-Math-PRM-7B ### Detailed Analysis 1. **FIN-PRM (Blue Line)**: - Starts at ~0.42 at step 0. - Gradually increases, peaking at ~0.58 around step 100. - Stabilizes between ~0.55–0.58 from step 120 onward. - Notable fluctuations: Dips slightly at step 60 (~0.50) and step 140 (~0.56). 2. **GRPO(Rule-Based) (Orange Line)**: - Begins at ~0.40 at step 0. - Rises steadily to ~0.54 by step 100. - Experiences minor declines after step 120, stabilizing at ~0.52–0.54 by step 180. - Sharp drop at step 150 (~0.52) followed by recovery. 3. **Gwen2.5-Math-PRM-7B (Green Line)**: - Starts at ~0.42 at step 0. - Peaks at ~0.48 around step 50. - Declines sharply to ~0.38 at step 80. - Remains flat between ~0.40–0.42 from step 100 onward. ### Key Observations - **FIN-PRM** consistently outperforms the other algorithms, particularly after step 100. - **GRPO** shows moderate improvement but lags behind FIN-PRM, with a notable dip at step 150. - **Gwen2.5** exhibits the most volatility, with a sharp decline after step 50 and no recovery. ### Interpretation The data suggests that **FIN-PRM** is the most robust algorithm, maintaining high accuracy across all steps. **GRPO** demonstrates steady but suboptimal performance, while **Gwen2.5**’s early peak and subsequent decline indicate potential instability or overfitting in later stages. The divergence between FIN-PRM and Gwen2.5 after step 100 highlights differences in algorithmic efficiency or adaptability. The GRPO dip at step 150 may reflect a specific challenge or limitation in its rule-based framework. Overall, FIN-PRM’s sustained performance makes it the preferred choice for this task. </details> Figure 3: Performance of GRPO policy optimization using different reward signals. The policy model is Qwen2.5-7B-Instruct. We report mean accuracy over multiple runs. Figure 3 presents the downstream reasoning performance after using different reward signals for GRPO policy optimization. We use Qwen2.5-7B-Instruct as the policy model and compare three reward sources: a rule-based signal using only $r_{\mathrm{out}}$ , the general-domain Qwen2.5-Math-PRM-7B, and our Fin-PRM. Across all evaluations, using Fin-PRM as the reward source consistently yields the best-performing policy. Integrating Fin-PRM boosts performance on CFLUE to 70.5% and on FinQA to 62.8%, a significant gain of 3.3 points on both benchmarks compared to the rule-based heuristic. Crucially, Fin-PRM also outperforms the strong Qwen2.5-Math-PRM-7B baseline, highlighting that the domain-specific, factually-grounded process supervision it provides is more effective for guiding RL in a specialized domain than a general-purpose reward model. These results demonstrate that the high-quality learned reward signals from Fin-PRM substantially enhance policy optimization, leading to more capable financial reasoning models. <details> <summary>AAAI_Ablo.png Details</summary>  ### Visual Description ## Line Graph: Accuracy vs ζ Value ### Overview The image displays a line graph comparing the accuracy of four distinct data series (labeled 2, 4, 8, 16) across a range of ζ values (0.00 to 2.00). The y-axis represents accuracy (0.62–0.68), while the x-axis represents ζ values. Four data series are differentiated by color and symbol: blue circles (2), orange squares (4), green triangles (8), and red crosses (16). The legend is positioned in the top-left corner. ### Components/Axes - **X-axis**: ζ value (0.00–2.00, increments of 0.25). - **Y-axis**: Accuracy (0.62–0.68, increments of 0.01). - **Legend**: Top-left corner, with four entries: - Blue circles: "2" - Orange squares: "4" - Green triangles: "8" - Red crosses: "16" ### Detailed Analysis 1. **Blue Line (2)**: - **Trend**: Relatively flat with minor fluctuations. - **Values**: Consistently hovers between **0.63–0.635** accuracy across all ζ values. - **Key Point**: Minimal variability, suggesting stable but low performance. 2. **Orange Line (4)**: - **Trend**: Slight upward trend with minor oscillations. - **Values**: Ranges from **0.64–0.65** accuracy, peaking near ζ=1.00. - **Key Point**: Moderate improvement over the blue line but limited overall growth. 3. **Green Line (8)**: - **Trend**: Steady increase until ζ≈1.00, followed by a plateau. - **Values**: Rises from **0.65–0.665** (ζ=0.00–1.00), then stabilizes near **0.67** (ζ=1.00–2.00). - **Key Point**: Highest sustained accuracy after ζ=1.00, with minimal decline. 4. **Red Line (16)**: - **Trend**: Highly variable, with sharp peaks and troughs. - **Values**: Peaks at **0.68** (ζ≈1.00), then declines to **0.66** by ζ=2.00. - **Key Point**: Highest accuracy at ζ=1.00 but unstable performance overall. ### Key Observations - The red line (16) achieves the highest accuracy (**0.68**) at ζ=1.00 but exhibits significant volatility. - The green line (8) demonstrates the most stable high performance, maintaining **0.67** accuracy after ζ=1.00. - The blue line (2) remains the least effective, with accuracy capped at **0.635**. - All lines show diminishing returns or instability beyond ζ=1.00, except the green line (8), which plateaus. ### Interpretation The data suggests a trade-off between accuracy and stability as ζ increases: - **ζ=16** (red line) achieves peak accuracy but at the cost of high variability, indicating potential overfitting or sensitivity to input changes. - **ζ=8** (green line) balances accuracy and stability, maintaining consistent performance after ζ=1.00, making it the most reliable choice. - **ζ=2** (blue line) underperforms across all ζ values, suggesting insufficient complexity or model capacity. The trends imply that higher ζ values may initially improve accuracy but require careful tuning to avoid instability. The green line (8) represents an optimal balance, while the red line (16) highlights the risks of over-optimization. </details> Figure 4: Ablation study on the ranking score weight $\zeta$ . The plot shows Best-of-N accuracy on the CFLUE test set as a function of $\zeta$ for different numbers of candidates (N). Fin-PRM’s best performance appears in $\zeta$ = 1 ## 6 Ablation Study To better understand the contributions of different components in our framework, we conduct a series of ablation studies on key hyperparameters. ### 6.1 Ablation on Ranking Score Weighting As described in Equation 13, the hyperparameter $\zeta$ controls the balance between the aggregated step-level reward ( $\hat{R}_{\mathrm{step}}$ ) and the trajectory-level reward ( $\hat{R}_{\mathrm{traj}}$ ) when calculating the final score for ranking candidate solutions. To assess the impact of this balance, we conduct an ablation study by varying $\zeta$ and observing its effect on Best-of-N (BoN) selection performance. ### 6.2 Experimental Setup. We perform BoN selection on our 1,000-sample CFLUE test set, using the same fine-tuned Qwen2.5-7B-Instruct model as the generator. For N values of 2, 4, 8, and 16, we vary $\zeta$ across the range [0.0, 2.0] and plot the resulting accuracy. ### 6.3 Results and Analysis. As shown in Figure 4, the model’s performance is sensitive to the value of $\zeta$ . For larger N (specifically N=8 and N=16), we observe a clear and consistent trend: accuracy rises as $\zeta$ increases from 0, reaches a peak near $\zeta=1.0$ , and then gradually declines. This pattern reveals several key insights: - When $\zeta=0$ , the ranking relies solely on step-level rewards. While this performs reasonably well, it is consistently suboptimal, indicating that step-level correctness alone is insufficient. - Performance peaks at $\zeta\approx 1.0$ , where step-level and trajectory-level rewards are given roughly equal importance. This suggests that the most effective evaluation considers both the granular correctness of individual steps and the holistic quality of the entire reasoning path. - As $\zeta$ becomes very large, performance degrades. This implies that over-relying on the trajectory score while ignoring step-level details leads to poorer selection, likely because the model may select trajectories that seem plausible overall but contain critical local flaws. These results strongly validate our dual-granularity reward design, demonstrating that a balanced integration of both local and global signals is essential for accurately identifying superior reasoning processes. More ablation study about each parameters we used in paper can be found in appendix. ## 7 Discussion Our experimental results across SFT, Best-of-N, and RL applications consistently demonstrate that Fin-PRM outperforms general-purpose baselines. This success validates our central thesis: for high-stakes domains like finance, effective process supervision requires a reward model that is not just logically coherent but deeply specialized and factually grounded. The key to Fin-PRM’s performance is its dual-level, knowledge-aware architecture. By integrating verifiable reward components ( $r_{\mathrm{acc}}$ and $r_{\mathrm{cover}}$ ) grounded in an expert-derived knowledge base, Fin-PRM moves beyond assessing mere logical plausibility to penalizing factual hallucinations. This confirms that for domains where truth is non-negotiable, a hybrid approach combining LLM-based qualitative assessment with explicit knowledge verification is critical. While this framework provides a robust proof-of-concept, we acknowledge several limitations that open important avenues for future research: the construction of our 3k-sample dataset, while high-quality, was resource-intensive. To make domain-specific PRMs more accessible, future work should explore efficient, semi-automated methods for generating and annotating such specialized data at scale. Meanwhile, our knowledge base $\mathcal{K}$ is static, which poses a risk in a dynamic field like finance where regulations and market conditions evolve. Integrating dynamic knowledge sources, such as real-time financial news feeds or regulatory update APIs, would be a critical enhancement to prevent knowledge decay and ensure long-term reliability. We select the hyperparameters in our work through ablation study, which were set fixed. A more advanced approach could involve learning these weights dynamically, perhaps through a meta-learning framework. This would allow the model to adapt the reward composition to the specific demands of each problem, potentially yielding further performance gains. Addressing these challenges is key to moving from a specialized model to an adaptive, continuously improving financial reasoning expert. ## 8 Conclusion In this work, we introduced Fin-PRM, demonstrating that a domain-specialized, knowledge-aware process reward model significantly enhances financial reasoning. We believe this framework serves as a blueprint for developing trustworthy AI evaluators in other high-stakes fields like law and medicine, promoting a shift towards a portfolio of specialized, reliable models. Our work provides a solid foundation for building the next generation of LLMs capable of expert-level reasoning in specialized, real-world applications. ## References - Bai et al. (2022) Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; Joseph, N.; Kadavath, S.; Kernion, J.; Conerly, T.; El-Showk, S.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Hume, T.; Johnston, S.; Kravec, S.; Lovitt, L.; Nanda, N.; Olsson, C.; Amodei, D.; Brown, T.; Clark, J.; McCandlish, S.; Olah, C.; Mann, B.; and Kaplan, J. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. - Cui et al. (2025) Cui, G.; Yuan, L.; Wang, Z.; Wang, H.; Li, W.; He, B.; Fan, Y.; Yu, T.; Xu, Q.; Chen, W.; Yuan, J.; Chen, H.; Zhang, K.; Lv, X.; Wang, S.; Yao, Y.; Han, X.; Peng, H.; Cheng, Y.; Liu, Z.; Sun, M.; Zhou, B.; and Ding, N. 2025. Process Reinforcement through Implicit Rewards. arXiv:2502.01456. - DeepSeek-AI et al. (2025) DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; Zhang, X.; Yu, X.; Wu, Y.; Wu, Z. F.; Gou, Z.; Shao, Z.; Li, Z.; Gao, Z.; Liu, A.; Xue, B.; Wang, B.; Wu, B.; Feng, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; Dai, D.; Chen, D.; Ji, D.; Li, E.; Lin, F.; Dai, F.; Luo, F.; Hao, G.; Chen, G.; Li, G.; Zhang, H.; Bao, H.; Xu, H.; Wang, H.; Ding, H.; Xin, H.; Gao, H.; Qu, H.; Li, H.; Guo, J.; Li, J.; Wang, J.; Chen, J.; Yuan, J.; Qiu, J.; Li, J.; Cai, J. L.; Ni, J.; Liang, J.; Chen, J.; Dong, K.; Hu, K.; Gao, K.; Guan, K.; Huang, K.; Yu, K.; Wang, L.; Zhang, L.; Zhao, L.; Wang, L.; Zhang, L.; Xu, L.; Xia, L.; Zhang, M.; Zhang, M.; Tang, M.; Li, M.; Wang, M.; Li, M.; Tian, N.; Huang, P.; Zhang, P.; Wang, Q.; Chen, Q.; Du, Q.; Ge, R.; Zhang, R.; Pan, R.; Wang, R.; Chen, R. J.; Jin, R. L.; Chen, R.; Lu, S.; Zhou, S.; Chen, S.; Ye, S.; Wang, S.; Yu, S.; Zhou, S.; Pan, S.; Li, S. S.; Zhou, S.; Wu, S.; Ye, S.; Yun, T.; Pei, T.; Sun, T.; Wang, T.; Zeng, W.; Zhao, W.; Liu, W.; Liang, W.; Gao, W.; Yu, W.; Zhang, W.; Xiao, W. L.; An, W.; Liu, X.; Wang, X.; Chen, X.; Nie, X.; Cheng, X.; Liu, X.; Xie, X.; Liu, X.; Yang, X.; Li, X.; Su, X.; Lin, X.; Li, X. Q.; Jin, X.; Shen, X.; Chen, X.; Sun, X.; Wang, X.; Song, X.; Zhou, X.; Wang, X.; Shan, X.; Li, Y. K.; Wang, Y. Q.; Wei, Y. X.; Zhang, Y.; Xu, Y.; Li, Y.; Zhao, Y.; Sun, Y.; Wang, Y.; Yu, Y.; Zhang, Y.; Shi, Y.; Xiong, Y.; He, Y.; Piao, Y.; Wang, Y.; Tan, Y.; Ma, Y.; Liu, Y.; Guo, Y.; Ou, Y.; Wang, Y.; Gong, Y.; Zou, Y.; He, Y.; Xiong, Y.; Luo, Y.; You, Y.; Liu, Y.; Zhou, Y.; Zhu, Y. X.; Xu, Y.; Huang, Y.; Li, Y.; Zheng, Y.; Zhu, Y.; Ma, Y.; Tang, Y.; Zha, Y.; Yan, Y.; Ren, Z. Z.; Ren, Z.; Sha, Z.; Fu, Z.; Xu, Z.; Xie, Z.; Zhang, Z.; Hao, Z.; Ma, Z.; Yan, Z.; Wu, Z.; Gu, Z.; Zhu, Z.; Liu, Z.; Li, Z.; Xie, Z.; Song, Z.; Pan, Z.; Huang, Z.; Xu, Z.; Zhang, Z.; and Zhang, Z. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. - Gu et al. (2025) Gu, J.; Jiang, X.; Shi, Z.; Tan, H.; Zhai, X.; Xu, C.; Li, W.; Shen, Y.; Ma, S.; Liu, H.; Wang, S.; Zhang, K.; Wang, Y.; Gao, W.; Ni, L.; and Guo, J. 2025. A Survey on LLM-as-a-Judge. arXiv:2411.15594. - Guha et al. (2025) Guha, E.; Marten, R.; Keh, S.; Raoof, N.; Smyrnis, G.; Bansal, H.; Nezhurina, M.; Mercat, J.; Vu, T.; Sprague, Z.; Suvarna, A.; Feuer, B.; Chen, L.; Khan, Z.; Frankel, E.; Grover, S.; Choi, C.; Muennighoff, N.; Su, S.; Zhao, W.; Yang, J.; Pimpalgaonkar, S.; Sharma, K.; Ji, C. C.-J.; Deng, Y.; Pratt, S.; Ramanujan, V.; Saad-Falcon, J.; Li, J.; Dave, A.; Albalak, A.; Arora, K.; Wulfe, B.; Hegde, C.; Durrett, G.; Oh, S.; Bansal, M.; Gabriel, S.; Grover, A.; Chang, K.-W.; Shankar, V.; Gokaslan, A.; Merrill, M. A.; Hashimoto, T.; Choi, Y.; Jitsev, J.; Heckel, R.; Sathiamoorthy, M.; Dimakis, A. G.; and Schmidt, L. 2025. OpenThoughts: Data Recipes for Reasoning Models. arXiv:2506.04178. - Gunasekar et al. (2023) Gunasekar, S.; Zhang, Y.; Aneja, J.; Mendes, C. C. T.; Giorno, A. D.; Gopi, S.; Javaheripi, M.; Kauffmann, P.; de Rosa, G.; Saarikivi, O.; Salim, A.; Shah, S.; Behl, H. S.; Wang, X.; Bubeck, S.; Eldan, R.; Kalai, A. T.; Lee, Y. T.; and Li, Y. 2023. Textbooks Are All You Need. arXiv:2306.11644. - He et al. (2024) He, J.; Wei, T.; Yan, R.; Liu, J.; Wang, C.; Gan, Y.; Tu, S.; Liu, C. Y.; Zeng, L.; Wang, X.; Wang, B.; Li, Y.; Zhang, F.; Xu, J.; An, B.; Liu, Y.; and Zhou, Y. 2024. Skywork-o1 Open Series. https://huggingface.co/Skywork. - Khalifa et al. (2025) Khalifa, M.; Agarwal, R.; Logeswaran, L.; Kim, J.; Peng, H.; Lee, M.; Lee, H.; and Wang, L. 2025. Process Reward Models That Think. arXiv:2504.16828. - Lightman et al. (2023a) Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023a. Let’s Verify Step by Step. arXiv:2305.20050. - Lightman et al. (2023b) Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023b. Let’s Verify Step by Step. arXiv:2305.20050. - Liu et al. (2025) Liu, R.; Gao, J.; Zhao, J.; Zhang, K.; Li, X.; Qi, B.; Ouyang, W.; and Zhou, B. 2025. Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling. arXiv:2502.06703. - Luo et al. (2025) Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; Tang, Y.; and Zhang, D. 2025. WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct. arXiv:2308.09583. - Muennighoff et al. (2025) Muennighoff, N.; Yang, Z.; Shi, W.; Li, X. L.; Fei-Fei, L.; Hajishirzi, H.; Zettlemoyer, L.; Liang, P.; Candès, E.; and Hashimoto, T. 2025. s1: Simple test-time scaling. arXiv:2501.19393. - Mukherjee et al. (2023) Mukherjee, S.; Mitra, A.; Jawahar, G.; Agarwal, S.; Palangi, H.; and Awadallah, A. 2023. Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv:2306.02707. - Setlur et al. (2024) Setlur, A.; Nagpal, C.; Fisch, A.; Geng, X.; Eisenstein, J.; Agarwal, R.; Agarwal, A.; Berant, J.; and Kumar, A. 2024. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. arXiv:2410.08146. - Shao et al. (2024) Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. - Snell et al. (2024) Snell, C.; Lee, J.; Xu, K.; and Kumar, A. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314. - Uesato et al. (2022) Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275. - Wang et al. (2024) Wang, P.; Li, L.; Shao, Z.; Xu, R. X.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv:2312.08935. - Wei and Zou (2019) Wei, J.; and Zou, K. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. arXiv:1901.11196. - Xia et al. (2024) Xia, M.; Malladi, S.; Gururangan, S.; Arora, S.; and Chen, D. 2024. LESS: Selecting Influential Data for Targeted Instruction Tuning. arXiv:2402.04333. - Xie et al. (2023) Xie, S. M.; Pham, H.; Dong, X.; Du, N.; Liu, H.; Lu, Y.; Liang, P.; Le, Q. V.; Ma, T.; and Yu, A. W. 2023. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. arXiv:2305.10429. - Yang, Liu, and Wang (2023) Yang, H.; Liu, X.-Y.; and Wang, C. D. 2023. FinGPT: Open-Source Financial Large Language Models. arXiv:2306.06031. - Ye et al. (2025) Ye, Y.; Huang, Z.; Xiao, Y.; Chern, E.; Xia, S.; and Liu, P. 2025. LIMO: Less is More for Reasoning. arXiv:2502.03387. - Yu et al. (2024) Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2024. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv:2309.12284. - Zhang et al. (2024) Zhang, K.; Zhang, J.; Li, H.; Zhu, X.; Hua, E.; Lv, X.; Ding, N.; Qi, B.; and Zhou, B. 2024. OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees. ICLR 2024. - Zhang et al. (2025) Zhang, Z.; Zheng, C.; Wu, Y.; Zhang, B.; Lin, R.; Yu, B.; Liu, D.; Zhou, J.; and Lin, J. 2025. The Lessons of Developing Process Reward Models in Mathematical Reasoning. arXiv:2501.07301. - Zheng et al. (2023) Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685. - Zhu et al. (2025) Zhu, J.; Chen, Q.; Dou, H.; Li, J.; Guo, L.; Chen, F.; and Zhang, C. 2025. DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models. arXiv:2504.15716. - Zhu et al. (2024) Zhu, J.; Li, J.; Wen, Y.; and Guo, L. 2024. Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 5673–5693. Bangkok, Thailand: Association for Computational Linguistics. - Zou et al. (2025) Zou, J.; Yang, L.; Gu, J.; Qiu, J.; Shen, K.; He, J.; and Wang, M. 2025. ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs. arXiv:2506.18896. ## Appendix A Appendix ## Table of Contents - Details on Models and Datasets - Model Implementations - Dataset Details - Details on Reward Signal Construction - Prompt for Qualitative Score ( $r_{\mathrm{qual}}$ ) - Prompt for Knowledge Extraction - Additional Experimental Setups - Fin-PRM Training - Downstream Tasks (SFT, BoN, GRPO) - Case Study: Fin-PRM in Action ## Appendix B A. Details on Models and Datasets ### B.1 A.1 Model Implementations Generator Models. Throughout our experiments, we utilize several powerful large language models as generators for reasoning traces and candidate solutions: - Deepseek-R1: A highly capable reasoning model used in our initial data synthesis phase to generate the base reasoning trajectories from the Cflue dataset. - Qwen3-235b-a22b: A state-of-the-art model used for two critical auxiliary tasks: (1) extracting the structured financial knowledge base $\mathcal{K}$ from expert analyses, and (2) serving as the powerful LLM-as-a-judge to provide the qualitative score, $r_{\mathrm{qual}}$ . - Qwen2.5-7B-Instruct: A versatile and efficient model used as the base for our Fin-PRM, as well as the student model in our SFT experiments and the policy model in our GRPO and BoN experiments. Reward Models (Baselines). To benchmark the performance of Fin-PRM, we compare it against a strong, publicly available, general-purpose PRM: - Qwen2.5-Math-PRM-7B: A state-of-the-art PRM specialized for the mathematics domain. It is trained on a vast corpus of math reasoning problems and serves as a powerful baseline to highlight the benefits of domain specialization. Its strong performance in a technical domain makes it a challenging benchmark for our finance-specific model. - Qwen2.5-Math-PRM-72B: A state-of-the-art PRM specialized for the mathematics domain. The larger 72-billion parameter version, included to establish a strong upper-bound for general-purpose PRM performance on our tasks. ### B.2 A.2 Dataset Details Primary Data Source. Our entire framework is built upon the CFLUE (Chinese Financial Language Understanding Evaluation) benchmark. We selected CFLUE because it is a high-quality, knowledge-intensive dataset where questions are accompanied by detailed, human-written expert analyses. This unique feature provides the ground-truth knowledge necessary for our fact-checking reward components ( $r_{\mathrm{acc}}$ ). Synthesized Reasoning Dataset. We constructed a new dataset of 3,000 samples for training Fin-PRM. Each sample in our dataset, $\mathcal{D}$ , is a tuple $(x,s,a,\mathcal{K}_{x},y,y_{\mathrm{analysis}})$ containing: - $x$ : The original question from CFLUE. - $s$ : The reasoning trace generated by Deepseek-R1. - $a$ : The final answer synthesized by Deepseek-R1. - $\mathcal{K}_{x}$ : The subset of our global knowledge base $\mathcal{K}$ relevant to question $x$ . - $y$ : The ground-truth correct answer from CFLUE. - $y_{\mathrm{analysis}}$ : The ground-truth expert analysis from CFLUE, used to construct $\mathcal{K}$ . This structure enables our multi-faceted reward signal construction and provides a robust foundation for training a knowledge-aware reward model. #### Prompt for reasoning trace $s$ and solution $a$ To synthesize the reasoning traces ( $s$ ) and their corresponding solutions ( $a$ ), we prompt the Deepseek-R1 model with a detailed set of instructions. This prompt is designed to elicit a long-form, step-by-step thought process, followed by a clean, final answer. The full prompt is shown in Listing 1. The ‘[Question Text]‘ placeholder is then replaced with the actual question from the CFLUE dataset. Listing 1: Prompt for Generating Reasoning Traces and Solutions from Deepseek-R1 ⬇ 1 Your role as an assistant involves thoroughly exploring questions through a systematic long thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop a well - considered thinking process. Please structure your response into two main sections: Thought and Solution. 2 3 In the Thought section, detail your reasoning process using the specified format: 4 <| begin_of_thought |> 5 {thought with steps separated with ’\ n \ n ’} 6 <| end_of_thought |> 7 Each step should include detailed considerations such as analyzing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. 8 9 In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The solution should maintain a logical, accurate, concise expression style and detail necessary steps needed to reach the conclusion, formatted as follows: 10 <| begin_of_solution |> 11 {final formatted, precise, and clear solution} 12 <| end_of_solution |> 13 14 Now, try to solve the following question through the above guidelines: 15 [Question Text] #### Prompt for Knowledge Extraction To construct the ground-truth knowledge base ( $\mathcal{K}_{x}$ ) for each question, we prompt the LLM judge to act as a domain expert. Its task is to read the trusted expert analysis ( $y_{\mathrm{analysis}}$ ) provided in the dataset and extract all key financial terms along with their definitions as presented in the text. This process creates a structured, verifiable source of facts for the downstream accuracy and coverage rewards. The prompt template is detailed in Listing 2. Listing 2: Template for the Knowledge Extraction Prompt ⬇ 1 You are a financial knowledge extraction expert. Read the following expert analysis and identify all key financial terms and concepts. For each term, provide a concise definition based on the text. 2 3 ** Expert Analysis Text:** 4 [Expert Analysis from the Dataset] 5 6 --- 7 ** Your Task:** 8 Output a JSON list where each object represents a key knowledge point. 9 10 ** Output Format (JSON list only):** 11 [ 12 { 13 " Term ": "< Identified_Term_1 >", 14 " Explanation ": "< Definition_of_Term_1 >" 15 }, 16 { 17 " Term ": "< Identified_Term_2 >", 18 " Explanation ": "< Definition_of_Term_2 >" 19 } 20] ## Appendix C B. Details on Reward Signal Construction This section provides further details on the prompts used to generate the multi-faceted reward signals described in the main paper. These prompts are designed to elicit structured, machine-readable outputs from a powerful LLM judge (Qwen3-235b-a22b). ### C.1 B.1 Prompt for Qualitative Score ( $r_{\mathrm{qual}}$ ) To assess the intrinsic quality of each reasoning step, we use a structured prompt that asks the LLM judge to evaluate a step ( $s_{t}$ ) based on three criteria. The prompt takes the original question ( $x$ ), the reasoning history ( $s_{<t}$ ), and the current step ( $s_{t}$ ) as input. The LLM is instructed to provide a score from 0.0 to 1.0 for each of the following aspects, ensuring the output is a machine-parsable JSON object: - Logical Soundness: How coherent and logically valid is the reasoning within this specific step? - Step Correctness: Is the information presented in the step factually or procedurally correct, independent of the overall strategy? - Target Progression: How effectively does this step move the overall reasoning process closer to a correct final answer? The template for this prompt is shown in Listing 3. Listing 3: Template for the Qualitative Score Prompt ⬇ 1 You are an expert financial analyst. Given the question, the previous reasoning steps, and the current step, evaluate the quality of the ** current step **. 2 3 ** Question:** 4 [Original Question Text] 5 6 ** Reasoning History:** 7 [Reasoning History up to step t -1] 8 9 ** Current Step to Evaluate:** 10 [Text of step t] 11 12 --- 13 ** Your Task:** 14 Provide a JSON object with your evaluation based on three criteria: 15 1. ‘ logical_soundness ‘: How logical is the current step? 16 2. ‘ step_correctness ‘: Is the information in the step correct? 17 3. ‘ target_progression ‘: Does the step help solve the problem? 18 19 ** Output Format (JSON only):** 20 { 21 " logical_soundness ": < float_from_0_to_1 >, 22 " step_correctness ": < float_from_0_to_1 >, 23 " target_progression ": < float_from_0_to_1 > 24} ### C.2 B.2 Prompts for Verifiable and Knowledge-Based Rewards This subsection details the prompts used to generate rewards that are grounded in external, verifiable information, such as the ground-truth answer or our extracted knowledge base. These prompts are crucial for ensuring the factual correctness and anti-hallucination properties of Fin-PRM. #### Prompt for Accuracy Score ( $r_{\mathrm{acc}}$ ) The accuracy score is a composite of two verifiable checks. We use distinct prompts for each to ensure a grounded, factual evaluation. Procedural Correctness Prompt. This prompt verifies if a step is a valid move towards the known correct answer ( $y$ ), assessing its logical utility in the context of a correct solution path. Listing 4: Template for Procedural Correctness Prompt ⬇ 1 You are a logical verifier. Given the reasoning so far and the known correct answer, determine if the current step is a logically sound and productive move towards that answer. 2 3 ** Question:** 4 [Original Question Text] 5 6 ** Reasoning History:** 7 [Reasoning History up to step t -1] 8 9 ** Correct Final Answer:** 10 [Ground Truth Answer y] 11 12 ** Current Step to Evaluate:** 13 [Text of step t] 14 15 --- 16 ** Your Task:** 17 Is this step a valid, logical progression towards the correct final answer? Respond with a JSON object containing a binary value. 18 19 ** Output Format (JSON only):** 20 { " procedural_correctness ": <1 _for_yes_or_0_for_no > } Factual Accuracy Prompt. This prompt validates the claims within a step against the extracted knowledge base ( $\mathcal{K}_{x}$ ), acting as a direct anti-hallucination check. Listing 5: Template for Factual Accuracy Prompt ⬇ 1 You are a fact - checking agent. Verify every factual claim and financial term in the " Current Step " against the provided " Knowledge Base ". 2 3 ** Knowledge Base:** 4 - < Term_1 >: < Definition_1 > 5 - < Term_2 >: < Definition_2 > 6 7 ** Current Step to Evaluate:** 8 [Text of step t] 9 10 --- 11 ** Your Task:** 12 Are all claims and terms in the current step supported by the knowledge base? Respond with a JSON object containing a binary value. 13 14 ** Output Format (JSON only):** 15 { " factual_accuracy ": <1 _if_all_claims_are_supported_or_0_otherwise > } #### Prompt for Knowledge Coverage ( $r_{\mathrm{cover}}$ ) To calculate the trajectory-level knowledge coverage score, this prompt asks the LLM to verify which of the required knowledge points ( $\mathcal{K}_{x}$ ) were used in the model’s full generated response ( $s$ and $a$ ). Listing 6: Template for the Knowledge Coverage Prompt ⬇ 1 You are a verification agent. Your task is to check if the required financial knowledge points were used in the provided model response. 2 3 ** Required Knowledge Points:** 4 1. < Term_1 >: < Definition_1 > 5 2. < Term_2 >: < Definition_2 > 6... 7 8 ** Model ’ s Reasoning Trace and Answer:** 9 [Model ’ s Full Generated Response] 10 11 --- 12 ** Your Task:** 13 Analyze the model ’ s response and determine how many of the required knowledge points were covered. Output a JSON object with the count and indices of the covered points. 14 15 ** Output Format (JSON only):** 16 { 17 " coverage_number ": < integer >, 18 " coverage_index ": [< list_of_covered_indices >] 19} ## Appendix D C. Additional Experimental Setups This section provides detailed configurations and hyperparameters for the training of Fin-PRM and its application in the three downstream tasks. ### D.1 C.1 Fin-PRM Training Details Fin-PRM was trained by fine-tuning the Qwen2.5-7B-Instruct model on our newly constructed financial reasoning dataset. The training objective combined step-level and trajectory-level losses, as described in Equation 16. The key hyperparameters used for training are summarized in Table 3. All training was conducted on NVIDIA A100 GPUs. Table 3: Hyperparameters for Fin-PRM Training. | Parameter Base Model Dataset Size | Value Qwen2.5-7B-Instruct 3,000 samples | | --- | --- | | Learning Rate | 2e-5 | | Batch Size | 1(per device) | | Gradient Accumulation Steps | 2 | | Max Sequence Length | 8192 | | Epochs | 3 | | Optimizer | AdamW | | LR Scheduler | Cosine with warmup | | Warmup Steps | 50 | | Loss Weight ( $\lambda$ in Eq. 16) | 1.0 | ### D.2 C.2 Downstream Task Setups This subsection details the experimental setups for the three application scenarios: Supervised Fine-Tuning (SFT), Best-of-N (BoN) selection, and Group Relative Policy Optimization (GRPO). #### SFT with Data Selection For the offline data selection task, we first used Fin-PRM to score and select the top 1,000 reasoning traces from a larger pool of synthetic data. We then fine-tuned the Qwen2.5-7B-Instruct model on this curated subset. The SFT process used the same set of hyperparameters as the PRM training (see Table 3), ensuring a fair comparison. #### Best-of-N (BoN) Selection In the test-time scaling experiments, the policy model (Qwen2.5-7B-Instruct) generated N candidate responses for each question. Fin-PRM then scored each candidate using the composite ranking score from Equation 20. The candidate with the highest score was selected as the final answer. Based on our ablation study (Figure 4), the hyperparameter $\zeta$ was set to 1.0 to give equal weight to step-level and trajectory-level rewards. #### GRPO Reinforcement Learning For the online policy optimization, we integrated Fin-PRM into the GRPO framework. The policy model was Qwen2.5-7B-Instruct, initialized from the same base checkpoint as other experiments. The composite reward from Equation 22 was used to calculate advantages. Key GRPO-specific hyperparameters are listed in Table 4. Table 4: Hyperparameters for GRPO Policy Optimization. | Parameter Policy Model Samples per Group (N) | Value Qwen2.5-7B-Instruct 6 | | --- | --- | | Reward Composition ( $\delta$ in Eq. 22) | 0.5 | | KL Penalty Coefficient ( $\beta_{\mathrm{KL}}$ ) | 0.001 | | PPO Clipping Parameter ( $\epsilon$ ) | 0.2 | | Epochs | 1 | | Batch Size | 32 | | Learning Rate | 1e-6 | ## Appendix E D. Case Study: A Data Sample in Detail To better illustrate the granular nature of our reward signals, this section presents a complete data sample in a format that explicitly links each reasoning step to its corresponding reward score. Question: A business of a commercial bank that does not constitute on-balance-sheet assets or liabilities but generates non-interest income is called its ( ). Choices: A: Investment business, B: Asset business, C: Intermediary business, D: Liability business ### Thinking Trajectories Step 1: Analyze the Problem The problem requires identifying a commercial bank’s business type. The key criteria are: (1) It does not constitute on-balance-sheet assets or liabilities, meaning it is an off-balance-sheet business. (2) It generates non-interest income, such as fees or commissions. The options are A. Investment, B. Asset, C. Intermediary, and D. Liability business. The reasoning must be based on the classification of commercial bank operations. [Step-level reward: 0.644] Step 2: Summarize Findings - Asset business (Option B): Involves the use of bank funds, forming on-balance-sheet assets (e.g., loans, securities investments). It primarily generates interest income and does not meet the ”off-balance-sheet” condition. - Liability business (Option D): Involves the bank’s sources of funds, forming on-balance-sheet liabilities (e.g., deposits, borrowings). It primarily incurs interest expenses and does not fit the criteria. - Investment business (Option A): Typically refers to the bank investing in financial instruments (bonds, stocks), which is a subset of asset business. It constitutes on-balance-sheet assets. - Intermediary business (Option C): Refers to the bank acting as a service intermediary, not using its own funds. It is off-balance-sheet and its income comes from fees and commissions (non-interest income). This matches the description. [Step-level reward: 0.769] Step 3: Brainstorm New Ideas To test the robustness of the conclusion, this step considers potential edge cases. For instance, do certain investment activities like derivatives trading count? Or do credit card fees (non-interest income) from a liability product complicate the definition? This deepens the analysis beyond a superficial match. [Step-level reward: 0.753] Step 4: Verify Step Accuracy This step cross-references the definition of ”intermediary business” with formal regulations, such as a country’s Commercial Bank Law or accounting standards. It confirms that these formal sources align with the conclusion from Step 2, providing an external, factual grounding. [Step-level reward: 0.686] Step 5: Refine Errors The model considers and clarifies potential misconceptions. For example, it distinguishes between a contingent liability (like a letter of credit, common in intermediary business) which is off-balance-sheet until triggered, and a direct on-balance-sheet liability. This refines the understanding and prevents oversimplification. [Step-level reward: 0.839] Step 6: Revisit Previous Steps The final step involves a full review of the reasoning chain. It confirms that the initial analysis in Step 1 was correct, the evaluation in Step 2 was sound, and the verifications in subsequent steps all support the same conclusion, ensuring the entire trajectory is logically coherent. [Step-level reward: 0.837] ### Final Response Based on the thought process, the business is the commercial bank’s intermediary business. - Reason: Intermediary business does not constitute on-balance-sheet assets or liabilities (i.e., it is an off-balance-sheet business) but generates non-interest income (such as fees and commissions) by providing intermediary services (e.g., payment settlement, agency services, consulting). - Exclusion of other options: - A. Investment business: Constitutes an on-balance-sheet asset. - B. Asset business: Directly forms an on-balance-sheet asset. - D. Liability business: Forms an on-balance-sheet liability. Therefore, the correct answer is C. Intermediary Business. [Trajectory-level Label: 1 (Correct)]