## Diagram: Comparison of Reward Models ### Overview The image presents a comparative diagram illustrating three different reward models: Outcome Reward Model (ORM), Multimodal Process Reward Model (PRM), and GM-PRM (a proposed model). The diagram highlights the flow of information and the reward mechanisms associated with each model. ### Components/Axes * **Title:** The image is divided into three sections, each representing a different reward model. * (a) Outcome Reward Model * (b) Multimodal Process Reward Model * (c) GM-PRM (Ours) * **Input:** Each model starts with an input. * ORM: "Question" (represented by a blue box containing the equation E=mc²) * PRM and GM-PRM: "Multimodal Math Qns" (represented by a blue box containing math equations) * **Process:** The models process the input through a series of steps. * ORM: A single "Output" arrow leading to "Answer" (green box). * PRM and GM-PRM: A series of steps labeled "Step 1", "Step 2", ..., "Step T" (green boxes) leading to "Answer" (green box). * **Reward:** Each model includes a reward mechanism. * ORM, PRM, and GM-PRM: A robot icon labeled "ORM", "PRM", and "GM-PRM" respectively, receiving a "Reward" (orange dashed arrow). * **Annotations:** Additional text annotations provide context and highlight key differences. * ORM: "ONLY Reward Final Ans" (red text) * PRM: "Limited Explainability", "No Correction Mechanism" (red text) * GM-PRM: "1st incorrect step" (purple text), "After Correction" (purple arrow), "Refined BoN" (green text), "Refined & Corrected Version" (purple text) * **Analysis & Judgement (GM-PRM):** A purple box at the bottom of the GM-PRM section contains the labels "Step Intent", "Image Alignment", and "Reasoning Logic". ### Detailed Analysis **Outcome Reward Model (ORM):** * The model takes a "Question" as input. * The question is processed to produce an "Answer". * The "ORM" agent receives a "Reward" only based on the final answer. * The annotation "ONLY Reward Final Ans" emphasizes that the reward is solely based on the outcome. **Multimodal Process Reward Model (PRM):** * The model takes "Multimodal Math Qns" as input. * The input is processed through a series of steps: "Step 1", "Step 2", ..., "Step T". * The final step leads to an "Answer". * The "PRM" agent receives a "Reward" based on the process. * The annotations "Limited Explainability" and "No Correction Mechanism" highlight the limitations of this model. **GM-PRM (Ours):** * The model takes "Multimodal Math Qns" as input. * The input is processed through a series of steps: "Step 1", "Step 2" (purple), ..., "Step T". * The annotation "1st incorrect step" indicates a point where a correction mechanism is applied. * The "After Correction" arrow shows the flow after a correction. * The final step leads to an "Answer". * The "GM-PRM" agent receives a "Reward" based on the refined process. * The "Refined BoN" annotation suggests a refined version of something (likely a "Bag of Neurons" or similar concept). * The "Refined & Corrected Version" annotation emphasizes the improvements made in this model. * The "Analysis & Judgement" box indicates the model's ability to analyze and judge the steps involved. ### Key Observations * The ORM is the simplest model, focusing only on the final outcome. * The PRM considers the process but lacks explainability and a correction mechanism. * The GM-PRM builds upon the PRM by incorporating a correction mechanism and analysis/judgment capabilities. ### Interpretation The diagram illustrates the evolution of reward models for problem-solving, particularly in the context of multimodal math questions. The GM-PRM model is presented as an improvement over existing models by incorporating a correction mechanism and analysis/judgment capabilities. This suggests that the GM-PRM is more robust and capable of handling complex problems compared to the ORM and PRM models. The "Refined BoN" annotation implies that the GM-PRM utilizes a more sophisticated approach to processing information and generating solutions. The diagram highlights the importance of considering the process and incorporating feedback mechanisms in reward models for complex tasks.

## Diagram: Comparison of Reward Model Architectures ### Overview The image presents a comparative architectural diagram illustrating three distinct approaches to training reward models for AI systems, specifically in the context of multimodal mathematical reasoning. The diagram progresses from a basic Outcome Reward Model (ORM) to a standard Process Reward Model (PRM), and finally to the proposed "GM-PRM" (Generative Multimodal Process Reward Model), which introduces correction mechanisms. ### Components/Axes The diagram is divided into three horizontal panels, each representing a different model architecture: * **Panel (a): Outcome Reward Model (ORM)** * **Input:** "Question" (represented by a chalkboard icon with $E=mc^2$). * **Process:** "Answer" (green box). * **Output:** Robot icon labeled "ORM". * **Feedback:** Orange dashed arrow labeled "Reward" pointing from ORM to "Answer". * **Annotation:** "ONLY Reward Final Ans" (red text, top-right). * **Panel (b): Multimodal Process Reward Model (PRM)** * **Input:** "Multimodal Math Qns" (represented by icons showing $\sqrt{4(x+y)}$ and $\pi \int \sqrt{x} \sum$). * **Process:** A sequence of steps: "Step 1" (green), "Step 2" (green), "..." (ellipsis), "Step T" (green), "Answer" (green). * **Output:** Robot icon labeled "PRM". * **Feedback:** Multiple orange dashed arrows labeled "Reward" pointing from PRM to each individual step. * **Annotations:** "Limited Explainability" and "No Correction Mechanism" (red text, top-right). * **Panel (c): GM-PRM (Ours)** * **Input:** "Multimodal Math Qns" (same as panel b). * **Process:** A sequence of steps: "Step 1" (green), "Step 2" (purple, labeled "1st incorrect step"), "..." (ellipsis), "Step T" (green), "Answer" (green). * **Transition:** An arrow labeled "After Correction" connects "Step 2" to "Step T". * **Output:** Robot icon labeled "GM-PRM". * **Feedback:** Orange dashed arrows from GM-PRM to steps; a purple dashed arrow from GM-PRM to "Step 2". * **Annotations:** "Refined BoN" (Best-of-N) near the output arrow. * **Footer (Bottom):** A dashed purple box labeled "Analysis & Judgement" containing three sub-components: "Step Intent", "Image Alignment", and "Reasoning Logic". A separate box to the right labeled "Refined & Corrected Version". ### Detailed Analysis * **Panel (a) - ORM:** The flow is linear and singular. The model only evaluates the final output. The annotation "ONLY Reward Final Ans" confirms that intermediate reasoning steps are ignored. * **Panel (b) - PRM:** The flow is granular. The model evaluates each step individually. However, the annotations "Limited Explainability" and "No Correction Mechanism" indicate that while the model can identify a bad step, it cannot rectify the reasoning path. * **Panel (c) - GM-PRM:** This architecture introduces a branching or corrective logic. * **Visual Indicator:** "Step 2" is highlighted in purple, indicating it is the "1st incorrect step." * **Correction Flow:** The diagram shows a path "After Correction" that bypasses or modifies the sequence following the incorrect step. * **Feedback Loop:** The purple dashed arrow from the GM-PRM robot back to "Step 2" suggests the model actively identifies and targets the error for correction. * **Supporting Components:** The footer indicates that the GM-PRM relies on "Analysis & Judgement" (Step Intent, Image Alignment, Reasoning Logic) to produce a "Refined & Corrected Version." ### Key Observations * **Evolution of Feedback:** The feedback mechanism evolves from a single point (ORM) to multiple points (PRM) to targeted, corrective feedback (GM-PRM). * **Color Coding:** Green boxes represent standard/correct steps, while the purple box in panel (c) specifically denotes an error ("1st incorrect step"). * **Multimodal Integration:** The inclusion of "Image Alignment" in the footer of panel (c) suggests that the GM-PRM architecture is specifically designed to handle multimodal inputs (e.g., math problems involving images or complex notation) more effectively than the previous models. ### Interpretation The diagram demonstrates a progression in AI reasoning evaluation. 1. **ORM** is depicted as insufficient for complex tasks because it treats the reasoning process as a "black box," only caring about the final result. 2. **PRM** improves upon this by evaluating intermediate steps, but the diagram highlights a critical failure mode: it lacks the ability to fix errors once they occur ("No Correction Mechanism"). 3. **GM-PRM** is presented as the superior solution. It not only evaluates steps but actively performs "Analysis & Judgement" (using Step Intent, Image Alignment, and Reasoning Logic) to identify the "1st incorrect step" and apply a correction. This implies that GM-PRM is capable of iterative self-correction, leading to a "Refined & Corrected Version" of the reasoning path, which likely results in higher accuracy for complex multimodal mathematical problems.

# Technical Document Analysis: Multimodal Math Problem-Solving Reward Models ## Overview The image presents a comparative analysis of three reward models for evaluating multimodal math problem-solving processes. The diagram uses flowcharts with labeled components, directional arrows, and annotations to illustrate differences in reward mechanisms and correction capabilities. --- ## Diagram Components ### (a) Outcome Reward Model **Title**: Outcome Reward Model **Key Components**: - **Question** (Input: Math equation "E=mc²") - **Output** (Intermediate processing step) - **Answer** (Final response) - **ORM** (Robot icon with "Reward" arrow) **Flow**: ``` Question → Output → Answer → Reward (only for final answers) ``` **Annotations**: - Red text: "ONLY Reward Final Ans" - ORM robot has a smiley face --- ### (b) Multimodal Process Reward Model **Title**: Multimodal Process Reward Model **Key Components**: - **Multimodal Math Qns** (Input: Math problems with equations and diagrams) - **Steps 1, 2, ..., T** (Intermediate reasoning steps) - **Answer** (Final response) - **PRM** (Robot icon with "Reward" arrow) **Flow**: ``` Multimodal Math Qns → Step 1 → Step 2 → ... → Step T → Answer → Reward ``` **Annotations**: - Red text box: "Limited Explainability" - Red text box: "No Correction Mechanism" - PRM robot has a neutral expression --- ### (c) GM-PRM (Ours) **Title**: GM-PRM (Ours) **Key Components**: - **Multimodal Math Qns** (Input: Math problems with equations and diagrams) - **Steps 1, 2, ..., T** (Intermediate reasoning steps) - **1st incorrect step** (Highlighted with dashed arrow) - **After Correction** (Dashed arrow indicating refinement) - **Refined BoN** (Refined version of the answer) - **GM-PRM** (Robot icon with "Reward" arrow) **Flow**: ``` Multimodal Math Qns → Step 1 → Step 2 → ... → Step T → Answer → Reward ``` **Sub-process**: ``` Analysis & Judgement → Step Intent → Image Alignment → Reasoning Logic ``` **Annotations**: - Red text: "Refined & Corrected Version" - Dashed arrows indicate iterative refinement - GM-PRM robot has a smiling face with cheeks --- ## Key Differences Between Models | Feature | Outcome Reward Model (a) | Multimodal Process Reward Model (b) | GM-PRM (c) | |------------------------|--------------------------|-------------------------------------|--------------------------------| | **Reward Scope** | Final answer only | All steps | All steps + corrections | | **Explainability** | Not specified | Limited | Improved through analysis | | **Correction** | No | No | Yes (via "After Correction") | | **Robot Icon** | ORM (smiley) | PRM (neutral) | GM-PRM (smiling with cheeks) | --- ## Spatial Grounding & Component Isolation 1. **Header**: Each section title (a, b, c) is bolded and positioned at the top of its respective flowchart. 2. **Main Chart**: - Components are arranged left-to-right with vertical stacking for sub-processes (e.g., "Analysis & Judgement" in c). - Arrows indicate sequential flow (solid for primary flow, dashed for corrections). 3. **Footer**: No explicit footer, but annotations are placed in red text boxes near relevant components. --- ## Trend Verification - **No numerical data** present; trends are represented through structural differences in component connectivity and annotation emphasis. --- ## Conclusion The diagram demonstrates a progression from simple outcome-based evaluation (a) to more sophisticated process-aware models (b and c). The proposed GM-PRM (c) introduces explicit correction mechanisms and multi-stage analysis, addressing limitations in explainability and adaptability observed in prior models.