## Diagram: MCTS-Driven Deep Thinking and Self-Evolution Process ### Overview The image is a technical diagram illustrating a multi-stage process for improving AI model reasoning through Monte Carlo Tree Search (MCTS) and iterative self-evolution. It is divided into three main panels: (a) a step-by-step verified reasoning trajectory, (b) the construction of preference pairs from Q-values, and (c) a four-round self-evolution cycle. The diagram uses a combination of tree structures, flowcharts, and labeled components to explain the methodology. ### Components/Axes The diagram is segmented into three distinct panels, each with its own components: **Panel (a): step-by-step verified reasoning trajectory** * **Location:** Left side of the image. * **Main Structure:** A tree diagram originating from a central blue node labeled "question". * **Key Labels & Components:** * **Top:** "MCTS-driven deep thinking" (Title). * **Left of Tree:** "SLM" with a cat icon. * **Right of Tree:** "PPM" with a robot icon. * **Tree Nodes:** Circles containing numerical values (e.g., 0.8, 0.7, 0.5, -0.7, -0.9, 0.6). * **Legend (Right of Tree):** * Dashed purple box: "Apply Verifiers (PPM/python)" * White circle: "One step" * Green circle: "Answer step (correct)" * Red circle: "Answer step (wrong)" * **Bottom Label:** "(a) step-by-step verified reasoning trajectory" **Panel (b): Construction of per-step preference pairs based on Q-values** * **Location:** Top-right quadrant. * **Main Structure:** A horizontal sequence of simplified tree structures showing progression. * **Key Labels & Components:** * **Top Arrow:** "Q-value filtering" pointing from left to right. * **Sequence Labels:** "Step 1", "Step 2", "final step", "full solutions". * **Node Colors:** Blue (root), Green (correct), Red (incorrect). The proportion of green nodes increases from left to right. * **Bottom Label:** "(b) Construction of per-step preference pairs based on Q-values" **Panel (c): 4 rounds of self-evolution** * **Location:** Bottom-right quadrant. * **Main Structure:** A horizontal flowchart showing four iterative rounds. * **Key Labels & Components:** * **Round 1:** "Terminal-guided MCTS" -> "SLM-r1" (with cat icon). * **Round 2:** "Terminal-guided MCTS" -> "SLM-r2" -> "PPM-augmented" (with robot icon). * **Round 3:** "SLM-r3" -> "PPM-augmented" (with robot icon). * **Round 4:** "SLM-r4" -> "PPM-augmented" (with robot icon). * **Icons:** A cat icon (labeled "SLM") and a robot icon (labeled "PPM") appear at various stages. * **Bottom Label:** "(c) 4 rounds of self-evolution" ### Detailed Analysis **Panel (a) Analysis:** * The process starts with a "question" node. * The tree expands with steps assigned numerical values (likely Q-values or confidence scores). Values range from positive (e.g., 0.8, 0.7) to negative (e.g., -0.7, -0.9). * Dashed purple boxes ("Apply Verifiers") enclose specific nodes and their children, indicating verification is applied at those decision points. * The terminal nodes are classified as correct (green) or wrong (red). The final row shows a mix: two green (correct) and two red (wrong) answer steps. **Panel (b) Analysis:** * This panel visualizes a filtering process. It starts with a tree containing many red (incorrect) nodes. * Through "Q-value filtering" across sequential steps, the incorrect (red) branches are pruned. * By the "final step" and "full solutions", the tree is dominated by green (correct) nodes, demonstrating the selection of higher-quality reasoning paths. **Panel (c) Analysis:** * This outlines an iterative training or refinement loop. * **Round 1:** Uses "Terminal-guided MCTS" to produce an initial Small Language Model (SLM-r1). * **Round 2:** The process is repeated, but the output is now augmented by a Preference/Policy Model (PPM), creating SLM-r2. * **Rounds 3 & 4:** The cycle continues, with each new SLM version (r3, r4) being refined using the PPM-augmented data from the previous round. The icons suggest the SLM (cat) and PPM (robot) are distinct models collaborating. ### Key Observations 1. **Verification Integration:** Panel (a) explicitly shows that verification (via PPM/python) is not applied uniformly but at specific, likely critical, decision points in the reasoning tree. 2. **Value-Driven Pruning:** Panel (b) clearly links the pruning of incorrect reasoning paths to Q-values, showing a direct mechanism for quality improvement. 3. **Evolutionary Progression:** Panel (c) depicts a clear, staged evolution where the system's capability is built incrementally over four rounds, with the PPM playing an increasingly integral role after the first round. 4. **Visual Coding Consistency:** The color scheme (green=correct, red=wrong) is consistent across panels (a) and (b), creating a coherent visual language for success and failure. ### Interpretation This diagram outlines a sophisticated framework for enhancing the reasoning capabilities of a Small Language Model (SLM). The core idea is to move beyond simple, single-path generation. * **What it demonstrates:** It shows a method that combines **exploration** (via MCTS to generate multiple reasoning paths), **evaluation** (using verifiers and Q-values to score steps), and **selection** (pruning poor paths to construct preference pairs). This curated data is then used in a **self-evolution loop**, where the model progressively learns from its own refined outputs, augmented by a stronger Preference Model (PPM). * **How elements relate:** Panel (a) is the core reasoning engine. Panel (b) is the data processing layer that converts the engine's outputs into training signals. Panel (c) is the macro-level training regimen that uses these signals to iteratively improve the base model (SLM). The PPM acts as a judge or guide throughout. * **Notable Implications:** The process aims to produce more reliable and accurate model outputs by systematically identifying and reinforcing correct reasoning chains while discarding flawed ones. The "self-evolution" aspect suggests a goal of reducing reliance on external human feedback over time, as the model learns to improve using its own verified trajectories. The use of "Terminal-guided MCTS" in the first round implies an initial strategy to ensure the search process is directed toward plausible final answers. **Language Note:** The diagram contains English text and labels. The cat and robot icons are symbolic and do not contain translatable text.