2503.09567v5

# Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models **Authors**: Project: https://long-cot.github.io/ ## Abstract Recent advancements in reasoning with large language models (RLLMs), such as OpenAI-o1 and DeepSeek-R1, have demonstrated their impressive capabilities in complex domains like mathematics and coding. A central factor in their success lies in the application of long chain-of-thought (Long CoT) characteristics, which enhance reasoning abilities and enable the solution of intricate problems. However, despite these developments, a comprehensive survey on Long CoT is still lacking, limiting our understanding of its distinctions from traditional short chain-of-thought (Short CoT) and complicating ongoing debates on issues like “overthinking” and “inference-time scaling”. This survey seeks to fill this gap by offering a unified perspective on Long CoT. Specifically, (1) We first distinguish Long CoT from Short CoT and introduce a novel taxonomy to categorize current reasoning paradigms. (2) Next, we explore the key characteristics of Long CoT: deep reasoning, extensive exploration, and feasible reflection, which enable models to handle more complex tasks and produce more efficient, coherent outcomes compared to the shallower Short CoT. (3) We then investigate key phenomena such as the emergence of Long CoT with these characteristics, including overthinking, and inference-time scaling, offering insights into how these processes manifest in practice. (4) Finally, we identify significant research gaps and highlight promising future directions, including the integration of multi-modal reasoning, efficiency improvements, and enhanced knowledge frameworks. By providing a structured overview, this survey aims to inspire future research and further the development of reasoning large language models Our logo refers to a cute cartoon image - Snake Puppy. Header Image is inspired by Yaoting et al. [959]. ## 1 Introduction In recent years, as shown in Figure 1, the emergence of reasoning large language models (RLLMs) such as OpenAI o1 [307] and DeepSeek R1 [227] has sparked a growing body of research into Long Chain-of-Thought (Long CoT) reasoning, greatly improving their mathematical reasoning, programming tasks, and multidisciplinary knowledge reasoning capabilities [696, 980, 722, 79, 961, 200, 1113, 793], even passing Turing Test [334]. This shift marks a significant departure from traditional approaches to task handling in large language models (LLMs) [1147, 619, 622, 599]. Unlike the shorter chain-of-thought (Short CoT) used in traditional LLMs, Long CoT reasoning entails a more detailed, iterative process of exploration and reflection within a given problem space by inference-time scaling [419, 733, 524]. This process has led to notable advancements in mathematical and logical reasoning, as well as in exploring how supervised fine-tuning (SFT) and reinforcement learning (RL) techniques can enhance the learning and exploration of extended reasoning chains [623, 550]. <details> <summary>x3.png Details</summary>  ### Visual Description ## Diagram: Taxonomy of AI Reasoning Methods and Models (2022-2025) ### Overview This image is a complex, tree-like taxonomy diagram illustrating the evolution and categorization of AI reasoning methods, models, and research directions from 2022 to March 2025. It organizes numerous named techniques and models into vertical streams, with time progressing from bottom (2022) to top (2025.03). The diagram uses color-coding, icons, and connecting lines to show relationships and lineage. ### Components/Axes * **Vertical Timeline (Left Axis):** Years and specific months are marked: 2022, 2023, 2024, 2025.01, 2025.02, 2025.03. * **Horizontal Categories (Bottom Legend):** The diagram's branches are categorized into three primary research directions, each with sub-categories: * **Deep Reasoning** (Teal/Green palette) * Deep Reasoning Format * Deep Reasoning Learning * **Feasible Reflection** (Brown/Tan palette) * Feedback * Refinement * **Extensive Exploration** (Blue/Purple palette) * Exploration Scaling * Internal Exploration * External Exploration * **Source Legend (Bottom Right):** A box defines the meaning of node border colors: * **Yellow Border:** Open Resource * **Grey Border:** Close Resource * **Node Elements:** Each node is a rounded rectangle containing a model/method name (e.g., "CoT", "STaR", "o1-coder"). Many nodes have a small icon to their left, indicating the associated organization or framework (e.g., Google's "G", OpenAI's spiral, Anthropic's "A", various university logos). * **Connectors:** Lines of varying colors (matching the category palettes) connect nodes, indicating relationships, inspiration, or evolutionary paths. The lines converge at a central pink tree-like structure at the bottom, symbolizing a common root or foundation. ### Detailed Analysis The diagram is densely populated. Below is a structured extraction of the labeled nodes, organized by their vertical timeline position and approximate horizontal category stream. **2022 (Bottom)** * **Root/Foundation:** `CoT` (Chain-of-Thought), `STaR` (Self-Taught Reasoner). **2023** * **Left (Deep Reasoning):** `PlanningTokens`, `CLP`, `Finetune-CoT`, `MathPrompter`, `ReST`. * **Center-Left (Feasible Reflection):** `CRITIC`, `Shepherd`, `Self-Verification`, `PDS`, `Step-DPO`, `Promptbreeder`, `ReFT`, `IRPO`, `CPO`, `DeepseekMath`, `PoT` (Program of Thoughts), `Brain`, `ENVISIONS`, `Lean-STaR`, `CoC`, `Quiet-STaR`, `MuSR`. * **Center-Right (Extensive Exploration):** `Self-Refine`, `Self-critiquing`, `Final Answer RL`, `G ReAct`, `DTV`, `DiVeRSe`, `CLSP`, `RT`, `Reflexion`, `SelfCheck`, `PHP`, `GLoRe`, `SCoRe`, `RISE`, `CFT`, `ReARTe`, `BackMATH`, `ReST-MCTS*`, `o1-Coder`, `DECRIM`, `MathMinos`, `MCTSr`, `Refiner`, `LEMA`, `LEVER`, `RAP`, `GenRM`, `Eurus`, `DART-Math`, `V-STaR`, `Qwen2.5Math`, `O1JourneyP2`, `s1`, `PGTS`, `LIMO`, `ITT`, `Sky-T1`, `BOLT`, `RecurrentBlock`, `CodeI/O`, `LTM`s. * **Right (Extensive Exploration - continued):** `PathFinder`, `GRPO`, `StepCoder`, `ToT` (Tree-of-Thoughts), `Residual-EBM`, `ReMax`, `RBF`, `CPL`, `AlphaLLM`, `MindStar`, `GoT` (Graph-of-Thoughts), `Inference Scaling Law`, `SSC-CoT`, `PPO-MCTS`, `DBS`, `OSCA`, `FoT`, `AgentQ`, `MAGIcoRe`, `MultiPoT`, `OmegaPRM`, `LE-MCTS`, `CoMCTS`, `AFlow`, `LlamaBerry`, `SPaR`, `rStar-Math`, `MCTS-AHD`, `QLASS`, `CoAT`, `CISC`, `SETS`, `ECM`, `ReasonFlux`, `TestNUC`, `MM-Verify`, `ARIES`, `ReVISE`, `S²R`, `URSA`, `CTRL`, `PRIME`, `COT STEP`, `EvalPlanner`, `AceCoder`, `AGSER`, `Step-KTO`, `AutoPSV`, `CodeRM`, `R³V`, `LLM2`, `RoT`, `BespokeStratos`, `RedStar`, `STILL-2`, `AceMath`, `Coconut`, `OpenAI-o1`, `DeepSeek-R1`. **2024** * **Left (Deep Reasoning):** `PlanningTokens`, `CLP`. * **Center/Right Streams:** Many models from 2023 continue or have derivatives. Notable new entries around the 2024 mark include: `Logic-RL`, `Full-Step-DPO`, `DVO`, `SCIR`, `AgentPRM`, `RefineCoder`, `GSM-Ranges`, `PAD`, `AURORA`, `RewardAgent`, `RLSP`, `OREAL`, `Satori`, `COS(M+O)S`, `Kimi-k1.5`, `DivPO`, `STILL-1`, `Macro-o1`, `REINFORCE++`, `cDPO`, `RBF`. **2025.01** * **Left:** `Coconut`, `OpenAI-o1`. * **Center/Right:** `DeepSeek-R1` (positioned between 2025.01 and 2025.02). **2025.02** * **Left:** `o1-ioi`, `LTM`s. * **Center/Right:** `RedStar`, `BespokeStratos`, `STILL-2`, `AceMath`, `ENVISIONS`. **2025.03 (Top)** * **Left:** `QwQ-32B`. * **Center/Right:** `Logic-RL`, `Full-Step-DPO`, `DVO`, `SCIR`, `AgentPRM`, `RefineCoder`, `GSM-Ranges`, `PAD`, `AURORA`, `RewardAgent`, `ARIES`, `MM-Verify`, `TestNUC`, `ReasonFlux`, `CISC`, `SETS`, `ECM`, `RealCritic`, `ExACT`, `CoR`, `S*`, `AoT`, `C-MCTS`, `QLASS`, `CoAT`, `PF`, `TT`, `CriticQ`, `STeCa`, `DVPO`, `LIMR`, `RFTT`, `RLSP`, `PRIME`, `OREAL`, `Satori`, `COS(M+O)S`, `Kimi-k1.5`, `ThinkPO`, `SWE-RL`, `Focusd-DPO`. ### Key Observations 1. **Temporal Clustering:** There is a significant density of new methods and models appearing in late 2024 and early 2025, indicating a period of rapid advancement and diversification in AI reasoning research. 2. **Categorical Overlap:** Many models, especially those related to Monte Carlo Tree Search (MCTS) variants (e.g., `ReST-MCTS*`, `CoMCTS`, `PPO-MCTS`), appear in the "Extensive Exploration" category but have connections to other streams. 3. **Prominent Organizations:** Icons for Google (G), OpenAI, Anthropic (A), and various academic institutions (e.g., CMU, Stanford, Tsinghua) are frequently attached to nodes, showing the key players in this field. 4. **Evolutionary Lines:** The connecting lines show clear evolutionary paths. For example, foundational methods like `CoT` and `STaR` at the root branch into numerous specialized techniques over time. 5. **Resource Type Distribution:** Both open-source (yellow border) and closed-source (grey border) models are intermixed throughout the taxonomy, suggesting parallel development in both domains. ### Interpretation This diagram serves as a **conceptual map of the "reasoning" sub-field within large language model (LLM) research**. It visually argues that progress is not linear but occurs along multiple, semi-parallel tracks: 1. **Deep Reasoning:** Focuses on improving the model's internal reasoning format and learning processes (e.g., specialized training for math, code). 2. **Feasible Reflection:** Centers on mechanisms for models to critique, verify, and refine their own outputs using feedback loops. 3. **Extensive Exploration:** Emphasizes search and exploration strategies (like tree search, MCTS) to navigate large solution spaces, often leveraging external tools or verifiers. The **central pink tree** is a powerful metaphor, suggesting that the diverse "branches" of modern reasoning techniques all grow from a common trunk of early foundational work (`CoT`, `STaR`). The explosion of branches in 2024-2025 highlights the field's shift from proving the viability of reasoning in LLMs to aggressively scaling and specializing these capabilities. The intermingling of open and closed resources indicates a vibrant, competitive ecosystem where ideas likely flow between academic and industrial labs. The diagram is less a strict hierarchy and more a **phylogenetic chart**, showing ancestry, divergence, and the complex ecosystem of ideas driving AI reasoning forward. </details> Figure 1: Evolution of selected Long CoT over the past three years, where colored branches represent different characteristics: deep reasoning, feasible reflection, and extensive exploration. Each characteristic is further divided into key areas: Deep reasoning includes its format and learning methods. Feasible reflection focuses on feedback and refinement techniques during reflection process as optimization strategies. Extensive exploration addresses scaling, internal, and external exploration as key improvements to Long CoT. However, there is no comprehensive survey to systematically understand the main factors and recent efforts of Long CoT for RLLMs, which hinders the development of RLLMs. As a result, there are ongoing debates about the effectiveness of simple “inference-time scaling” for Longer CoT [864, 486] versus the argument that “over-thinking” from excessively long scaling can harm LLMs and introduce unnecessary complexity [103, 142, 357]. Moreover, some researchers argue that, when solving specific problems, there is no clear relationship between length and accuracy [886]. To address this gap, we provide an extensive and comprehensive survey of Long CoT. Specifically, as illustrated in Figure 2, we first define and examine the distinctions between Long CoT and traditional Short CoT, focusing on the following key aspects: (1) Deep Reasoning, which requires a sufficient depth of logical processing to manage an extensive set of logical nodes; (2) Extensive Exploration, which involves generating parallel uncertain nodes and transitioning from known to unknown logic; and (3) Feasible Reflection, which involves feedback and refinement of logical connections. These characteristics enable Long CoT paradigms to integrate more intricate reasoning and accommodate a broader range of logical structures, ultimately leading to more efficient and coherent outcomes. Subsequently, we systematically explore the underlying explanations for key phenomena associated with Long CoT, such as its emergence, the overthinking phenomenon, inference-time scaling during testing, and the "Aha Moment," among others. To our knowledge, This is the first comprehensive survey dedicated to these specific topics. Finally, considering the extensive body of literature, we highlight promising areas for future research and suggest valuable open-resource frameworks and datasets that can serve as a foundation for future investigations. The main contributions of this work are as follows: - Systematic Distinction: In this work, we first introduce the concept of Long CoT reasoning and distinguish it from the traditional Short CoT, thereby providing a clear framework for understanding both paradigms and their respective characteristics. - Explanation of Hot Phenomena: We systematically investigate the notable phenomena associated with Long CoT reasoning, such as overthinking, inference-time scaling, and the “Aha Moment”, offering valuable insights into the cognitive processes involved in complex reasoning. - Emerging Challenges and Frontiers: We explore the emerging challenges within the field of Long CoT reasoning and identify key research frontiers. Given the vast body of literature, we highlight areas where further inquiry could significantly advance the development of Long CoT methodologies. <details> <summary>x4.png Details</summary>  ### Visual Description ## Diagram: Reasoning Approaches for a Number Theory Proof ### Overview This image is a conceptual diagram illustrating different reasoning methodologies for solving a specific mathematical proof problem. It contrasts a limited "Short Chain of Thought (CoT)" approach with a more robust "Long CoT" approach, which is built upon three key characteristics: Deep Reasoning, Extensive Exploration, and Feasible Reflection. The diagram uses a combination of flowcharts, cartoon characters (stylized seals), and text boxes to explain the processes. ### Components/Axes The diagram is organized into three main vertical sections, flowing from left to right: 1. **Left Section: "Short CoT"** * **Label:** "Short CoT" (top-left). * **Visual:** A simple, linear flowchart of circles (nodes) connected by downward arrows. * **Outcome:** The flow ends at a red circle with a white "X", labeled "Limited Reasoning Boundary" in red text. This indicates failure or a dead end. 2. **Central Section: "Three Key Characteristics"** * **Main Title:** "Three Key Characteristics" (centered at top). * **Sub-components (three boxes):** * **Top-Left Box: "Deep Reasoning"** * **Label:** "Deep Reasoning". * **Visual:** A more complex, multi-path flowchart. A character is shown at the top. The flow splits and converges, ending at a green circle with a white checkmark. A dashed blue line separates an upper "Expanded Reasoning Boundary" from the lower successful path. * **Top-Right Box: "Extensive Exploration"** * **Label:** "Extensive Exploration". * **Visual:** A tree diagram with a root node branching into three child nodes (colored with yellow diagonal stripes). A character with a magnifying glass observes. The text "Possible Reasoning Paths" is written below the tree. * **Bottom Box: "Feasible Reflection"** * **Label:** "Feasible Reflection". * **Visual:** A cyclical diagram showing a process of "Feedback" and "Refine". It starts with a red "X", moves through a green checkmark node, and loops back. A character is shown thinking with a lightbulb. 3. **Right Section: "Long CoT" and Proof Details** * **Left Part: "Long CoT" Flowchart** * **Label:** "Long CoT" (top). * **Visual:** An integrated flowchart combining the three characteristics. It shows a sequence: **Reflection** (green hatched circle) -> **Exploration** (yellow striped circles) -> **Deep Reasoning** (blue hatched circles), ending with a green checkmark. The words "Reflection", "Exploration", and "Deep Reasoning" are placed next to their respective stages. * **Right Part: Text Box with Mathematical Reasoning** * This box contains the transcribed text of the reasoning process. It is divided into three colored sections corresponding to the stages in the adjacent flowchart. ### Detailed Analysis / Content Details **Transcription of the Mathematical Text Box (Rightmost Element):** * **Green Section (Reflection):** * "Assume m=kn, then m²+1=k²n²+1, k²n² is divisible by n, but k²n² + 1 may not be." * "Feedback: Direct construction isn't viable." * "Refine: Need to find m such that m² ≡ -1 (mod n) ..." * **Yellow Section (Exploration):** * "1. Analysis of prime cases using..." * "3. Analysis of equation solvability using ..." (Note: The numbering jumps from 1 to 3; point 2 is not visible in this excerpt). * **Blue Section (Deep Reasoning):** * "Choose Quadratic Residue Theory path:..." * "Therefore, for any positive integer n, there exists a positive integer m such that m² + 1 ≡ 0 (mod n)." **Visual Metaphors and Flow:** * The cartoon seal characters are used consistently to represent the "reasoner" at different stages of the process. * **Color/Pattern Coding:** * **Green (solid/hatched):** Associated with "Reflection" and successful outcomes (checkmarks). * **Yellow (striped):** Associated with "Exploration" and possible paths. * **Blue (hatched):** Associated with "Deep Reasoning" and the final proof steps. * **Red:** Associated with failure or limitations (the "X" in Short CoT and the starting point in Feasible Reflection). * **Spatial Grounding:** The legend (color/pattern coding) is consistently applied across the "Three Key Characteristics" boxes and the integrated "Long CoT" flowchart. The text box on the far right uses the same color scheme for its section headers, creating a direct visual link between the abstract process and the concrete mathematical steps. ### Key Observations 1. **Progression of Complexity:** The diagram clearly shows a progression from a simple, linear, and ultimately failing model (Short CoT) to a complex, iterative, and successful model (Long CoT). 2. **Integration is Key:** The "Long CoT" is not just a longer sequence; it is the deliberate integration of the three core characteristics (Reflection, Exploration, Deep Reasoning) shown in the central panel. 3. **Mathematical Content:** The text box provides a genuine, albeit abbreviated, glimpse into the proof strategy for the stated number theory problem. It moves from a failed initial assumption (direct construction) to a refined modular arithmetic approach, mentions exploratory analysis (prime cases, solvability), and concludes with a specific theoretical path (Quadratic Residue Theory). 4. **Visual Storytelling:** The use of characters, flowcharts, and color creates a narrative about the problem-solving journey, making an abstract concept more accessible. ### Interpretation This diagram serves as a pedagogical or conceptual model for advanced problem-solving, using a number theory proof as its concrete example. It argues that tackling complex problems requires more than a straightforward, step-by-step approach (Short CoT). Instead, success depends on a cyclical and integrated methodology that includes: * **Reflection:** Critically evaluating initial assumptions and feedback to refine the approach. * **Exploration:** Actively generating and considering multiple potential solution paths. * **Deep Reasoning:** Applying sophisticated theoretical knowledge to construct a rigorous final proof. The underlying message is that robust reasoning is non-linear, involves backtracking and refinement, and leverages both creative exploration and deep domain expertise. The specific mathematical problem is used to ground these abstract reasoning characteristics in a tangible context, demonstrating how each characteristic manifests in an actual proof attempt. The diagram essentially maps a metacognitive strategy onto a technical domain. </details> Figure 2: The differences between advanced Long CoT and traditional Short CoT are characterized by three key characteristics: deep reasoning, feasible reflection, and extensive exploration. Moreover, Long CoT integrates all these characteristics to achieve substantial logical efficacy. ## 2 Discussion of Long CoT v.s. Short CoT This section formalizes the key differences between Long Chain-of-Thought (Long CoT) and Short Chain-of-Thought (Short CoT), emphasizing reasoning depth, revisiting connections, and logical node exploration [858]. These distinctions are clearly separate from System 1 and System 2 thinking. The comparison between Long CoT and Short CoT is framed within System 2, with Long CoT involving more thorough reasoning, reflection, and exploration, while Short CoT generally prioritizes shallow and efficient logic over exhaustive reasoning. forked edges, for tree= grow=east, reversed=true, anchor=base west, parent anchor=east, child anchor=west, base=left, font=, rectangle, draw=hidden-black, rounded corners, align=left, minimum width=4em, edge+=darkgray, line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=4pt, line width=1.1pt, ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center, , where level=1text width=9em,font=,, where level=2text width=9em,font=,, where level=3text width=9.5em,font=,, where level=4text width=52em,font=,, [Taxonomy, ver [ Deep Reasoning for Long CoT (§ 4),ver [ Deep Reasoning Formation (§ 4.1) [ Natural Language Deep Reasoning [e.g., CoT [836], Reflection of thought [1118], MathPrompter [303], CLP [617], AutoCAP [1070], Plan- Search [765], NaturalProgram [460], CodeI/O [401], Wang et al. [823], etc., leaf, text width=42em] ] [ Structured Language Deep Reasoning [e.g., PoT [100], CoC [389], Brain [107], SIaM [978], ENVISIONS [897], SKIntern [443], QuaSAR [634], TinyGSM [464], MathDivide [687], Payoungkhamdee et al. [591], GPT-f [605], STP [158], etc., leaf, text width=42em] ] [ Latent Space Deep Reasoning [e.g., Quiet-STaR [1013], PlaningToken [810], Coconut [236], RecurrentBlock [204], MuSR [684], SERT [1069], Heima [662], LTMs [356], ITT [109], Deng et al. [151], etc., leaf, text width=42em] ] ] [ Deep Reasoning Learning (§ 4.2) [ Deep Reasoning Imitation [e.g., GSM8K [141], AceMath [500], DART-Math [738], O1-Journey-P2 [299], STILL-2 [550], LIMO [967], s1 [560], RedSTaR [902], Fine-tune-CoT [256], CoT-Collection [352], FastMCTS [410], etc. , leaf, text width=42em] ] [ Deep Reasoning Self-Learning [e.g., STaR [1012], ReST [225], DynaThink [574], V-STaR [262], PGTS [422], CPO [1065], TPO [446], OpenRFT [1077], MCMC-EM [258], BOLT [581], Weak2Strong [948], Iterative RPO [582], etc. , leaf, text width=42em] ] ] ] [ Feasible Reflection for Long CoT (§ 5), ver [ Feedback (§ 5.1) [ Overall Feedback [e.g., Self-Critique [645], SelfCheck [549], Self-Verification [848], Critic-CoT [1109], Verifier [141], STaR [1012],ReST [225],Critic [216], ReFT [745], AceCoder [1014], DeepSeek-R1 [227], Critic-RM [991], o1- coder [1076], RM [536], Logic-RL [886], Self-Contrast [1062],AGSER [484],DeepSeek-Math [658], etc. , leaf3, text width=42em] ] [ Process Feedback [e.g., ReAct [956],Reflexion [669],Math-Minos [196],Math-Shepherd [792], ER-PRM [1036],Eurus [1001], PAD [219], PAVs [651], CTRL [891], QwQ [731], Skywork-o1 [570], AceMath [500], PRIME [143], AURORA [710], RewardAgent [594], PDS [910], COT STEP [135], Step-DPO [365], ORPS [996], etc. , leaf3, text width=42em] ] [ Hybrid Feedback [e.g., Step-KTO [454], Zhang et al. [1078], etc. , leaf3, text width=42em] ] ] [ Refinement (§ 5.2) [ Prompt-based Refine- ment Generation [e.g., Reflexion [669], SelfCheck [549], Self-Critique [645], Self-Refine [539], Refiner [590], MCTSr [1033], ReST-MCTS* [1032],LLM2 [930], GLoRe [238], Yang et al. [950],RR-MP [240], PAD [578], De- CRIM [186],StepCo [868], BackMATH [1055], ReARTeR [703], START [289], PHP [1101], etc. , leaf3, text width=42em] ] [SFT-based Refinement Imitation [e.g., RealCritic [714], Self-Debugging [104], ProgCo [682], S3c-Math [924], Math-Minos [196], LEMA [14], Reflection-tuning [406], CFT [822], Chen et al. [99], Xi et al. [873], rStar [615], RISE [627],R3V [120], MM-Verify [697], URSA [519], SIMA [813], CoT-based Synthesizer [1027], Qin et al. [623], etc. , leaf3, text width=42em] ] [ RL-based Refinement Learning [e.g., SCoRe [358], DeepSeek-r1 [227], Zeng et al. [1017], S 2 R [529], ReasonFlux [935], ReVISE [374], ARIES [1018], etc. , leaf3, text width=42em] ] ] ] [Extensive Exploration for Long CoT (§ 6), ver [ Exploration Scaling (§ 6.1) [ Vertical Scaling [e.g., OpenAI [307], Fu et al. [189], Jaech et al. [307], s1 [560], Geiping et al. [204], ITT [109], METAL [383], RaLU [391], Ji et al. [314], etc., leaf2, text width=42em] ] [ Parallel Scaling [e.g., Self-Consistency [816],Inference Scaling Law [864], Liu et al. [486], Zhao et al. [1083],WoT [1071], DIVERSE [425], DnA-Eval [408], SSC-CoT [1098], ExACT [988], Kim et al. [354], S* [392], CISC [716], OmegaPRM [517], AFT [421],Seed-CTS [770],CLSP [617], MultiPoT [520], ECM [93], etc. , leaf2, text width=42em] ] ] [ Internal Exploration (§ 6.2) [ RL Strategies [e.g., PPO [648], GRPO [658], REINFORCE++ [265], OREO [773], DAPO [476], LIMR [420] DAPO [476], LIMR [420], TRPO [648], DVPO [277], RPO [699],PRIME [143], DivPO [369],COS(M+O)S [542],CPL [801], Focused-DPO [1043],RFTT [1046],OREO [773], DeepSeekMath [658],TPO [942], etc. , leaf2, text width=42em] ] [ Reward Strategies [e.g., DeepSeek-R1 [227],Kimi-k1.5 [722], T1 [264],ReST-EM [674],SWE-RL [841], DeepScaleR [518], ReST-MCTS* [1032], rSTaR-Math [222], Logic-RL [886], OREAL [522], StepCoder [161], RLSP [962], Verifier [141], TS-LLM [755], STeCa [768], OREO [773], Chu et al. [137], Shen et al. [661], etc. , leaf2, text width=42em] ] ] [ External Exploration (§ 6.3) [ Human-driven Exploration [e.g., SPaR [118], Forest-of-thought [54],Scattered ForestSearch [448], Kang et al. [339],AlphaLLM [737], PATHFINDER [213],Least-to-Most [1117], ToT [955], TreeBoN [625], CodeTree [400], Tree-of-Code [565] TouT [556], GoT [48], GraphReason [64], Besta et al. [49], AoT [733], Chen et al. [95], etc. , leaf2, text width=42em] ] [ Model-driven Exploration [e.g., DBS [1142], Lehnert et al. [378], MindSTaR [338], Residual-EBM [901], Mulberry [952], C-MCTS [453], PPO-MCTS [478], Llama-Berry [1034], Marco-o1 [1095], AtomThink [879], Puri et al. [612], LE- MCTS [586], rStar-Math [222], MC-NEST [629], CoAT [575], CoPlanner [764], CritiQ [228], etc. , leaf2, text width=42em] ] ] ] ] Figure 3: Taxonomy of Long CoT, which includes deep reasoning, feasible reflection, and extensive exploration methodologies. ### 2.1 Overview of Short CoT As illustrated by Figure 2, Short CoT is typically characterized by a shallow, linear reasoning process, where conclusions are drawn sequentially, often relying on a limited number of logical nodes [551]. This reasoning is usually rapid and straightforward, with simple, surface-level transitions and minimal exploration of alternative paths, which restricts its generalizability [683]. Formally, given a reasoning model $R$ , we can define the rationale of Short CoT ( $\texttt{CoT}_S$ ) as follows: $$ \texttt{CoT}_S=R(\{n_i\}^k_i=1|(k≤B_s)\wedge( j=1⇔∀ i≤k,n_i→n_i+j)\wedge( ∀ i≠j≤k,n_i≠n_j)), \tag{1} $$ where $n_1$ to $n_k$ represent a sequence of logical nodes, which naturally satisfy that $∀ i,n_i→ n_i+1$ . Here, $B_s$ denotes the upper boundary on the number of logical nodes, as defined by Chen et al. [90]. In this paradigm, the reasoning progresses sequentially from one node to the next, with minimal revisitation of previous nodes and little exploration of alternative logical paths. ### 2.2 Overview of Long CoT In contrast, Long CoT involves deeper reasoning, reflective analysis, and a broader exploration of logical structures. It facilitates reasoning across a wider range of logical steps, addressing both known and unknown elements of a problem [194, 858]. Building on this, Long CoT expands upon the constraints presented in Equation 1 based on tree structures by incorporating three critical components: deep reasoning, exploration, and reflection. These components play distinct yet complementary roles in enhancing cognitive processes. Deep reasoning ensures each logical step is executed rigorously, even within complex structures, fostering robust logic across intricate relationships. Exploration encourages the identification of new pathways, revealing potential avenues that may not be immediately obvious. Reflection enables iterative analysis and reassessment of conclusions, allowing reasoning to evolve throughout problem-solving. By distinguishing these three categories, Long CoT enhances its ability to address a broader range of problems with precision and depth. As shown in Figure 3, we will now discuss these key differences in detail. #### 2.2.1 Deep Reasoning for Long CoT As shown by Figure 2, deep reasoning refers to the capability to perform deep and thorough logical analysis across multiple interconnected logical nodes, where Short CoT generally can never achieve. This capability is essential when tackling complex problems that require a massive number of logical deductions to arrive at a valid conclusion. To better define and understand deep reasoning, we frame it as a capability that primarily relaxes the first constraint in Equation 1, as expressed by the following: $$ k≤B_s→tail k≤B_l\wedgeB_s \llB_l, \tag{2} $$ where $B_l$ represents the upper boundary for Long CoT reasoning, which can accommodate much more intricate logical nodes compared to the smaller boundary $B_s$ for Short CoT. The larger boundary $B_l$ alleviates issues related to insufficient depth in reasoning, thereby reducing the risk of generating unresolved answers or hallucinated responses in short-form reasoning. Key Difference: Reasoning Depth • Short CoT typically addresses a limited set of logical nodes, involving shallow reasoning, and struggles with problems requiring complex or intricate logical structures. • Long CoT is designed to accommodate a significantly larger set of logical nodes, allowing for deeper logic and more thorough analysis during the reasoning process. #### 2.2.2 Extensive Exploration for Long CoT As shown by Figure 2, Long CoT encourages branching out to extensively explore uncertain or unknown logical nodes, thereby expanding the potential set of reasoning paths. This exploration is particularly critical when solving problems characterized by ambiguity, incomplete information, or multiple possible solutions [43, 1016, 871]. More specifically, we describe how extensive exploration primarily addresses the relaxation of the second constraint in Equation 1, which can be formalized as follows: $$ j=1⇔∀ i≤ k,n_i→ n_i+j→tail ∃ m,∀ i,∀ j≤ m,n_i→ n_i+j, \tag{3} $$ where the condition indicates that for a logical node $n_i$ , there are $m$ nodes that are explored in parallel. The acceptability of parallel exploration allows for a more systematic approach, enabling the exploration of previously unconsidered logical paths. This, in turn, helps maximize the understanding of all possible solutions, ultimately leading to the correct final answer. Key Difference: Exploration of Logical Nodes • Short CoT generally restricts exploration to a fixed set of logical nodes, often resulting in oversimplified reasoning and limited exploration. • Long CoT explores more various paths, including uncertain or uncharted areas, fostering more nuanced and comprehensive problem-solving. #### 2.2.3 Feasible Reflection for Long CoT As shown by Figure 2, Long CoT involves revisiting previous logical nodes to verify their connections are valid and accurate, and then correcting them or selecting an alternative logical path. Formally, feasible reflection relaxes the third constraint in Equation 1, which originally requires acyclic reasoning such that $n_i≠ n_j$ for all $i≠ j≤ k$ . In contrast, feasible reflection permits the reasoning path to return to a previously visited node, captured as: $$ ∀ i≠ j≤ k,n_i≠ n_j→tail∃ i<j≤ k,n_i=n_ {j}, \tag{4} $$ where this condition indicates that, for a logical node $n_j-1$ , the subsequent node is not limited to the original next node $\hat{n}_j$ . Instead, it may transition to $n_i$ (i.e., the next logical node becomes $n_j$ , where $n_j=n_i$ ). Practically, reflection implementation consists of two components: Feedback refers to evaluating both overall and intermediate outputs for correctness and quality, also known as critique or verification. It can be derived from external sources, validation checks, or by reflecting on prior conclusions within the reasoning process. Formally, at each step $n_i$ , a verification process $V_i$ ensures the correctness, feasibility, and consistency of the reasoning. If an issue is identified, the process redirects $n_i$ to the nearest correct node $n_j$ , where $j<i$ . This relationship is formalized as: $$ F_i,n_j←Feedback(\texttt{CoT}_L^i) \tag{5} $$ where $\texttt{CoT}_L^i=\{n_1,\dots,n_i\}$ represents the current logical path up to the $i$ -th logical node for Long CoT. Refinement involves adjusting intermediate steps or modifying the logical flow to correct inconsistencies or address gaps based on the given feedback. This process can be expressed mathematically as follows: $$ \widetilde{n}_i+1←Refine(n_i+1|\texttt{CoT}_L^i, F_i,n_j), \tag{6} $$ where $\widetilde{n}_i+1$ represents the refined version of the subsequent logical node $n_i+1$ , according to the current logic $\texttt{CoT}_L^i$ , feedback result $F_i$ , and previous logical node $n_j$ . Overall, incorporating reflection ensures that errors are identified and corrected promptly. This capability enables LLMs to quickly shift to alternative reasoning paths or correct their current trajectory. By doing so, error propagation is minimized, resulting in more accurate conclusions. Key Difference: Feedback & Refinement • Short CoT typically moves in a straightforward, non-repetitive manner from one node to the next, so that cannot correct their logic. • Long CoT allows for revisiting and revising earlier decisions by feedback and refinement, ensuring that optimizable and prior logical conclusions during the reasoning progress. #### 2.2.4 Unified Application and Development History of Three Capabilities The Long CoT discussed here represents a unified reasoning system that seamlessly integrates and applies three key capabilities: deep reasoning, reflective mechanisms, and exploration capabilities. In contrast, during the Short CoT era, these capabilities developed independently, each evolving in isolation. As shown in Figure 2, early efforts primarily focused on enhancing deep reasoning within traditional CoT paradigms. This was followed by the gradual introduction of reflective mechanisms, which were initially based on human-designed pipelines. Over time, exploration capabilities were added, and these components were ultimately merged, giving rise to the modern concept of Long CoT, a unified approach to reasoning that seeks to enhance all three capabilities in harmony. The progression of Long CoT is gradual, rather than a sudden emergence through isolated models like o1 [307] and R1 [227]. Instead, it develops gradually. For example, earlier systems, such as ToT [955], enhance exploration but lack reflective mechanisms, disqualifying them as Long CoT [95]. While GoT [48] incorporates self-reflection based on ToT, its original model still lacked robust deep reasoning, preventing it from qualifying as Long CoT at that time. It is also notable that modern Long CoT systems, often neglect earlier technologies. This article addresses this gap by tracing the evolution of each capability, with the final section offering a comprehensive analysis of the integrated Long CoT system. In summary, Long CoT and Short CoT represent distinct paradigms. Long CoT features a deeper, broader, and more reflective reasoning process, enhancing both accuracy and coherence. Short CoT, by contrast, is better suited to simpler, well-defined problems. This distinction highlights the scalability and adaptability of Long CoT, making it particularly effective for more complex reasoning. Key Difference: Unified Application of Three Capabilities It is important to highlight that Long CoT integrates these three distinct capabilities to perform complex reasoning. In contrast, traditional Short CoT optimization typically focuses on only one of these characteristics. ## 3 Long CoT Analysis & Evaluation ### 3.1 Analysis & Explanation for Long CoT Research on Long CoT has significantly enhanced RLLMs by improving reasoning accuracy, reducing errors, and supporting dynamic decision-making. However, several phenomena and their corresponding mechanisms remain inadequately summarized. This section addresses key topics, including the mechanisms of Long CoT and their underlying principles [644, 63, 545, 642]. Methodologically, two main perspectives have emerged to explain Long CoT: (1) External Behavior Analysis (§ 3.1.1) and (2) Internal Mechanism Analysis (§ 3.1.2). #### 3.1.1 Long CoT External Behavior Analysis The primary research stream focuses on explaining RLLM behaviors for Long CoT [25]. As illustrated in Figure 4, six key phenomena are identified and discussed for Long CoT in this part. Long CoT Emergence Phenomenon Research shows that contextual examples improve large models’ generative abilities by guiding the formation of reasoning chains [1012, 671, 417, 343, 532, 846, 1017, 1141]. Wang et al. [759] and Lippmann and Yang [461] demonstrate that these examples standardize reasoning chain generation relevant to the answers both in in-context-learning and supervised-finetuning. In an experiment by Madaan et al. [538], removing problem-specific entities from contextual examples, while retaining only the logical structure, led to similar performance as using complete examples, highlighting the logical structure imitation of Long CoT during inference. From a learning perspective, Ye et al. [963] analyzes and reveals the three-stage developmental trajectory of Long CoT: early memorization, followed by in-distribution generalization, and ultimately cross-distribution generalization, thereby enabling the model to exhibit Long CoT capabilities. More recently, Stechly et al. [688] and Wang and Zhou [815] have shown that modifying the decoding process or designing specific prompts can activate the Long CoT within pre-trained models. They propose that CoT is embedded during pre-training and requires specific activation [941]. Further, Sadr et al. [642] focus the Long CoT source from the training data, and build on this with the notion of “model attribution”, to specifically identify the training data most influential for specific outputs. Building on this, Guo et al. [227] and Xie et al. [886] investigate using rule-based reinforcement learning to directly activate Long CoT during pre-training, aiming to enhance performance [881]. Furthermore, Gandhi et al. [194] identify four key cognitive behaviors, including verification, backtracking, sub-target setting, and backward chaining, which successfully facilitate Long CoT. Qwen series models [926] inherently demonstrate these behaviors, which can be easily triggered by rule-based reinforcement. In contrast, the models of Llama series [168] lack these capabilities and thus requires example-based reinforcement learning to improve significantly [65]. Moreover, Wang et al. [812] identify a pretraining scaling law that explains how increasing calculation size in RLLMs enhances their reasoning capabilities. Wang et al. [796] further explore the scaling law of Long CoT, showing that more fine-grained Long CoT granularity leads to more efficient and effective generalization performance. <details> <summary>x5.png Details</summary>  ### Visual Description ## Multi-Panel Diagram: AI Reasoning Phenomena ### Overview The image is a composite diagram consisting of six distinct panels, labeled (a) through (f), each illustrating a different conceptual phenomenon related to reasoning in artificial intelligence systems. The panels use a consistent cartoon character—a worm-like creature wearing a red scarf—as an agent to metaphorically depict these abstract concepts. The overall style is illustrative and metaphorical rather than a data chart. ### Components/Axes The image is segmented into six rectangular panels arranged in a 2x3 grid (two rows, three columns). Each panel has a dashed border and a title at the top. **Panel Titles (in order):** * (a) Reasoning Emergence Phenomenon * (b) Reasoning Boundary Phenomenon * (c) Overthinking Phenomenon * (d) Inference-Time Scaling Phenomenon * (e) PRM & ORM Selection Phenomenon * (f) Aha Moment Phenomenon **Key Visual Components Across Panels:** * **Agent:** A cartoon worm character with a red scarf, used to represent an AI model or reasoning entity. * **Process Arrows:** Yellow block arrows indicating flow or transformation between states. * **Metaphorical Objects:** Plants, rulers, swords, magnifying glasses, stars, and neural network icons used as symbols. * **Text Labels:** Embedded within each panel to name processes, states, or concepts. ### Detailed Analysis **Panel (a): Reasoning Emergence Phenomenon** * **Left:** The agent looks curious. * **Center:** A green box labeled **"Training"** containing a plant with three growing stalks, symbolizing growth or development. * **Right:** The agent appears satisfied, with small flower icons around it, suggesting a positive outcome or capability emergence. * **Flow:** Agent → Training → Enhanced Agent. **Panel (b): Reasoning Boundary Phenomenon** * **Left:** The agent looks on. * **Center:** A blue box labeled **"Measure"** containing a ruler and a set square. * **Right:** The agent holds a large, vertical, tapered object (like a sword or spike). A vertical double-headed arrow next to it is labeled **"Reasoning Boundary"**, indicating the extent or limit of its reasoning capability is being measured. * **Flow:** Agent → Measure → Agent with Defined Reasoning Boundary. **Panel (c): Overthinking Phenomenon** * **Top Left:** The agent holds a short version of the tapered object. A vertical double-headed arrow next to it is labeled **"Reasoning Boundary"**. * **Top Right:** The agent holds a much longer version of the same object. The "Reasoning Boundary" arrow is correspondingly longer. The agent's eyes are replaced with 'X's, suggesting exhaustion or negative consequence from the extended boundary. * **Bottom:** A purple box labeled **"Scaling"** with an icon of a ruler being stretched. A yellow arrow points from the top-right scene to this box. * **Flow:** Short Boundary → Scaling → Long Boundary (with negative effect). **Panel (d): Inference-Time Scaling Phenomenon** This panel is split into two vertical sub-panels. * **Left Sub-panel (Sequential Scaling):** Shows two agents stacked vertically, each holding a tapered object. A curved yellow arrow labeled **"Sequential Scaling"** loops from the bottom agent back to the top, with a small icon of a person climbing stairs inside the arrow. * **Right Sub-panel (Parallel Scaling):** Shows three agents side-by-side, each holding a tapered object. The word **"Sampling"** appears above each agent. Below them is a blue box labeled **"Parallel Scaling"** with three downward arrows (↓↓↓). * **Flow:** Illustrates two methods for scaling reasoning at inference time: one after another (sequential) or multiple at once (parallel sampling). **Panel (e): PRM & ORM Selection Phenomenon** * **Left:** The agent holds a magnifying glass over a small, complex object (possibly a problem or state). * **Center:** Two green boxes. The top box is labeled **"ORM"** and contains a single large star. The bottom box is labeled **"PRM"** and contains three smaller stars on a ribbon. * **Right:** The agent holds the magnifying glass over the complex object again, with a yellow arrow pointing from the PRM box to this final state. * **Flow:** Agent examines problem → Considers ORM (single outcome) vs. PRM (multiple process steps) → Selects PRM for final evaluation. **Panel (f): Aha Moment Phenomenon** * **Left:** The agent looks puzzled. * **Center:** A pink box labeled **"RL Train"** (Reinforcement Learning Train) containing a neural network diagram with a plus sign (++). * **Right:** The agent holds a magnifying glass, looking enlightened. A speech bubble says: **"Aha! There is a mistake here!"**. * **Flow:** Puzzled Agent → Undergoes RL Training → Achieves Insight ("Aha Moment") to identify errors. ### Key Observations 1. **Consistent Metaphor:** The "tapered object" (sword/spike) is a recurring visual metaphor for the "Reasoning Boundary" or the scope of reasoning capability. 2. **Negative Connotation of Overthinking:** Panel (c) explicitly links scaling the reasoning boundary (overthinking) to a negative state (agent with 'X' eyes). 3. **Process vs. Outcome:** Panel (e) visually distinguishes between Outcome Reward Models (ORM, single star) and Process Reward Models (PRM, multiple stars on a path), suggesting PRM evaluates steps rather than just the final result. 4. **Training as Transformation:** Panels (a) and (f) frame training ("Training", "RL Train") as a transformative process that changes the agent's state or capabilities. ### Interpretation This diagram serves as a conceptual framework for understanding challenges and phenomena in training and deploying reasoning-focused AI models. * **Core Narrative:** It traces a potential journey: reasoning capabilities **emerge** through training (a), but must be **measured** (b). Unchecked **scaling** of this capability can lead to detrimental **overthinking** (c). To manage this at deployment, different **inference-time scaling** strategies (sequential vs. parallel) can be employed (d). The choice of evaluation method—focusing on the final outcome (ORM) versus the reasoning process (PRM)—is critical for selection and alignment (e). Ultimately, the goal is to train models that can achieve **insightful "aha moments"** (f), enabling them to self-correct and reason effectively. * **Relationships:** The phenomena are interconnected. The "Reasoning Boundary" measured in (b) is what is scaled in (c) and (d). The "Overthinking" in (c) is a potential pitfall of scaling. The "Selection" in (e) is a method to mitigate poor reasoning, and the "Aha Moment" in (f) represents an ideal outcome of successful training (like RL Train) that avoids such pitfalls. * **Notable Implications:** The diagram suggests that more reasoning (a longer boundary) is not always better and can be harmful. It advocates for careful measurement, controlled scaling, and process-aware evaluation (PRM) to develop AI that doesn't just compute more, but reasons better and can recognize its own errors. The "Aha Moment" is positioned as the pinnacle of this process—a shift from brute-force computation to genuine understanding. </details> Figure 4: Analysis of the six classic phenomena of Long CoT external behavior: (a) emergence of Long CoT in current RLLMs; (b) reasoning boundaries and limitations of current Long CoT systems; (c) overthinking caused by scaling beyond RLLMs’ reasoning boundaries, leading to performance decay; (d) inference-time scaling, discussing mainstream scaling methods, corresponding scaling laws and their limitations; (e) use of process reward model (PRM) or outcome reward model (ORM); (f) exploration of the "aha" moment and its underlying causes. Reasoning Boundary Phenomenon Recent research has highlighted the upper bounds and limitations of RLLMs across various reasoning tasks [303, 283, 684, 261, 185, 252]. Specifically, Bi et al. [53] investigate these bounds in code generation, showing that RLLMs struggle with tasks that exceed certain complexity thresholds [600], especially when imitating Long CoT samples of varying complexity. In the context of upper-bound performance, Merrill and Sabharwal [548] and Li et al. [430] focus on single-step arithmetic tasks, concluding that model performance is constrained by input length. Moreover, Feng et al. [177] proposes a mathematical model indicating that fixed-size models cannot produce accurate numerical answers beyond specific limits. However, increasing the number of reasoning steps improves a model’s capability requirements to solve more complex problems. Inspired by these explorations, Chen et al. [90] first define the “reasoning boundary” phenomenon and quantify these limits, showing that surpassing an RLLM’s reasoning capacity leads to performance decline [92]. Similarly, Zhou et al. [1130] introduce GSM-Infinite, linking different upper limits to accuracy levels. Chen et al. [90] also examine the interaction between these boundaries across tasks of varying complexity, providing insights into the effectiveness of Long CoT strategies [1085]. Moreover, Amiri et al. [12] propose a “tight lower bound” fo r Long CoT further guiding reasoning error reductions. Further, Baeumel et al. [28] suggest that due to its reliance on a single-digit lookahead heuristic, there are inherent boundaries in performing addition with multiple operands, which thus hinders the fundamental limitation of LLMs in scaling to more complex numerical reasoning. Liu et al. [483] further investigate the role of reinforcement learning in expanding these reasoning boundaries instead of relying solely on pretraining capabilities. Overthinking Phenomenon Research has highlighted the overthinking phenomenon [103, 330, 574, 142, 357, 595], where performance improves with longer reasoning chains up to a threshold, after which it declines. In contrast, Xie et al. [886] and Ma et al. [534] find no significant correlation between reasoning length and accuracy. To explain this, one line of research suggests that Long CoT strategies [21, 441], like avoiding “snowball errors” [192]. Alternatively, Chen et al. [90], Wolf et al. [851] highlight a performance drop when the reasoning boundaries are exceeded, providing an explanation for the overthinking phenomenon. This suggests that reasoning length and logical complexity should be kept below a certain boundary [1080]. Building on this, Wu et al. [867] mathematically determine the feasible reasoning length for Long CoT. Finally, Chen et al. [93] introduces Ohm’s law of Long CoT, which accurately predicts and controls performance. Inference-Time Scaling Phenomenon Recent advances in inference-time scaling algorithms [524, 843] have garnered significant attention, particularly for their ability to extend reasoning length and improve performance [524, 455, 875]. Specifically, Brown et al. [57] identify a phenomenon called “Large Language Monkeys”, in which a series of reasoning tasks show that with enough trials, a correct result can be achieved. Additionally, o1 [307] and R1 [227] demonstrated that directly scaling the length of model inference improves final performance. To understand inference-time scaling, we will discuss these two paradigms: (1) Sequential Scaling: Sequential scaling involves increasing the reasoning path length. While this can enhance performance, studies by Jin et al. [330] show that, beyond a certain point, longer reasoning paths can degrade performance due to error accumulation. They suggest an optimal path length that depends on the model’s capabilities and task complexity [15, 652, 31]. Furthermore, Chen et al. [90] and Wu et al. [867] explain that excessive exploration lengths beyond the RLLM’s inherent reasoning boundary lead to performance decay, which guides RLLMs for deeper reasoning capabilities [32]. (2) Parallel Scaling: Parallel scaling involves performing multiple reasoning steps and verifying the results. While it shows promise, Parashar et al. [583] and Wang et al. [820] argue that simply increasing inference time does not guarantee improved performance. Wu et al. [864] show that the computational FLOPs $N$ of inference are correlated with the lower bound of performance error, which scales with $\log N$ . Additionally, Chen et al. [93] establish an upper bound for parallel scaling, showing that RLLMs cannot exceed Pass@k verification through various verifiers. They further argue that sampling optimization cannot exceed the model’s internal reasoning limitations, demonstrating that for $N$ samples, accuracy is proportional to $\frac{m}{(k/\log N+b)^2}$ , where $m$ , $n$ , and $b$ are model-dependent constants. PRM & ORM Selection Phenomenon As RLLMs evolve, it is crucial to navigate the debate around the selection between process supervision and outcome supervision, two key reinforcement learning paradigms for complex reasoning tasks. The phenomenon of choosing between these two approaches has become a pivotal issue, as it is essential to differentiate and decide which supervision strategy is more suitable for specific tasks [899, 187, 1059]. While process supervision is intuitively advantageous for long-term reward assignments, the exact relationship between the two approaches remains unclear. It is commonly believed that process supervision is more challenging due to the trajectory-level coverage problem, which demands significant effort to collect fine-grained supervision data [1102, 679]. Additionally, PRM faces the issue of reward hacking [13, 152, 573, 30, 399], where agents exploit flaws in the reward function to produce unintended behaviors [227]. Addressing this to surpass rule-based reward systems has become an important research area [227, 886, 594]. Furthermore, Lampinen et al. [368] and Tan [708] establish a causal link between intermediate steps and final answers in qualitative experiments. Building on this, Jia et al. [317] demonstrate that, under the standard data coverage assumption, reinforcement learning with outcome supervision is not statistically more challenging than process supervision, aside from polynomial factors. More strictly, He et al. [247] mathematically demonstrate that outcome-level rewards suffice for online reinforcement learning in RLLMs. Aha Moment Phenomenon Earlier, Guo et al. [227] demonstrated that direct RL using rule-based rewards can trigger the aha moment, fostering natural self-reflection without supervision [172]. Following this, Team [721], Xie et al. [886] replicate this phenomenon. Further, Zhou et al. [1119] and Meng et al. [547] further extend this phenomenon to multimodal scenarios. However, Liu et al. [498] argue that the aha moment may not emerge in R1-Zero-like training. Instead, they observe that self-reflection patterns, such as superficial self-reflection (SSR), appear at epoch 0, the stage of base models. In this case, self-reflections do not necessarily lead to correct answers. Upon closer examination of R1-Zero training via RL, they find that the increasing response length results not from self-reflection, but from RL optimizing well-designed rule-based rewards. Moreover, Yang et al. [939] demonstrate that the “aha moment” is externally marked by increased use of anthropomorphic language during self-reflection and a dynamic adjustment of uncertainty in response to problem difficulty. This process enables the model to maintain reasoning without succumbing to "Reasoning Collapse." Internally, it is characterized by a clear distinction between anthropomorphic traits and logical reasoning, with anthropomorphic language intensifying as the problem becomes more complex. Reinforcement Learning Entropy Phenomenon In reinforcement learning for Long CoT, the entropy mechanism is a crucial factor influencing the performance of RLLMs. Policy entropy measures the diversity and exploratory strength of a model’s outputs. By managing this entropy effectively, a model preserves exploration and thus excels on complex reasoning tasks. Earlier, Jang and Kim [310] investigate how initial entropy affects exploration in deep RL and proposed an entropy-aware initialization to encourage effective exploration. Building on this, Zhang et al. [1036] developed an Entropy-Regularized PRM that balances policy updates against large deviations from the starting distribution, thereby improving reasoning. Cheng et al. [116] found that high-entropy regions correlate positively with three exploratory reasoning behaviors: (1) key tokens linking logical steps, (2) self-verification and correction, and (3) rare behaviors underrepresented in the base model. Most recently, Agarwal et al. [5] introduced an Entropy Minimization method and demonstrated its strong impact on LLM performance in mathematical, physical, and coding tasks. However, recent research indicates that, during early training, policy entropy declines sharply, causing the model to converge prematurely on specific output patterns and limiting further reasoning improvement [144]. In reinforcement learning, policy entropy ( $H$ ) and downstream task performance ( $R$ ) follow an exponential relation: $R=-a· e^H+b,$ so a drop in entropy produces a rapid performance decline until saturation. This “policy entropy collapse” is common without entropy control, as reduced entropy constrains exploration and stalls reasoning gains [144]. To counter this collapse, two methods, Clip-Cov and KL-Cov, regulate entropy by constraining updates on high-covariance tokens. Clip-Cov clips their update magnitudes, whereas KL-Cov imposes a Kullback–Leibler penalty. Empirical results show both techniques prevent collapse and enhance reasoning performance [144]. #### 3.1.2 Long CoT Internal Mechanism Analysis The second stream of research investigates the internal mechanisms of Long CoT-related RLLMs. Reasoning Interal Mechanism Recent studies have explored the internal mechanisms underlying the coherent rationale outputs of Long CoT, with particular emphasis on attention mechanisms [675, 632]. These studies primarily examine neural substructures in RLLMs, framing CoT reasoning from a white-box perspective [819, 992, 233, 169]. Weston and Sukhbaatar [849] introduces the concept of System 2 Attention (S2A), which demonstrates Long CoT generation by selectively focusing attention on relevant information. Additionally, Li et al. [407] explore gradient distributions between direct output and Long CoT layers, revealing that Long CoT layers help maintain stability by distinguishing relevant from irrelevant reasoning [840]. Finally, Zhang et al. [1068] conceptualize RLLMs as finite state automata, offering further insight into how internal dynamics influence external behavior. Despite Short CoT’s struggles with self-correction, Bertolazzi et al. [47] show that these models rely on consistency heads (attention heads) to assess the alignment of numerical values in arithmetic solutions through internal shortcuts. Knowledge Incorporating Mechanism Current RLLMs primarily focus on mathematics and coding but have shown potential for generalization to other knowledge-rich domains, sparking growing interest in the mechanism for integrating domain-specific knowledge into Long CoT [860, 886, 1105]. Prystawski et al. [609] suggest that generative models store entity knowledge learned during pre-training independently, with the reasoning process in Long CoT linking this knowledge across entities. Radha and Goktas [630] recently introduced the Probabilistic Mixture Model (PMM), which categorizes model outputs into reasoning, memorization, and guessing. They also propose an Information-Theoretic Consistency (ITC) analysis to quantify the relationship between model confidence and strategy selection. Additionally, Jin et al. [331] define "Concept Depth" as the lowest layers at which complex concepts are understood, demonstrating varying levels of knowledge integration in RLLMs. Ou et al. [572] examine RLLM knowledge internalization through knowledge loop evolution, arguing that new knowledge acquisition is shaped by its connection to existing knowledge, with the loop evolving from formation to optimization and from shallow to deep. ### 3.2 Long CoT Evaluations #### 3.2.1 Metrics In benchmarking, various metrics assess model performance across reasoning tasks, each focusing on different aspects of reasoning ability. These metrics evaluate both RLLMs’ effectiveness in achieving desired outcomes and their learning efficiency. As a result, metrics for RLLMs have gained increasing attention in recent research. For mathematical or code-related tasks, three key metrics are commonly used: Accuracy, Pass@k, and Cons@k based on regex extraction: - Accuracy measures the proportion of correct outputs. - Pass@k evaluates the likelihood of generating at least one correct solution within $k$ attempts. - Cons@k assesses consistency by determining the model’s ability to consistently produce correct or logically coherent solutions across multiple attempts. In scientific or commonsense question-answering tasks, evaluation often uses Exact Match (EM) and Accuracy based on regex extraction, where EM determines whether the model’s output exactly matches the expected solution. For feedback techniques like ORM or PRM, Rank and Best-of-N metrics are often used: - Rank measures whether the reward model correctly prioritizes the best reasoning processes from the top $k$ candidates. - Best-of-N selects the highest-scoring solution from $N$ generated reasoning trajectories, indirectly measuring the reward model’s effectiveness based on final outcomes. #### 3.2.2 Decoding Strategies Decoding strategies are essential for controlling the inference process. Common approaches include Greedy Decoding, Beam Search, and Major@k. Both Greedy Decoding and Beam Search limit the sampling range to reduce randomness, guiding the model toward more consistent outputs. In contrast, Major@k identifies the most reliable solution by selecting the one with the highest consistency from a set of $k$ candidate solutions. #### 3.2.3 Benchmarks In the realm of Benchmarks, the focus lies on assessing the reasoning capabilities of RLLMs across diverse domains. There are two primary categories: (1) Outcome Benchmarks, which focus on the holistic view of Long CoT reasoning, and (2) Process Benchmarks, which concentrate on the local view of the Long CoT process or individual capabilities. Outcome Benchmarks In the realm of Outcome Benchmarks, the first focus lies on evaluating the logical reasoning capabilities: - Complex Mathematics: A central focus in complex mathematics is evaluating benchmarks like GSM8K [141] and MATH [253], which assess basic mathematical problem-solving abilities [1125, 1112]. Recent additions, such as AIME 2024 [8], AIME 2025 [571], MATH-500 [449], AMC 2023 [9], USAMO [598], OlympiadBench [239], and OlympiadArena [298], expand the evaluation of LLM performance in mathematics. Moreover, Putnam-AXIOM [224] and FrontierMath [210] introduce more complex problems that challenge future reasoning systems. Additionally, ThinkBench [291] and MATH-Perturb [288] focus on robust evaluation for Long CoT [38, 987]. - Complex Coding: Complex coding benchmarks are also vital, with competitions like Codeforces, SWEbench [327], CodeContests [427], and LiveCodeBench [309] evaluating LLM coding and problem-solving skills. Notable additions such as MHPP [148], ProBench [934], HumanEval Pro, MBPP Pro [993], and EquiBench [833] enhance the scope and complexity of coding challenges. Moreover, some studies have explored applying these benchmarks in real-world code development scenarios for automatic code generation and evaluation [243, 744]. - Commonsense Puzzle: Commonsense puzzle benchmarks, including LiveBench [850], BIG-Bench Hard [705] and ZebraLogic [450], assess models’ ability to reason about commonsense situations. The ARC [131] and DRE-Bench [947] is often viewed as a challenging commonsense-based AGI test. JustLogic [87] further contributes to the evaluation of deductive reasoning and commonsense problem-solving. Moreover, Li et al. [382] introduce QuestBench, a benchmark designed to evaluate the ability of RLLMs to generate insightful and meaningful questions. The second focus area concerns Knowledge Benchmarks, essential for evaluating a model’s capability in complex reasoning across various tasks for out of distribution evaluation [776]: - Scientific Reasoning: Scientific Reasoning benchmarks, such as GPQA Diamond [637], MMLU-Pro [821], and SuperGPQA [165], assess multi-domain reasoning in fields like chemistry, biology, and physics [157]. These benchmarks test models’ ability to not only accumulate knowledge but also integrate it for problem-solving. Humanity’s Last Exam (HLE) [602] further challenges models by requiring deep interdisciplinary reasoning across scientific disciplines. Further, Chung et al. [140] propose TPBench to evaluate the effectiveness of RLLMs in solving theoretical physics problems. - Medical Reasoning: In the realm of Medical Reasoning, the need for complex, domain-specific, and accurate reasoning is paramount [1094, 1024, 905, 589]. Benchmarks, such as MedQA [328], JAMA Clinical Challenge [76], LLMEval-Med [1049] and Medbullets [76], simulate diagnostic and treatment decision-making processes, reflecting real-world medical practice. These benchmarks evaluate a model’s handling of medical knowledge and reasoning, from diagnosis to treatment planning. Additionally, MedXpertQA [1150] introduces a comprehensive evaluation framework combining text and multimodal data, specifically assessing AI’s reasoning capabilities in healthcare. #### 3.2.4 Process Evaluations Deep Reasoning Benchmarks Recent progress in RLLMs underscores the need for specialized benchmarks to evaluate their deep reasoning abilities in Long CoT [375, 1133]. Notably, Lin et al. [450] introduces ZebraLogic, a framework for assessing logical reasoning, especially in complex non-monotonic scenarios. Similarly, BigGSM [90] and GSM-Ranges [670] focus on perturbing numerical values to test logical and arithmetic reasoning in edge cases beyond the models’ training distribution. ROSCOE [212], ReCEval [606], DiVeRSe [425], HLV [71], and CoT-Kinetics [51] are designed to assess each step in the deep reasoning process during Long CoT tasks. Exploration Benchmarks Several studies assess RLLMs’ exploration capabilities in Long CoT tasks. Specifically, Sys2Bench [583] evaluates the exploration and scaling abilities of RLLMs, emphasizing generalization across diverse tasks. BanditBench [566] extends this by testing model performance in interactive environments, offering insights into practical applications. Additionally, Heyman and Zylberberg [254] introduce a graph coloring problem to assess reasoning and spatial exploration in complex problem-solving scenarios. Reflection Benchmarks Reflection benchmarks measure RLLMs’ ability to identify, reflect upon, and correct errors in Long CoT reasoning. These benchmarks fall into two categories: feedback and refinement. (1) Feedback Benchmark: These benchmarks assess the ability of LLMs to detect errors and respond to feedback for improvement. For example, Lambert et al. [367] introduces RewardBench to evaluate RLLMs’ reward capabilities. This framework is extended by Multimodal RewardBench [960], and CodeCriticBench [1025] to include multimodal and code contexts, respectively. Benchmarks such as ProcessBench [1102], PRMBench [679], MR-Ben [1021], and DeltaBench [250] focus on error detection and correction across various tasks at the step level. Additionally, ReaLMistake [337] and JudgeBench [709] address more real-world error evaluation. (2) Refinement Benchmark: These benchmarks focus on error correction in complex tasks. CriticBench [456] assesses critique-correction capabilities, while MLDebugging [287], and ErrorRadar [922] specializes in coding or multimodal reasoning error detection and refinement. FinerReason [72] introduces a commonsense puzzle for broader feedback and refinement evaluations. Medec [1] adapts error correction to healthcare, addressing medical issues. #### 3.2.5 Advanced Evaluation Agentic & Embodied Reasoning Agentic and Embodied reasoning requires models to demonstrate an understanding of real-world interactions, tool use, and adaptive reasoning in response to change. To assess real-world understanding, Wang et al. [798] introduce a benchmark that evaluates agents’ ability to reason about physical concepts. Zhang et al. [1064] extend this by assessing agents’ interactions with real-world physics. Additionally, realistic tasks often demand complex planning and tool usage, necessitating benchmarks to evaluate agent reasoning. These benchmarks assess agents’ abilities to navigate and complete tasks in digital environments. Building on this, Huang et al. [283] propose a framework for evaluating decision-making in multi-agent, competitive settings. Nath et al. [562] introduce ToolComp, a benchmark designed to evaluate multi-step tool-use reasoning. To analyze adaptive reasoning in the face of real-world change, OSWorld [887], CogAgent [260], Mobile-Agent-E [828], WebShop [954], WebArena [1126], WGSR-Bench [972], and WebGames [735] assess AI systems across domains such as operating systems, mobile GUIs, browser tasks, and interactive entertainment [1106, 780, 512, 552]. Hu et al. [272] present Text2World, which evaluates agents’ ability to generate interactive environments from text to test agent adaptability [995]. Multimodal Reasoning Multimodal reasoning refers to a system’s ability to integrate and reason across diverse input types, including text, images [316]. This capability is crucial for solving complex problems that require information from diverse formats. - Complex Mathematics: Mathematical reasoning often integrates both textual and visual components, such as equations, graphs, or diagrams [921]. Specifically, challenges like MathVista [508], MathVision [782], MathVerse [1054], M3CoT-Math [91], CMMaTH [433], EnigmaEval [763], CoMT-Geometry [125], and PGPS9K [1050] aim to advance multimodal reasoning in mathematics, improving the evaluation of multimodal Long CoT logic. - Complex Code: The second area of focus involves code-related reasoning, where systems interpret textual descriptions and code snippets. Benchmarks like HumanEval-V [1035], Code-Vision [767], Plot2Code [852], and ChartMimic [931] evaluate systems’ capabilities to generate or interpret code from natural language and multimodal inputs for assessing systems that integrate natural language processing with programming tasks. - Complex Science: This area involves integrating scientific texts with related diagrams or experimental data. Benchmarks like ScienceQA [507], M3CoT-Science [91], BMMR [874], and ScienceBoard [698] evaluate how well models combine science information with Long CoT reasoning across various scientific domains [966]. Further, Guo et al. [229] propose MolPuzzle for the evaluation of molecular structure elucidation. - Commonsense Puzzle: This area focuses on commonsense reasoning, where systems combine reasoning cues and images to make deeper conclusions. Chen et al. [91] introduce M3CoT-Commensense, which incorporates commonsense Long CoT reasoning for complex multimodal interactions. Further, PuzzleVQA [128], MMReason [953] and LEGO-Puzzles [711] focus more on abstract and spatial puzzle reasoning, respectively. Additionally, Wang et al. [760] propose two benchmarks: Clue-Visual Question Answering (CVQA), which tests visual comprehension through three task types, and Clue of Password-Visual Question Answering (CPVQA), which features two task types focusing on the interpretation and application of visual data. AI for Research Recent advancements in AI have significantly advanced scientific research [94, 1124, 817, 215], with platforms like SciWorld [798] improving the research process. Simultaneously, Pricope [608] and Chan et al. [67] introduce a machine-learning platform to evaluate the potential of RLLMs in automating experiments. Several studies also examine RLLMs’ ability to generate innovative research ideas. For instance, Si et al. [672] conduct evaluations with over 100 NLP researchers to assess RLLMs’ creativity, revealing notable limitations [404, 856, 726]. Additionally, Li et al. [434] introduce SolutionBench, a benchmark for assessing systems’ ability to generate feasible solutions for complex engineering problems. ## 4 Deep Reasoning for Long CoT Deep reasoning capabilities primarily require profound depth and comprehensiveness in cognitive and reasoning processes. In the absence of such capabilities, RLLMs suffer significant performance declines [758, 823]. Current methods for enhancing deep reasoning can be categorized into two main approaches: (1) Deep Reasoning Format (§ 4.1), which involves utilizing various reasoning execution formats to maximize the reasoning step length $k$ within reasoning boundary $B_l$ in Equation (2), by selecting the most suitable reasoning format; and (2) Deep Reasoning Learning (§ 4.2), which focuses on improving the model’s internal capabilities to enhance its deep reasoning abilities, thereby extending the reasoning boundary $B_l$ in Equation (2) intrinsically. <details> <summary>x6.png Details</summary>  ### Visual Description ## Diagram: Three Approaches to Deep Reasoning ### Overview The image is a composite diagram illustrating three distinct paradigms for "Deep Reasoning," likely in the context of artificial intelligence or computational problem-solving. It uses Conway's Game of Life as a unifying example to contrast the approaches. The diagram is divided into three main sections, labeled (a), (b), and (c), each with a distinct title and visual style. ### Components/Axes The diagram is organized into three primary panels: 1. **Panel (a):** Titled "Natural Language Deep Reasoning." Contains a block of explanatory text and bullet points. 2. **Panel (b):** Titled "Structured Language Deep Reasoning." Contains a block of Python code. 3. **Panel (c):** Titled "Latent Space Deep Reasoning." Contains three sub-diagrams illustrating different internal architectures for reasoning. Each panel features a small, similar cartoon character (a worm-like figure with a red scarf) in the bottom-right corner, acting as a mascot or visual anchor. ### Detailed Analysis #### **Panel (a): Natural Language Deep Reasoning** This section presents reasoning in human-readable, explanatory prose. * **Text Transcription:** > To predict the output of the given input for Conway's Game of Life, we need to apply the rules of the game to each cell on the board. The rules are as follows: > 1. Any live cell with fewer than two live neighbors dies (underpopulation)... > ◆ Given Input Board:... > ◆ Step-by-Step Analysis: ... > ◆ Final Output: After applying the rules to each cell... * **Visual Elements:** A small icon of a notepad with a pencil is in the top-left corner of the text box. #### **Panel (b): Structured Language Deep Reasoning** This section presents reasoning in the form of executable code. * **Code Transcription:** ```python # import necessary packages from collections import Counter # import necessary packages from collections import Counter # all class and function definitions in the code file, if any class Solution(object): def gameOfLifeInfinite(self, live): ctr = Counter((I, J) for i, j in live ``` * **Visual Elements:** A small icon of a code editor window (`</>`) with a gear is in the top-left corner of the code box. The code syntax is color-highlighted (e.g., `from`, `import`, `class`, `def` in blue/purple; strings in green). #### **Panel (c): Latent Space Deep Reasoning** This section illustrates three abstract, internal architectures for reasoning within a model's latent space. Each sub-diagram shows a flow from input tokens to output tokens via a specialized processing block. **Sub-diagram 1 (Left): "Reasoning Token Driven Latent Space Deep Reasoning"** * **Components & Flow:** * **Input:** A stack of yellow boxes labeled "Token 1", "...", "Token N-1". * **Processing:** These tokens feed into a central green block labeled "RLLM" (with a magnifying glass icon). Above this block is a blue box labeled "Continuous Reasoning Token," with arrows pointing down into the RLLM block. * **Output:** The RLLM block outputs a stack of white boxes labeled "Token 2", "...", "Token N". * **Caption:** "Reasoning Token Driven Latent Space Deep Reasoning" is written below the diagram. **Sub-diagram 2 (Center): "Reasoning Vector Driven Latent Space Deep Reasoning"** * **Components & Flow:** * **Input:** A stack of yellow boxes labeled "Token 1", "...", "Token N-1". * **Processing:** These tokens feed into a vertical stack of two green "Thought Block" icons (with Lego-like bricks). Above the top Thought Block is a blue box labeled "Continuous Reasoning Vector," with arrows pointing down into the block. * **Output:** The bottom Thought Block outputs a stack of white boxes labeled "Token 2", "...", "Token N". * **Caption:** "Reasoning Vector Driven Latent Space Deep Reasoning" is written below the diagram. **Sub-diagram 3 (Right): "Reasoning Manager Driven Latent Space Deep Reasoning"** * **Components & Flow:** * **Input:** A stack of yellow boxes labeled "Token 1", "...", "Token N-1". * **Processing:** These tokens feed into a blue box labeled "Continuous Reasoning Manager." This manager has arrows pointing down into a vertical stack of two green "Thought Block" icons. * **Output:** The bottom Thought Block outputs a stack of white boxes labeled "Token 2", "...", "Token N". * **Caption:** "Reasoning Manager Driven Latent Space Deep Reasoning" is written below the diagram. ### Key Observations 1. **Progression of Abstraction:** The panels show a clear progression from explicit, human-centric reasoning (natural language), to formal, machine-executable reasoning (code), to an internalized, sub-symbolic reasoning process (latent space). 2. **Common Problem:** All three approaches are framed as solutions to the same problem: implementing Conway's Game of Life. 3. **Latent Space Variants:** The three sub-diagrams in (c) propose different mechanisms for guiding latent space reasoning: via a special token, via a continuous vector, or via a managing module that controls thought blocks. 4. **Visual Consistency:** The use of consistent colors (yellow for input tokens, white for output tokens, green for processing blocks, blue for continuous reasoning elements) and the recurring cartoon character creates visual cohesion across the different concepts. ### Interpretation This diagram serves as a conceptual taxonomy for how advanced AI systems might perform complex reasoning tasks. * **Natural Language Reasoning (a)** mimics human explanation, emphasizing transparency and interpretability. It's suitable for communicating logic but may be inefficient for execution. * **Structured Language Reasoning (b)** represents the current standard for precise, verifiable computation. It's executable but can lack the flexibility and intuitive leaps of human thought. * **Latent Space Reasoning (c)** represents a frontier where reasoning occurs as a continuous, distributed process within a neural network's internal representations. The three variants explore how this process might be structured and controlled—either by injecting a reasoning signal (token/vector) or by using a dedicated manager module to orchestrate "thought" steps. This approach aims to combine the flexibility of neural networks with the structured, multi-step reasoning capabilities of symbolic systems. The overarching message is that "deep reasoning" in AI is not a monolithic concept but can be implemented through fundamentally different paradigms, each with its own trade-offs between interpretability, precision, and computational efficiency. The progression from (a) to (c) suggests a movement towards more powerful, but less directly interpretable, forms of machine cognition. </details> Figure 5: Three main categories of deep reasoning formats: natural language, structured language, and latent-space reasoning (subdivided into token-, vector-, and manager-driven latent reasoning), with examples drawn from Li et al. [401]. ### 4.1 Deep Reasoning Format As illustrated in Figure 5, deep reasoning formats can be categorized into three main types: natural language (§ 4.1.1), structured language (§ 4.1.2), and latent-space reasoning (§ 4.1.3), the latter of which is further subdivided into token-, vector-, and manager-driven latent reasoning. The reasoning performance across these formats is presented in Table 1. | Model Latent Space Deep Reasoning No-CoT [151] | Base Model Mistral-7B [318] | GSM8k 38.0 | MATH - | GPQA - | OlympiadBench - | LiveCodeBench - | | --- | --- | --- | --- | --- | --- | --- | | SQ-VAE [810] | Llama-2-7B [743] | 40.0 | 7.0 | - | - | - | | RecurrentBlock-3.5B [204] | - | 42.1 | - | - | - | - | | ICoT-SI [151] | Mistral-7B [318] | 51.0 | - | - | - | - | | Natural Language Deep Reasoning | | | | | | | | Self-Rewarding [114] | Llama-2-7B [743] | 40.0 | 10.7 | - | - | - | | Llama-3.1-8B [168] | - | 56.7 | 20.3 | - | - | | | MetaMath [983] | Llama-2-7B [743] | 66.5 | - | - | - | - | | OVM [979] | Llama-2-7B [743] | 73.7 | - | - | - | - | | NuminaMath-7B-CoT [397] | - | 75.4 | 55.2 | - | 19.9 | - | | Qwen2-7B [925] | - | 79.9 | 44.2 | - | 21.3 | - | | Qwen2-Math-7B [927] | - | 80.4 | 50.4 | - | 38.2 | - | | Internlm2-math-plus-7B [974] | - | 84.0 | 54.4 | - | 18.8 | - | | OMI2 [401] | Qwen2.5-Coder-7B [301] | 84.1 | 72.3 | 36.2 | - | 27.2 | | Llama-3.1-70B [168] | - | 85.5 | 41.4 | - | - | - | | CODEI/O++ [401] | Qwen2.5-Coder-7B [301] | 85.7 | 72.1 | 40.6 | - | 29.1 | | CODEI/O [401] | Qwen2.5-Coder-7B [301] | 86.4 | 71.9 | 43.3 | - | 28.5 | | WI [401] | Qwen2.5-Coder-7B [301] | 87.0 | 71.4 | 39.1 | - | 26.0 | | WI (Full) [401] | Qwen2.5-Coder-7B [301] | 87.0 | 71.1 | 42.9 | - | 27.6 | | OMI2 (Full) [401] | Qwen2.5-Coder-7B [301] | 88.5 | 73.2 | 40.9 | - | 28.4 | | DeepSeekMath-7B-RL [658] | - | 88.2 | 51.7 | - | 19.0 | - | | Llama-3.1-405B [168] | - | 89.0 | 53.8 | - | - | - | | CoMAT [371] | GPT-4 [3] | 93.7 | - | 40.4 | - | - | | CoT [634] | GPT-4 [3] | 94.5 | - | 41.8 | 50.2 | - | | FCoT [523] | GPT-4 [3] | 95.0 | - | - | - | - | | Qwen2.5-Math-7B-Instruct [927] | - | 95.2 | 83.6 | - | 41.6 | - | | MathPrompter [303] | GPT-4 [3] | 95.6 | - | - | - | - | | Qwen2.5-Math-72B-Instruct [927] | - | 95.9 | 85.9 | - | 49.0 | - | | DeepSeek-R1-Distill-Qwen-7B [227] | - | - | 92.8 | - | 49.1 | 37.6 | | DeepSeek-R1-Distill-Qwen-32B [227] | - | - | 94.3 | - | 62.1 | 57.2 | | Structured Language Deep Reasoning | | | | | | | | STaR [1012] | Llama-2-7B [743] | 58.2 | 16.0 | - | - | - | | ENVISIONS [897] | Llama-2-7B [743] | 59.0 | 19.0 | - | - | - | | MAmmoTH [1006] | Code-Llama-7B [639] | 59.4 | - | - | - | - | | MathCoder-CL [783] | Code-Llama-7B [639] | 67.8 | 30.2 | - | - | - | | ToRA-Code [217] | Llama-2-7B [743] | 72.6 | - | - | - | - | | Brain [107] | Code-Llama-7B [639] | 74.0 | - | - | - | - | | DeepSeek-Coder-7B [226] | - | 77.4 | 44.4 | - | - | - | | SIaM [978] | Qwen-2-Math-Base | 81.5 | 50 | - | - | - | | OC-SFT-1 [401] | Qwen2.5-Coder-7B [301] | 86.7 | 70.9 | 37.7 | - | 27.5 | | PyEdu [401] | Qwen2.5-Coder-7B [301] | 85.8 | 71.4 | 40.9 | - | 25.8 | | Qwen2.5-Math-7B-Instruct [927] | - | 94.6 | 85.2 | - | 55.6 | - | | Qwen2.5-Math-72B-Instruct [927] | - | 95.8 | 88.1 | - | 60.6 | - | | QuaSAR [634] | GPT-4 [3] | 96.5 | - | 55.4 | 44.6 | - | | MathDivide [687] | GPT-4 [3] | 96.8 | - | - | - | | Table 1: Performance of various deep reasoning formats, sorted primarily by GSM8K scores. “-” indicates that the paper did not report this score. #### 4.1.1 Natural Language Deep Reasoning Traditionally, researchers have sought to adapt natural language for intuitive and free-flowing deep reasoning [836, 1118, 303, 617, 1070, 765, 205]. Early work by Wei et al. [836] demonstrated that the use of natural language Long CoT significantly enhances the reasoning capabilities of RLLMs. Further, the Natural Program framework [460] allows RLLMs to engage in deeper natural language reasoning by ensuring a more structured and rigorous logical analysis. More recently, CodeI/O [401] has introduced a technique that reorganizes code-based reasoning patterns into natural language formats, further boosting the reasoning potential of RLLMs [36]. Similarly, Li et al. [387] propose CoRT, which integrates code into reasoning to facilitate a mixture of formats, resulting in improved cognitive performance. #### 4.1.2 Structured Language Deep Reasoning Structured language deep reasoning encompasses various approaches designed to program [100, 464, 687, 591, 198, 845, 830, 1044] or symbolic language [605, 158, 451, 372, 933, 604, 37, 40, 797, 380] format for enhanced deep reasoning. In this context, most studies focus on utilizing code to better enhance the mathematical reasoning capabilities [389, 107, 978, 85]. Xu et al. [897] propose a neural-symbol self-training framework guided by the environment, addressing both the scarcity of symbolic data and the limitations of symbolic processing in LLMs. Additionally, Liao et al. [443] present SKIntern, which refines symbolic RLLMs through curriculum learning and linear attenuation, enabling the internalization of symbolic knowledge with fewer examples, reducing computational costs, and accelerating inference. Furthermore, Ranaldi et al. [634] introduce QuaSAR, a CoT variant that directs LLMs to operate at higher abstraction levels through quasi-symbolic reasoning, thus improving natural language reasoning and providing more precise structural representations. #### 4.1.3 Latent Space Deep Reasoning Latent space deep reasoning encompasses techniques designed to enhance the reasoning abilities of LLMs by leveraging operations within continuous latent spaces [684, 151, 640, 324]. These approaches can be categorized into three main paradigms: (1) Reasoning Token-Driven Latent Space Deep Reasoning: Early work [810, 1013] introduce the concept of “planning tokens” or “thought tokens” to guide reasoning within latent spaces [949, 1008]. Further, Coconut [236] expands on this through the maintenance of multiple alternative reasoning paths, increasing both complexity and efficiency [1069, 706]. At the extreme, Heima [662] condenses the entire Long CoT process into a single token, yielding substantial computational savings. (2) Reasoning Vector Driven Latent Space Deep Reasoning: Building on the previous paradigm, LTM [356] conceptualizes the layers of LLMs as “thought blocks” and introduces the concept of “thought vectors” for each layer. This approach allows for the scaling of inference-time computations by implicitly performing reasoning within the latent space through recurrent depth. (3) Reasoning Manager Driven Latent Space Deep Reasoning: Inspired by these, Schöne et al. [647], Geiping et al. [204], and Saunshi et al. [646] propose a mechanism similar to a continuous reasoning manager, which iteratively governs a trained “recurrent block” as a recurrent “thought block” [511]. This method integrates deeper model layers during reasoning, enhancing performance without needing specialized training data, and even outperforming larger RLLMs. Additionally, ITT [109] leverages the original transformer layer as a recurrent “thought block”, selecting key tokens via adaptive token routing and controlling reasoning depth with residual thinking connections, enabling more efficient processing of critical tokens. Further, System-1.5 Reasoning [808] defines two dynamic shortcuts. The Model Depth Shortcut (DS) lets non-critical tokens exit early via lightweight adapter branches while routing critical tokens through deeper Transformer layers, thus supporting adaptive, vertical reasoning. The Step Shortcut (SS) reuses hidden states across decoding steps to bypass trivial iterations and enable horizontal reasoning in latent space. <details> <summary>x7.png Details</summary>  ### Visual Description ## Diagram: Comparison of Deep Reasoning Training Methodologies ### Overview The image is a technical diagram comparing two distinct methodologies for training Large Language Models (LLMs) to perform complex reasoning tasks. It is divided into two side-by-side panels, labeled (a) and (b), each enclosed in a dashed-line box. The diagram uses a combination of text labels, icons, and directional arrows to illustrate the workflow and components of each approach. ### Components/Axes The diagram is structured into two primary panels: **Panel (a): Deep Reasoning Imitation** * **Title:** "(a) Deep Reasoning Imitation" (top center of the left panel). * **Components (from left to right):** 1. **Complex Query:** Represented by a lightbulb icon with puzzle pieces. This is the input. 2. **Advanced Deep Reasoning System:** A yellow box with an icon of a person at a whiteboard. This is the source of the "teacher" signal. 3. **Deep Reasoning Process:** A beige, rounded rectangle representing the intermediate reasoning steps. 4. **Reasoning LLM:** A green box with a lightning bolt icon. This is the model being trained. 5. **Supervised Finetuning:** A purple box with a building blocks icon. This is the training mechanism. * **Flow:** Arrows indicate the process flow: `Complex Query` → `Advanced Deep Reasoning System` → `Deep Reasoning Process` → `Supervised Finetuning`. A separate arrow shows `Complex Query` also feeding directly into the `Reasoning LLM`. The `Supervised Finetuning` block then updates the `Reasoning LLM`. **Panel (b): Deep Reasoning Self-Learning** * **Title:** "(b) Deep Reasoning Self-Learning" (top center of the right panel). * **Components (from left to right):** 1. **Complex Query:** Same icon as in panel (a). 2. **Reasoning LLM:** Same green box and icon as in panel (a). 3. **Deep Reasoning Process:** Same beige, rounded rectangle. 4. **Correction Recheck:** A blue box with a magnifying glass over a document icon. It contains three sub-labels stacked vertically: "Rule", "ORM", "PRM". 5. **Reward:** A beige, rounded rectangle. 6. **Reinforcement Learning:** A purple box with a building blocks icon (same icon as "Supervised Finetuning" in panel (a)). * **Flow:** Arrows indicate: `Complex Query` → `Reasoning LLM` → `Deep Reasoning Process` → `Correction Recheck`. The `Correction Recheck` block outputs to `Reward`, which then feeds into `Reinforcement Learning`. The `Reinforcement Learning` block updates the `Reasoning LLM`. ### Detailed Analysis The diagram explicitly contrasts two training paradigms: 1. **Deep Reasoning Imitation (Panel a):** * This is a **supervised learning** approach. * The `Reasoning LLM` learns by imitating the outputs of a more capable `Advanced Deep Reasoning System`. * The process is unidirectional: the advanced system generates a reasoning process, which is used to directly finetune the target LLM via supervised methods. 2. **Deep Reasoning Self-Learning (Panel b):** * This is a **reinforcement learning** approach. * The `Reasoning LLM` generates its own reasoning process. * This process is evaluated by a `Correction Recheck` module, which assesses it against specific criteria: "Rule" (likely logical rules), "ORM" (likely Outcome Reward Model), and "PRM" (likely Process Reward Model). * The evaluation produces a `Reward` signal. * This reward is used to update the LLM via `Reinforcement Learning`, creating a closed-loop, self-improving system. ### Key Observations * **Shared Components:** Both panels start with a "Complex Query" and involve a "Reasoning LLM" and a "Deep Reasoning Process." The final training block in both is represented by the same purple building-blocks icon, though labeled differently. * **Core Difference:** The fundamental difference is the source of the training signal. Panel (a) relies on an external, pre-existing "Advanced System" (imitation). Panel (b) relies on an internal evaluation and reward mechanism (self-learning). * **Complexity:** The self-learning pathway (b) is more complex, introducing additional components (`Correction Recheck`, `Reward`) and a feedback loop for iterative improvement. * **Sub-labels in Correction Recheck:** The terms "Rule," "ORM," and "PRM" are critical technical details within the self-learning framework, specifying the types of evaluative models or criteria used. ### Interpretation This diagram illustrates a conceptual shift in AI training for complex tasks. The "Imitation" approach (a) is simpler and more direct, akin to a student learning by copying a master's work. It is effective but limited by the quality and availability of the "teacher" system. The "Self-Learning" approach (b) represents a more advanced, autonomous paradigm. Here, the model learns by trial and error, guided by a reward system that critiques both its final outcomes (ORM) and its step-by-step reasoning process (PRM), while also adhering to formal rules. This mimics a more human-like form of learning through practice and self-correction. The presence of "ORM" and "PRM" suggests the methodology is concerned not just with getting the right answer, but with the validity and soundness of the reasoning path itself. The diagram argues that for developing truly robust reasoning capabilities in AI, moving from pure imitation to structured self-evaluation and reinforcement may be a necessary and more powerful direction. The visual contrast emphasizes the increased architectural complexity required for this autonomous learning loop. </details> Figure 6: The different learning strategies of deep reasoning learning, including deep reasoning imitation of the data from advanced deep reasoning systems, like advanced RLLMs, MCTS, etc.; deep reasoning self-learning from preference-based RL by implicit reward. ### 4.2 Deep Reasoning Learning Insufficient deep reasoning in RLLMs can significantly degrade performance [758, 823]. As a result, research has focused on improving reasoning through training. Supervised fine-tuning (SFT) [1058] stabilizes model outputs by serving as a memory process [883], while reinforcement learning (RL) enables generalization and self-learning [227, 137, 276, 898]. Recent studies for deep reasoning learning have explored using SFT to imitate advanced reasoning in RLLMs and applying RL to enhance self-improvement in reasoning. As illustrated in Figure 6, this section outlines two key approaches to improve deep reasoning: (1) Deep Reasoning Imitation (§ 4.2.1), which involves learning reasoning from human-annotated or distilled data through SFT, and (2) Deep Reasoning Self-Learning (§ 4.2.2), where models improve reasoning through preference-based RL with implicit rewards. The performance of these methods is shown in Table 2. #### 4.2.1 Deep Reasoning Imitation Deep reasoning in RLLMs can be effectively achieved by mimicking advanced reasoning systems, such as human reasoning [558, 61, 115, 403], advanced RLLMs [227, 58, 957, 370, 102], and scaling-augmented RLLMs [410, 1003, 596, 1136, 41]. This approach enables the model to learn complex reasoning patterns and generalize across tasks [937, 416]. Specifically, (1) Imitation from Human: Earlier, Cobbe et al. [141] first propose the deep reasoning imitation paradigm using human examples. ALT [558] improves RLLM reasoning by generating larger datasets of human-annotated logical templates, which fosters deeper reasoning [241]. To enhance diversity, EIT [61] promotes simpler human-generated plans, while LLMs contribute more nuanced reasoning, facilitating collaboration between human input and AI. (2) Imitation from Advanced RLLMs: A body of work utilizes zero-shot prompting to guide large teacher RLLMs in generating reasoning rationale, which is then used to fine-tune smaller RLLMs, marking the beginning of deep reasoning imitation [256, 352, 938, 521]. Additionally, AceMath [500] applies few-shot prompting to distill Long CoT samples from advanced LLMs, followed by multi-stage quality-guided SFT to enhance performance. Chen et al. [107] separate the data synthesis process into planning and reasoning stages, thereby improving reasoning quality. DART-Math [738] effectively distills complex queries requiring deeper reasoning during synthesis, advancing deep reasoning capabilities. Further, Ahmad et al. [7] propose OpenCodeReasoning, expanding this paradigm to the code scenarios. (3) Imitation from Scaling-augmented RLLMs: Earlier, Bansal et al. [34] enhance data quality by scaling the sampling size and length, boosting imitation performance [481, 1005]. Yang et al. [927] and Zhao et al. [1090] further improve data quality by scaling sampling and selecting samples through sample feature or an additional reward model. Additionally, Li et al. [410] identify optimal deep reasoning paths through MCTS, advancing imitation effectiveness. Recent studies [299, 550] show that distilling knowledge from advanced RLLM APIs like O1 [307] and R1 [227] significantly enhances the performance of smaller LLMs [424, 223]. This method, employing supervised fine-tuning, boosts model performance on complex mathematical reasoning tasks, sometimes surpassing the teacher models’ performance. Building on these findings, LIMO [967], S1 [560], and RedStar [902] argue that a large number of imitation samples is unnecessary. They demonstrate that even a minimal set of samples can activate deep reasoning capabilities in foundational LLMs. For practical applications, Turtel et al. [747] showcase how these techniques can predict future events beyond a model’s knowledge cutoff. Sun et al. [701], Yang et al. [928] and Zhao et al. [1093] further enhance deep reasoning imitation by selecting high-quality samples from large datasets, thereby improving the quality of the imitation data. | Model | Data Size | Base Model | GSM8K | MATH | MATH-500 | AIME2024 | GPQA | OlympiadBench | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Deep Reasoning Imitation | | | | | | | | | | SFT [938] | 200K | Llama-3.1-8B [168] | - | - | - | 54.1 | 3.5 | - | | Retro-Enh [115] | 14M | Llama-3-8B [168] | 45.1 | 21.7 | - | - | - | - | | Query-Exp [115] | 24M | Llama-3-8B [168] | 51.3 | 23.1 | - | - | - | - | | Res-Div [115] | 14M | Llama-3-8B [168] | 53.0 | 23.2 | - | - | - | - | | MetaMath [738] | 0.40M | Mistral-7B [318] | 76.5 | 29.8 | - | - | - | 5.9 | | ALT-FLDx2 [558] | 100K | Llama-3.1-70B [168] | 83.3 | 24.4 | - | - | - | - | | EIT [61] | 15K | Llama-2-70B [743] | 84.1 | 32.5 | - | - | - | - | | MathScale [738] | 2.0M | Mistral-7B [318] | 74.8 | 35.2 | - | - | - | - | | Tutor-Amp [115] | 11M | Llama-3-8B [168] | 64.4 | 35.9 | - | - | - | - | | MMIQC [738] | 2.3M | Mistral-7B [318] | 75.4 | 37.4 | - | - | - | 9.4 | | VRT [738] | 0.59M | Mistral-7B [318] | 82.3 | 38.7 | - | - | - | 8.7 | | KPMath-Plus [738] | 1.6M | Mistral-7B [318] | 82.1 | 46.8 | - | - | - | - | | Llama-2-70B-Xwin-Math-V1.1 [385] | 1.4M | Llama-2-70B [743] | 90.2 | 52.5 | - | - | - | 16.3 | | DART-Math-Mistral-7B [738] | 591K | Mistral-7B [318] | 81.1 | 45.5 | - | - | - | 14.7 | | DART-Math-Llama-3-70B [738] | 591K | Llama-3-70B [168] | 89.6 | 56.1 | - | - | - | 20.0 | | Rejection Sampling [410] | 197K | Qwen2.5-7B [926] | 87.1 | 70.0 | - | 10.0 | - | 27.1 | | Evol-Instruct-7B [514] | 905K | Qwen2.5-Math-7B [927] | 88.5 | - | 77.4 | 16.7 | - | - | | FastMCTS [410] | 288K | Qwen2.5-7B [926] | 88.9 | 74.0 | - | 20.0 | - | 27.5 | | KPDDS-7B [295] | 800K | Qwen2.5-Math-7B [927] | 89.9 | - | 76.0 | 10.0 | - | - | | DeepSeek-R1-Distill-Qwen-7B [227] | 800K | Qwen2.5-7B-Instruct [926] | 91.7 | - | 91.6 | 43.3 | - | - | | Openmathinstruct-7B [740] | 14M | Qwen2.5-Math-7B [927] | 92.0 | - | 79.6 | 10.0 | - | - | | NuminaMath [967] | 100K | Qwen2.5-Math-7B [927] | 92.9 | - | 81.8 | 20.0 | - | - | | PromptCoT-DS-7B [1090] | 115K | DeepSeek-R1-Distill-Qwen-7B [227] | 92.6 | - | 93.0 | 60.0 | - | - | | PromptCoT-Qwen-7B [1090] | 905K | Qwen2.5-Math-7B [927] | 93.3 | - | 84.0 | 26.7 | - | - | | AceMath-7B-Instruct [500] | 1.2M | Qwen2-Math-7B-Instruct [927] | 93.7 | 83.1 | - | - | - | 42.2 | | AceMath-72B-Instruct [500] | 1.2M | Qwen2.5-Math-72B-Instruct [927] | 96.4 | 86.1 | - | - | - | 48.4 | | NuminaMath [967] | 100K | Qwen2.5-32B-Instruct [926] | - | - | 59.2 | 6.5 | 25.8 | 36.7 | | OpenThoughts [967] | 114K | Qwen2.5-32B-Instruct [926] | - | - | 80.6 | 50.2 | 42.9 | 56.3 | | Sky-T1-32B-Preview [724] | 17K | Qwen2.5-32B-Instruct [926] | - | - | 82.4 | 43.3 | 56.8 | - | | Journey Learning [299] | 5K | Qwen2.5-Math-72B [927] | - | - | 87.2 | 43.3 | - | - | | STILL-2 [550] | 3.9K | Qwen2.5-32B-Instruct [926] | - | - | 90.2 | 46.7 | 55.1 | - | | Bespoke-32B [362] | 17K | Qwen2.5-32B-Instruct [926] | - | - | 93.0 | 63.3 | 58.1 | - | | s1 [560] | 1K | Qwen2.5-32B-Instruct [926] | - | - | 93.0 | 56.7 | 59.6 | - | | DeepSeek-R1-Distill-Qwen-32B [227] | 800K | Qwen2.5-32B-Instruct [926] | - | - | 94.3 | 72.6 | 62.1 | - | | LIMO [967] | 817 | Qwen2.5-32B-Instruct [926] | - | - | 94.8 | 15.8 | 66.7 | 66.8 | | Deep Reasoning Self-Learning | | | | | | | | | | DPO [302] | 40K | DeepSeek-Math-7B-Base [658] | 74.8 | 34.9 | - | - | - | - | | ReFT [302] | 40K | DeepSeek-Math-7B-Base [658] | 71.4 | 36.0 | - | - | - | - | | Self-Explore [302] | 40K | DeepSeek-Math-7B-Base [658] | 78.6 | 37.7 | - | - | - | - | | SimPO [723] | 10K | Qwen2.5-Math-7B-Instruct [927] | 88.8 | 40.0 | 56.6 | - | - | - | | DPO [446] | 11K | DeepSeek-Math-7B-Instruct [658] | - | 48.7 | - | - | - | - | | TPO [446] | 11K | DeepSeek-Math-7B-Instruct [658] | - | 51.3 | - | - | - | - | | DPO [446] | 11K | Qwen2-7B-Instruct [925] | - | 54.3 | - | - | - | - | | TPO [446] | 11K | Qwen2-7B-Instruct [925] | - | 55.5 | - | - | - | - | | MCTS [74] | 15K | DeepSeek-Math-7B-Base [658] | 83.2 | 64.0 | - | - | - | - | | SBS [74] | 15K | DeepSeek-Math-7B-Base [658] | 84.1 | 66.3 | - | - | - | - | | FastMCTS+Branch-DPO [410] | 152K | FastMCTS-7B [410] | 89.9 | 75.4 | - | 20.0 | - | 29.6 | Table 2: Performance of various deep reasoning learning methods, sorted primarily by Math or Math-500 scores. “-” indicates that the paper did not report this score. #### 4.2.2 Deep Reasoning Self-Learning While simple imitation can yield strong performance, current models still rely heavily on human annotations or outputs from more advanced models for both imitation and distillation [502]. To address this limitation, recent research has focused on enabling more advanced reasoning through techniques like self-play and self-learning [948, 1077, 409, 624]. Specifically, self-learning methods can be classified into two paradigms, differentiated by their sampling strategies: (1) Self-Learning from Direct Sampling: The earliest method, STaR [1012], utilizes In-Context Learning (ICL) to sample deep reasoning results [657] and uses the correctness of the final answer as an implicit reward for self-learning [258, 581, 582, 1059, 826, 462]. Further, ReST [225] extends this by introducing a Grow-Improve paradigm, where self-generated reasoning is first annotated with rewards and then enhanced via offline RL algorithms. However, these approaches can be fragile, especially when the reward process lacks robustness. Inspired by the Expectation-Maximization (EM) algorithm, Singh et al. [674] propose a method that generates rewards and iteratively optimizes LLMs to achieve the best performance on a validation set, significantly improving robustness. To further strengthen the reward process, a series of work introduce a method to adapt incorrect solutions, training a verifier [155, 262] or utilize entropy [809, 1040] to select or refine the reward process and improve self-learning quality. (2) Self-Learning from Tree Search: Early deep learning methods, such as EXIT [18], combined MCTS with deep neural networks for reinforcement learning, iteratively self-training the network to guide the tree search and enhance reasoning. Building on this, CPO [1065] and TPO [446] align each step of Long CoT reasoning with the corresponding tree search path, using Tree of Thoughts (ToT) [955] preference information to support deeper reasoning [951, 302]. Li [422] propose Policy-Guided Tree Search (PGTS), integrating RL with structured tree exploration for more efficient navigation of reasoning paths. Further developments, such as AlphaMath [74], AlphaLLM-CPL [814], and TongGeometry [1029], refine MCTS behavior through stepwise trajectory pair extraction and curriculum preference learning, boosting LLM reasoning abilities [611, 412, 872]. Takeaways: Imitation & Self-Learning • Imitating deep reasoning from advanced RLLMs, and scaling-augmented methods like MCTS can help models learn complex reasoning patterns with fewer samples. • Self-learning techniques, including reinforcement learning and tree search, allow RLLMs to enhance their reasoning abilities over time. • The combination of imitation from advanced RLLMs and self-learning techniques strengthens RLLM reasoning, leading to strong performance on complex tasks. ## 5 Feasible Reflection for Long CoT Feasible Reflection is a pivotal component of Long CoT reasoning, enabling LLMs to handle complex tasks through iterative feedback and refinement [406, 192]. Specifically, it comprises two primary stages: (1) Feedback (§ 5.1), which generates feedback signals $F_i$ to correct node $n_j$ in Equation (5); and (2) Refinement (§ 5.2), which adjusts the subsequent node $n_i+1$ according to the feedback in Equation (6). <details> <summary>x8.png Details</summary>  ### Visual Description ## Diagram: Feedback Mechanisms in AI Reasoning Systems ### Overview The image is a conceptual diagram illustrating two distinct feedback mechanisms for evaluating and improving AI reasoning systems, specifically referred to as "RLLMs" (Reasoning Large Language Models). The diagram is divided into two horizontal panels, each enclosed by a dashed border. The top panel is labeled "(a) Overall Feedback," and the bottom panel is labeled "(a) Process Feedback" (note: the duplicate label "(a)" appears to be a typographical error in the source image). The diagram uses cartoon-style illustrations of a worm-like character with a magnifying glass (representing an evaluator or critic) and a hand holding a trophy (representing a reward or scoring model) to depict the flow of information and feedback. ### Components/Axes The diagram is not a chart with axes but a process flow diagram. Its key components are: **Top Panel: (a) Overall Feedback** * **Title:** "(a) Overall Feedback" (top-left corner). * **Three Sequential Stages:** Arranged left to right, each stage shows an interaction between the "evaluator" worm and a "reward" hand. 1. **Stage 1 (Left):** * **Reward Model Box:** Green box labeled "ORM" (likely Outcome Reward Model). * **Output Magnifying Glass:** Held by the worm, displays "Output 0.6". 2. **Stage 2 (Center):** * **Reward Model Box:** Green box labeled "Rule Extraction". * **Output Magnifying Glass:** Held by the worm, displays a green checkmark icon. 3. **Stage 3 (Right):** * **Reward Model Box:** Green box labeled "RLLMs' Reasoning". * **Output Magnifying Glass:** Held by the worm, displays the word "Correct". **Bottom Panel: (a) Process Feedback** * **Title:** "(a) Process Feedback" (top-left corner). * **Two Main Stages & Environment Loop:** Arranged left to right. 1. **Stage 1 (Left):** * **Reward Model Box:** Yellow box labeled "PRM" (likely Process Reward Model). * **Step Magnifying Glass:** Held by the worm, displays "Step 0.8". 2. **Stage 2 (Center):** * **Reward Model Box:** Yellow box labeled "RLLMs' Reasoning". * **Output Magnifying Glass:** Held by the worm, displays the word "Correct". 3. **Environment Component (Right):** * **Environment Label:** Text "Environment" at the bottom. * **Globe Icon:** A stylized Earth with leaves. * **Environment Type Labels:** Three stacked, rounded rectangles in light purple, labeled from top to bottom: "OS Env.", "Real Env.", "GUI Env.". * **Interaction Arrows:** Two curved, green arrows form a loop between the "RLLMs' Reasoning" stage and the "Environment" globe, indicating continuous interaction. ### Detailed Analysis The diagram contrasts two evaluation paradigms: 1. **Overall Feedback (Top Panel):** This process evaluates the final outcome of a reasoning task. * **Flow:** The process moves sequentially from left to right. * **Trend/Progression:** The feedback becomes more refined. It starts with a numerical score ("Output 0.6") from an ORM, progresses to a rule-based verification (checkmark from "Rule Extraction"), and culminates in a definitive "Correct" judgment based on the RLLMs' own reasoning. * **Color Coding:** All reward model boxes in this panel are green. 2. **Process Feedback (Bottom Panel):** This process evaluates intermediate steps and involves interaction with an external environment. * **Flow:** It shows a step-wise evaluation and a cyclical interaction with the environment. * **Trend/Progression:** It begins with a step-level score ("Step 0.8") from a PRM, leads to a "Correct" judgment from the RLLMs' Reasoning, and then enters a loop with the Environment. * **Environment Interaction:** The RLLMs' reasoning interacts with three types of environments: Operating System (OS Env.), Real-world (Real Env.), and Graphical User Interface (GUI Env.). The loop suggests an iterative process of action and observation. * **Color Coding:** Reward model boxes in this panel are yellow. Environment labels are light purple. ### Key Observations * **Duplicate Labeling:** Both main panels are labeled "(a)", which is likely an error in the original figure. * **Visual Metaphor:** The consistent use of the worm with a magnifying glass symbolizes scrutiny and evaluation. The hand offering a trophy symbolizes reward or scoring. * **Score vs. Binary Judgment:** The initial stages provide numerical scores (0.6, 0.8), while the final stages in both panels provide a binary "Correct" judgment. * **Complexity:** The "Process Feedback" model is more complex, incorporating an explicit, multi-faceted environment and a feedback loop, whereas "Overall Feedback" is a linear, self-contained sequence. * **Acronyms:** Key technical terms are presented as acronyms: ORM, PRM, RLLMs, OS, GUI. ### Interpretation This diagram illustrates a conceptual framework for training or evaluating reasoning AI models. It argues for a distinction between two levels of feedback: * **Overall Feedback** is akin to grading a final exam. It looks at the end result ("Output") and may use intermediate steps like rule checking, but the reward signal is tied to the final outcome. The progression from a low score (0.6) to "Correct" suggests a process of refinement or learning based on this outcome-based reward. * **Process Feedback** is akin to a tutor guiding a student through each step of a problem. It provides feedback ("Step 0.8") on the reasoning process itself, not just the answer. Crucially, it places the AI's reasoning within an interactive loop with diverse environments (digital OS/GUI and physical "Real Env."). This suggests the model learns by taking actions, observing consequences in these environments, and receiving step-wise rewards. The higher initial step score (0.8 vs. 0.6) might imply that process-level feedback provides a stronger or more immediate learning signal. The overall message is that while outcome-based evaluation (Overall Feedback) is important, a more sophisticated and potentially more effective paradigm involves evaluating and rewarding the reasoning process itself, especially when that process is grounded in interaction with realistic environments (Process Feedback). This aligns with trends in AI research towards process reward models and embodied or interactive learning. </details> Figure 7: The feedback capabilities framework for feasible reflection consists of Overall Feedback and Process Feedback. Overall Feedback includes the Outcome Reward Model (ORM) in a value format, rule extraction for correctness judgment, and overall RLLMs based on RLLMs. Process Feedback includes the Process Reward Model (PRM) in a value format and step-level RLLMs, also based on RLLMs. ### 5.1 Feedback Feedback refers to the process of providing evaluations of both overall outputs and the processes that lead to them, with the goal of assessing their accuracy and quality [394, 396, 838, 220, 862]. This process, also referred to as critique or verification, can be executed using either natural language or structured data formats, which serve as the foundation for tree-search methods [113]. Specifically, as shown in Figure 7, feedback can be categorized into three distinct types: (1) Overall Feedback (§ 5.1.1); (2) Process Feedback (§ 5.1.2); (3) Hybrid Feedback (§ 5.1.3). #### 5.1.1 Overall Feedback The overall feedback focuses on providing a global view of the entire process and results, rather than assessing each step individually. This feedback significantly enhances reasoning skills and reward modeling in reinforcement learning for RLLMs. Specifically, as shown in Figure 7 (a), the overall feedback can be categorized into three main sources: Outcome Reward Model, Rule Extraction, and RLLMs Feedback. The performance across these categories is summarized in Table 3. | Model RLLMs GPT-4o-mini [3] | Base Model - | Chat 95.0 | Chat_Hard 60.7 | Safety 80.8 | Reasoning 83.7 | Overall 80.1 | | --- | --- | --- | --- | --- | --- | --- | | Llama3.1-70B-Instruct [168] | - | 97.2 | 70.2 | 86.0 | 82.8 | 84.0 | | Llama3.1-405B-Instruct [168] | - | 97.2 | 74.6 | 87.1 | 77.6 | 84.1 | | GPT-4 [3] | - | 95.3 | 74.3 | 86.9 | 87.6 | 86.0 | | GPT-4o [3] | - | 96.1 | 76.1 | 86.6 | 88.1 | 86.7 | | Gemini-1.5-pro [719] | - | 92.3 | 80.6 | 87.9 | 92.0 | 88.2 | | Self-taught Evaluator [803] | Llama-3.1-70B-Instruct [168] | 96.6 | 84.2 | 81.0 | 91.5 | 88.3 | | SFR-LLaMA-3.1-8B-Judge [791] | Llama-3.1-70B-Instruct [168] | 95.5 | 77.7 | 86.2 | 95.1 | 88.7 | | SFR-NeMo-12B-Judge [791] | Mistral-NeMo-Instruct-12B [725] | 97.2 | 82.2 | 86.5 | 95.1 | 90.3 | | SFR-LLaMA-3.1-70B-Judge [791] | Llama-3.1-70B-Instruct [168] | 96.9 | 84.8 | 91.6 | 97.6 | 92.7 | | Skywork-Critic-Llama-3.1-70B [791] | Llama-3.1-70B-Instruct [168] | 96.6 | 87.9 | 93.1 | 95.5 | 93.3 | | LMUnit [641] | Llama-3.1-70B-Instruct [168] | - | - | - | - | 93.4 | | EvalPlanner [643] | Llama-3.1-70B-Instruct [168] | 97.5 | 89.4 | 93.0 | 95.5 | 93.9 | | Outcome Reward Models | | | | | | | | tulu-v2.5-13b-uf-rm [306] | TULU-2-13B [305] | 39.4 | 42.3 | 55.5 | 47.4 | 46.1 | | Prometheus-2-7B [353] | Mistral-7B-Instruct-v0.2 [318] | 85.5 | 49.1 | 77.1 | 76.5 | 72.0 | | Prometheus-8x7b-v2 [353] | Mixtral-8x7B-Instruct [319] | 93.0 | 47.1 | 80.5 | 77.4 | 74.5 | | Critic-RM-Rank [991] | Llama-3.1-70B-Instruct [168] | 97.0 | 58.0 | 84.0 | 92.0 | 82.8 | | RM [689] | Llama-3.1-70B-Instruct [168] | 98.3 | 74.5 | 83.8 | 88.0 | 86.4 | | SynRM [968] | Llama-3.1-70B-Instruct [168] | 97.5 | 76.8 | 86.3 | 88.5 | 87.3 | | CLoud [17] | Llama-3-70B-Instruct [168] | 98.0 | 75.6 | 87.6 | 89.0 | 87.6 | | FLAMe-RM-24B [753] | PaLM-2-24B [16] | 92.2 | 75.7 | 89.6 | 93.8 | 87.8 | | SteerLM-RM 70B [829] | Llama-2-70B-chat [743] | 91.3 | 80.3 | 90.6 | 92.8 | 88.8 | | Llama-3-OffsetBias-RM-8B [585] | Llama-3-8B-Instruct [168] | 97.2 | 81.8 | 86.8 | 91.9 | 89.4 | | InternLM-20B-Reward [62] | InternLM2-8B-Instruct [62] | 98.9 | 76.5 | 89.9 | 95.8 | 90.2 | | ArmoRM-Llama3-8B-v0.1 [771] | Llama-3-8B-Instruct [168] | 96.9 | 76.8 | 92.2 | 97.3 | 90.8 | | Nemotron-4-340B-Reward [829] | Nemotron-4-340B [4] | 95.8 | 87.1 | 92.2 | 93.6 | 92.2 | | Skywork-Reward-Llama-3.1-8B [466] | Llama-3.1-70B-Instruct [168] | 95.8 | 87.3 | 90.6 | 96.2 | 92.5 | | Skywork-Reward-Gemma-2-27B [466] | Gemma-2-27B-it [720] | 95.8 | 91.4 | 92.0 | 96.1 | 93.8 | Table 3: Performance of various overall feedback methods, sorted primarily by Overall scores in RewardBench [367]. “-” indicates that the paper did not report this score. Overall Feedback from Outcome Reward Model Since many tasks cannot be directly evaluated using accuracy or other standard metrics, research has increasingly focused on Outcome Reward Models (ORM), which provide value-based rewards for more general and quantifiable feedback [1127, 986, 467]. In 2021, OpenAI [141] has proposed a “Gen-Verifier” paradigm, which uses a specialized ORM to evaluate the accuracy of generated rationales, showing significant progress in feedback capabilities [658]. Ji et al. [315] introduce a trained knowledge scorer to analyze hallucinations in the reasoning process, providing feedback to RLLMs and improving the accuracy of their outputs over time. Moreover, Generative Reward Models [1048] use next-token prediction for overall feedback, which seamlessly integrates with instruction adjustments, leveraging inference-time calculations to improve ORM feedback. However, specifically trained ORMs are often costly and not sufficiently robust. Building on this, Self-Rewarding Language Models (SRLMs) [1129] incorporate a self-consistency framework, optimizing feedback to improve model alignment and consistency [1047]. Yu et al. [991] introduce Critic-RM, combining RLLM-generated natural language criticism with corresponding feedback. This method filters high-quality feedback while jointly fine-tuning reward prediction and criticism generation, optimizing ORM performance. Overall Feedback from Rule Extraction Although ORM has achieved significant improvements, its accuracy still falls short of 100%, preventing it from outperforming rule-based answer correction feedback [955, 234, 1079]. Previous studies, such as STaR [1012], ReST [225], and ReFT [745], have demonstrated that feedback based on final answer rewards is more effective than both PRM and ORM in mathematical scenarios [197]. Furthermore, Guo et al. [227] and Xie et al. [886] introduce a multi-stage RL framework that incorporates rule-based rewards, significantly enhancing both output accuracy and length while mitigating reward hacking through simple yet robust rules [30], such as format validation and result verification. In coding scenarios where direct rule-based feedback is difficult, OpenCodeInterpreter [1108], AceCoder [1014], O1-Coder [1076], and VerMCTS [56] address this challenge by implementing an automated test-case synthesis pipeline, deriving rewards based on program performance [564, 216, 1115]. Additionally, Ma et al. [536] propose an automated approach to training a test case generator, which alleviates the scarcity of test cases and demonstrates that increasing the number of test cases correlates with improved reward quality. Moreover, Ma et al. [535] decompose problem-solving into structured coding subtasks: file localization, function localization, line localization, and code editing generation, and applies multi-viewed rule-based rewards. Overall Feedback from RLLMs Research on feedback from RLLMs centers on detecting errors and biases through natural language feedback, also known as LLM-as-Judge, self-reflection or self-critique [274, 336, 29, 638, 549, 802, 1002, 895, 529]. This method has led to significant improvements across various tasks, particularly in self-correction [848, 1109, 206, 184, 1075]. Huang et al. [286] contend that traditional LLMs struggle to generate effective feedback without external signals, requiring the development of RLLMs with enhanced feedback capabilities [645, 398]. As a result, many studies leverage RLLMs’ error-identification strengths, often stemming from their pretraining phase, to improve feedback generation and correction [965, 39, 40, 282]. Earlier, McAleese et al. [544] found that training RLLMs to learn self-critique and deep reasoning can further boost performance. Zhang et al. [1062] propose a self-contrast mechanism that compares multiple perspectives, identifies differences, and summarizes insights to resolve inconsistencies. However, these methods often offer task-independent feedback. To address this, Hao et al. [235] introduce AutoRace, which tailors evaluation criteria for specific tasks. The Reversal of Thought (RoT) framework [999] introduces a novel paradigm combining reverse reasoning with self-reflection, helping models identify the limits of their knowledge and enhance reasoning efficiency. Furthermore, ACR [1116] implements a scoring system for coding tasks, using LLM-as-a-Judge for quality assessment and LLM-as-a-Critic for critiquing low-quality code, improving consistency across benchmarks. Zheng et al. [1107] integrate code execution error data and feedback from RLLMs to improve code generation performance. Liu et al. [484] present AGSER, a method using attention-guided self-reflection to address hallucinations by splitting input queries into attentive and non-attentive components. Finally, Saha et al. [643] introduce EvalPlanner, which separates feedback into planning and reasoning components for more streamlined expression using existing RLLMs. More comprehensively, Hu et al. [274] outline the complete pipeline, key insights, and practical lessons for training RLLMs to function as judges. #### 5.1.2 Process Feedback Techniques combine process feedback with MCTS or RL rewards to provide automated, step-by-step guidance, reducing the need for labor-intensive annotations while enhancing reasoning capabilities [749, 344]. These techniques can be categorized into two main types based on the source of feedback: process reward models (PRMs) and prompted LLMs. The performance comparison are mainly shown in Table 4. Process Feedback from Process Rewarded Model Recent studies highlight the significance of feedback in developing effective PRMs for complex reasoning tasks, particularly in a step-level view [134, 423, 528]. (1) Process Annotated PRM Training: Earlier, Lightman et al. [449] demonstrate that training process feedback with human-annotated data (PRM800K) surpasses outcome supervision in creating reliable reward models. However, this approach requires significant human effort. To address this, Wang et al. [792] introduce Math-Shepherd, a dataset that generates step-by-step supervision using a Tree Search-inspired method [73, 1001]. Following this, methods like QwQ [731], Skywork-o1 [570], AceMath [500], and PRIME [143] adopt similar techniques to enhance PRM performance. Additionally, Zhang et al. [1036] propose entropy regularization to improve model convergence. Rather than focusing solely on the first error step, Full-Step-DPO [903] assigns rewards for the entire reasoning chain, including error steps. VersaPRM [1015] extends PRMs across multiple domains, broadening their applicability. Similarly, Gu et al. [219] and Zhang et al. [1074] suggest training models with student preferences aligned to teacher preferences, ensuring effective preference distillation. Further, Wang et al. [807] propose VisualPRM400K and expand this paradigm to multimodal scenarios. (2) Outcome Annotated PRM Training: Alternative approaches, such as ReST-MCTS* [1032], OVM [979], Implicit PRM [1000], AutoPSV [506], and DVO [1038], leverage outcome supervision or implicit feedback to train PRMs, reducing the need for extensive human-annotated data [891, 643]. UAS [981] incorporates uncertainty-aware value models [275] into feedback predictions [495, 167, 945, 1089]. Additionally, Aurora [710] utilizes ensemble prompting strategies and reference answers for reverse verification, training stronger PRMs that better align with the Long CoT data distribution. Furthermore, PAV [651] suggests that rewards should reflect reasoning progress, as measured by changes in the likelihood of producing a correct future response before and after each step. Yang et al. [932], Lee et al. [376], Yoon et al. [975] extend these paradigms to the token level. Moreover, Chen et al. [110] expand these into interactive agent scenarios, allowing for automatically learning reward models from the environment without additional manual annotation. Wang et al. [832] equip a dual-layer MLP module to evaluate the reward at each step, successfully integrating the policy model and PRM into a unified interface without additional process annotations, reducing over 99% of PRM parameters for efficient reasoning. | | | ProcessBench | PRMBench | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | GSM8K | MATH | OlympiadBench | OmniMATH | Simplicity | Soundness | Sensitivity | | | | Process Reward Models | | | | | | | | | | Qwen2.5-Math-7B-PRM [1102] | Qwen2.5-Math-7B [927] | 39.4 | 52.2 | 39.4 | 33.1 | - | - | - | | Math-Shepherd-PRM-7B [792] | Mistral-7B [318] | 47.9 | 29.5 | 24.8 | 23.8 | 47.1 | 45.7 | 60.7 | | RLHFlow-PRM-Mistral-8B [156] | Mistral-7B [318] | 50.4 | 33.4 | 13.8 | 15.8 | 46.7 | 57.5 | 68.5 | | RLHFlow-PRM-DeepSeek-8B [156] | DeepSeek-7B [52] | 38.8 | 33.8 | 16.9 | 16.9 | 47.6 | 57.5 | 68.1 | | Skywork-PRM-1.5B [466] | Qwen2.5-Math-1.5B-Instruct [926] | 59.0 | 48.0 | 19.3 | 19.2 | 33.6 | 28.6 | 48.8 | | Skywork-PRM-7B [466] | Qwen2.5-Math-7B-Instruct [926] | 70.8 | 53.6 | 22.9 | 21.0 | 38.4 | 32.7 | 54.3 | | Qwen2-1.5B-PRM800k [700] | Qwen2-Math-1.5B-Instruct [927] | 34.0 | 55.3 | 34.2 | 41.0 | - | - | - | | Qwen2-1.5B-Math-Shepherd [700] | Qwen2-Math-1.5B-Instruct [927] | 48.9 | 34.1 | 9.8 | 13.7 | - | - | - | | Qwen2-1.5B-Epic50k [700] | Qwen2-Math-1.5B-Instruct [927] | 55.6 | 36.1 | 20.2 | 30.0 | - | - | - | | Qwen2.5-Math-7B-PRM800K | Qwen2.5-Math-7B-Instruct [927] | 68.2 | 62.6 | 50.7 | 44.3 | - | - | - | | Qwen2.5-Math-PRM-7B [1102] | Qwen2.5-Math-7B-Instruct [927] | 82.4 | 77.6 | 67.5 | 66.3 | - | - | - | | Universal-PRM-7B [710] | Qwen2.5-Math-7B-Instruct [927] | 85.8 | 77.7 | 67.6 | 66.4 | - | - | - | | Critic Model | | | | | | | | | | Llama-3.1-8B-Instruct [168] | - | 27.5 | 26.7 | 18.5 | 19.2 | - | - | - | | GPT-4o [3] | - | 61.9 | 53.9 | 48.3 | 44.6 | 59.7 | 70.9 | 75.8 | | QwQ-32B-Preview [731] | Qwen2.5-32B-Instruct [926] | 62.3 | 52.7 | 46.2 | 43.9 | - | - | - | | DeepSeek-R1-Distill-Qwen-14B [227] | Qwen2.5-14B-Instruct [926] | 67.3 | 38.8 | 29.9 | 32.1 | - | - | - | | Dyve-14B [1111] | DeepSeek-R1-Distill-Qwen-14B [227] | 68.5 | 58.3 | 49.0 | 47.2 | - | - | - | | Qwen2.5-72B-Instruct [926] | - | 76.2 | 61.8 | 54.6 | 52.2 | - | - | - | | SCRIT [713] | Qwen2.5-72B-Instruct [926] | 80.2 | 60.0 | 32.5 | 27.8 | - | - | - | | o1-mini [307] | - | 93.2 | 88.9 | 87.2 | 82.4 | 64.6 | 72.1 | 75.5 | | Llemma-PRM800k-7B [679] | Llemma-7B [26] | - | - | - | - | 51.4 | 50.9 | 66.0 | | Llemma-MetaMath-7B [679] | Llemma-7B [26] | - | - | - | - | 50.3 | 49.0 | 66.0 | | Llemma-oprm-7B [679] | Llemma-7B [26] | - | - | - | - | 49.0 | 49.8 | 64.1 | | MATHMinos-Mistral-7B [195] | Mistral-7B [318] | - | - | - | - | 51.4 | 54.4 | 66.5 | | ReasonEval-7B [877] | Llemma-7B [26] | - | - | - | - | 55.5 | 63.9 | 71.0 | | ReasonEval-34B [877] | Llemma-34B [26] | - | - | - | - | 51.5 | 63.0 | 73.1 | | Gemini-2.0-flash-exp [679] | - | - | - | - | - | 62.7 | 67.3 | 75.4 | | Gemini-2.0-thinking-exp-1219 [679] | - | - | - | - | - | 66.2 | 71.8 | 75.3 | Table 4: Performance of various process feedback methods on ProcessBench [1102] and PRMBench [679]. “-” indicates that the paper did not report this score. Process Feedback from RLLMs As PRM training remains heavily dependent on manually annotated data, recent research has explored methods for enabling models to generate their natural language feedback to optimize performance [910]. These approaches fall into two primary categories: (1) Model-Driven Feedback Reasoning: Earlier work such as React [956] and Reflexion [669] enhances RLLMs with natural language feedback at each action and reasoning step [196, 135, 89], improving decision-making in diverse tasks. Similarly, Step-DPO [365] uses RLLM to self-verify step-level positive and negative pairs for training through the DPO paradigm, achieving strong performance. Additionally, Sun et al. [702] propose a dynamic error classification framework that adapts based on model outputs, improving performance in mathematical reasoning tasks by addressing specific error patterns in math word problems. Furthermore, Xie et al. [889] and He et al. [245] iteratively apply MCTS to collect preference data, utilizing its forward-looking capabilities to decompose instance-level rewards into more precise step-level signals, thereby enhancing feedback accuracy. However, step-wise feedback often suffers from reliability issues, which can be mitigated by uncertainty quantification [973, 969], improving the reliability of step-wise verification in reward models for mathematical reasoning tasks. Moreover, Fu et al. [187] define the CoT Average Causal Effect (CACE) to capture causal relationships between steps, resulting in a causalized Long CoT where all steps are both correct and comprehensible. (2) Environment-Driven Feedback Reasoning: Given the increasing complexity of large models, there is growing interest in combining prompt-based LLMs with external environments to generate more interpretable and controllable feedback [885, 271]. For example, ORPS [996] and Drori et al. [162] minimize dependence on human annotations by using execution feedback, enabling models to autonomously refine their solutions. Additionally, Shrestha et al. [670] contribute by translating model outputs into Python code, helping to identify logical errors, gain insights into flawed reasoning processes, and guide improvements in mathematical reasoning. Xu et al. [897] integrate reasoning models with an interactive environment, enabling learning in more dynamic scenarios and creating a more generalizable self-learning framework. #### 5.1.3 Hybrid Feedbacks Given the respective advantages and limitations of Overall Feedback and Process Feedback, recent studies have sought to combine both for optimal feedback. Specifically, Zhang et al. [1078] propose a consensus filtering mechanism that integrates Monte Carlo estimation with an LLM-as-judge to enhance both overall and stepwise feedback, thus improving reasoning accuracy. In a similar vein, Lin et al. [454] introduce Step-KTO, a framework combining stepwise process-level and outcome-level binary feedback, using PRM and ORM to guide language models toward coherent reasoning, with a focus on error correction through reflection mechanisms. Takeaways: Feedback • Evolving Feedback Models: Feedback mechanisms, including overall, process, and hybrid feedback, are crucial for improving the reasoning capabilities of RLLMs. • Innovative Approaches in Process Feedback: Process feedback using techniques like PRMs with MCTS enhances Long CoT, though challenges like reward hacking remain. • Self-Reflection and Model-Driven Feedback: Self-reflection and model-driven feedback improve RLLM performance by enabling error detection, task-specific insights, and more autonomous learning. ### 5.2 Refinement Refinement refers to the process of addressing errors in reasoning based on prior feedback. As shown in Figure 8, refinement methods can be grouped into three primary categories: prompt-based refinement generation (§ 5.2.1), SFT-based refinement imitation (§ 5.2.2), and RL-based refinement learning (§ 5.2.3). <details> <summary>x9.png Details</summary>  ### Visual Description \n ## Diagram: Three Methods for AI Model Refinement ### Overview The image is a technical diagram illustrating three distinct methodologies for refining the outputs of AI models, specifically focusing on correcting errors (using the example "1+1=3" being corrected to "1+1=2"). The diagram is divided into three panels labeled (a), (b), and (c), each depicting a different approach using cartoon characters, text boxes, and flow arrows. The overall theme is the process of error detection and correction in AI systems. ### Components/Axes The diagram is structured into three main rectangular panels with dashed borders, arranged in a 2x2 grid (top-left, top-right, bottom). Each panel has a title and contains the following core components: * **Characters:** A recurring cartoon character (a white, worm-like figure with a red scarf) represents the AI model in various states. * **Process Boxes:** Colored boxes with text describe the core technique. * **Text Bubbles/Boxes:** Contain example dialogue or internal reasoning. * **Flow Arrows:** Indicate the direction of the process or information flow. * **Icons:** Visual symbols (magnifying glass, brain, stars) reinforce the meaning of each process. ### Detailed Analysis #### Panel (a): Prompt-based Refinement Generation * **Title:** `(a) Prompt-based Refinement Generation` * **Layout:** Top-left panel. * **Components & Flow:** 1. **Left:** The base AI character looks puzzled, with thought bubbles containing "x" marks. 2. **Arrow:** Points right to a green process box. 3. **Green Process Box:** Labeled `Prompting Strategies` with a magnifying glass icon. 4. **Arrow:** Points down to a white text box. 5. **White Text Box:** Labeled `Feasible Reflection Process`. Inside, the text reads: `1+1 is not equal to 3. Let's correct it 1+1=2.` 6. **Arrow:** Points left, back to the character, who now appears satisfied. #### Panel (b): SFT-based Refinement Imitation * **Title:** `(b) SFT-based Refinement Imitation` * **Layout:** Top-right panel. * **Components & Flow:** 1. **Left:** The base AI character looks puzzled. 2. **Arrow:** Points right to a blue process box. 3. **Blue Process Box:** Labeled `Supervised Fine-Tuning` with a magnifying glass icon. 4. **Above the Box:** A larger, more advanced character labeled `Advanced RLLMs` (likely "Reinforcement Learning Language Models") holds a magnifying glass and a speech bubble stating: `1+1=3 should be corrected to 2!` 5. **Arrow:** Points from the blue box to a new character on the right. 6. **Right Character:** A refined version of the base character, now also holding a magnifying glass, indicating it has learned from the advanced model. #### Panel (c): RL-based Refinement Learning * **Title:** `(c) RL-based Refinement Learning` * **Layout:** Bottom panel, spanning the full width. * **Components & Flow:** 1. **Left:** The base AI character looks puzzled. 2. **Arrow:** Points right to a pink process box. 3. **Pink Process Box:** Labeled `Reinforcement Learning` with a brain icon showing connections and "++" symbols. 4. **Circular Arrows:** Connect the pink box to a yellow box, indicating an iterative loop. 5. **Yellow Process Box:** Labeled `Feedback` with an icon of three stars on a ribbon. 6. **Arrow:** Points from the yellow box to a character on the right. 7. **Right Character:** The refined character, holding a magnifying glass, has a speech bubble with red text for emphasis: `Aha! I think 1+1=3 should be corrected 1+1=2!` ### Key Observations 1. **Consistent Example:** All three methods use the same simple arithmetic error ("1+1=3") as a running example to illustrate the correction process. 2. **Progression of Autonomy:** The methods show a progression from internal prompting (a), to learning from external expert demonstrations (b), to learning through interactive feedback and reward (c). 3. **Visual Metaphors:** The use of a magnifying glass consistently symbolizes inspection, analysis, or scrutiny. The character's expression changes from puzzled to satisfied/enlightened after the process. 4. **Color Coding:** Each method is associated with a distinct color for its main process box: Green (Prompting), Blue (SFT), Pink (RL). 5. **Text Emphasis:** In panel (c), the word "Aha!" and the correction are in red, highlighting the moment of insight gained through reinforcement learning. ### Interpretation This diagram serves as a conceptual comparison of paradigms for improving AI model accuracy and reliability. * **Prompt-based Refinement (a)** represents an **in-context learning** approach. The model uses its existing capabilities, guided by carefully designed prompts ("Prompting Strategies"), to self-reflect and generate its own correction. It requires no external training but relies heavily on the model's inherent reasoning ability and prompt engineering. * **SFT-based Refinement (b)** represents **imitation learning**. The model is explicitly trained (via Supervised Fine-Tuning) on datasets created by "Advanced RLLMs" or human experts. It learns to mimic the correction behavior demonstrated by these superior agents. This method is data-dependent but can instill reliable correction patterns. * **RL-based Refinement (c)** represents **learning through interaction and reward**. The model engages in a trial-and-error process (Reinforcement Learning loop) where its attempts are evaluated by a "Feedback" mechanism (which could be a reward model, human feedback, or a rule-based system). The model learns to optimize its corrections to maximize positive feedback, leading to more robust and generalized improvement. The diagram suggests that while prompting is a quick, training-free method, SFT and RL offer more profound and potentially more reliable pathways to refinement, with RL emphasizing experiential learning and self-discovery ("Aha!"). The choice of method involves trade-offs between data requirements, computational cost, and the desired level of model autonomy. </details> Figure 8: The three main categories of refinement methods, including Prompt-based Refinement Generation, SFT-based Refinement Imitation, and RL-based Refinement Learning. #### 5.2.1 Prompt-based Refinement Generation Research on prompt-based refine generation focuses on enhancing the performance of LLMs through iterative self-refinement mechanisms [578, 1091, 98, 469, 1028, 754, 818, 546]. A prominent approach involves prompting RLLMs to generate initial outputs, followed by self-feedback that iteratively refines and improves performance across tasks such as dialogue generation and mathematical reasoning [645, 539, 1101, 669, 549, 345, 750, 482], which even much reduce the hallucinations [289, 315]. Noteworthy methods, like Self-Backtracking [944], Refiner [590], and BackMath [1055], allow LLMs to adjust their reasoning autonomously, reducing unnecessary complexity in decision-making [868]. Further, Havrilla et al. [238] extend the paradigm by integrating overall-level and step-level refinements, improving refinement performance. Yang et al. [950] propose a method to decompose the self-correction capability of LLMs into "confidence" and "critique" capacities, designing probabilistic metrics to evaluate them and exploring the role of reflection mechanisms in model behavior. Additionally, MCTSr [1033], LLM2 [930], ReST-MCTS* [1032] and ReARTeR [703] emphasize dynamic reflection through iterative error correction and confidence adjustments, allowing models to autonomously refine reasoning strategies [186]. He et al. [240] extend this paradigm to multi-agent scenarios, improving both reasoning and agent system performance [936, 1128]. Moreover, Yuksekgonul et al. [1009] and Peng et al. [593] further expand the paradigm by enabling automatic prompt optimization driven by LLMs. This approach facilitates more generalized and automated refinement of input prompts across a range of tasks, as opposed to focusing solely on refining output results. However, without oracle feedback, RLLM’s self-refinement process fails, causing instability in both intermediate and final answers, leading to biases in simple factual queries and introducing cognitive biases in complex tasks [1051, 908]. #### 5.2.2 SFT-based Refinement Imitation Recent advancements in reflection-based reasoning for LLMs have led to frameworks that enhance model reasoning through self-refinement and error correction. A key approach is directly supervised fine-tuning, which allows models to learn error correction processes from advanced LLMs, thereby improving their reflective capabilities [14, 104, 406, 822, 99, 873]. Notable frameworks, such as rStar [615], improve smaller language models through self-play mutual reasoning, while Recursive Introduction [627] and RealCritic [714] use iterative feedback mechanisms to identify and correct errors to better self-improve [393]. Yan et al. [924] propose constructing step-wise self-correction data and implementing a training strategy that uses the above-constructed data to equip LLMs with spontaneous step-level self-correction capacities. Building upon these, Gao et al. [196] and Zhang et al. [1027] propose Math-Minos, which employs step-by-step natural language feedback as rationale tags, offering both correctness and detailed explanations for each step to train feedback mechanisms that justify and refine the reasoning process. Journey Learning [623] employs MCTS to parse node backtracking as natural language refinement, enhancing supervised fine-tuning and, thereby, improving reasoning performance. Additionally, approaches like ProgCo [682] emphasize iterative feedback and program-driven refinement to enhance critique and self-correction. Expanding these ideas to multimodal settings, frameworks, such as R3V [120] and MM-Verify [697], focus on integrating visual and textual reasoning [519, 813]. #### 5.2.3 RL-based Refinement Learning In recent research, several approaches have been proposed to enhance the performance of refinement through reinforcement learning [673, 1056]. Earlier, Kumar et al. [358] observed that SFT of RLLMs often fails to promote self-refinement behaviors. This limitation stems from a distributional mismatch between data collection strategies and model responses, as well as the risk of behavioral collapse. To address this, SCoRe [358] enhances self-refinement by training the model on its own self-generated correction trajectories and employing regularization to guide the learning process. This method prioritizes fostering self-refinement during testing, rather than merely maximizing reward for specific prompts [1018]. Further, Guo et al. [227] demonstrate that applying outcome-level rewarded RL can trigger an “Aha moment,” activating the model’s natural feedback and refinement behaviors without the need for human guidance. Moreover, Guo et al. [227], Zeng et al. [1017] and Ma et al. [529] explore initializing LLMs with iterative self-verification and self-correction behaviors, which are strengthened through supervised fine-tuning and further enhanced by outcome-level RL. Ma et al. [529] and Yang et al. [935] extend these capabilities with process-level RL, minimizing resource usage while enabling adaptive reasoning refinements during inference. More recently, Lee et al. [374] introduce an intrinsic verifier module to decide when refinements should be applied, using RL to further encourage self-refinement when errors are detected. Takeaways: Refinement • Prompt-Based Refinement for Iterative Improvement: Iterative self-refinement through feedback loops helps LLMs improve reasoning and reduce errors like hallucinations but requires stable feedback to maintain accuracy. • Supervised Fine-Tuning (SFT) for Error Correction: Supervised fine-tuning enhances LLMs by using iterative feedback and self-correction strategies to improve reasoning accuracy, especially for smaller models. • Reinforcement Learning (RL) for Refinement: Reinforcement learning enhances self-refinement in LLMs by using self-generated corrections and adaptive strategies, reducing human intervention and resource consumption. ## 6 Extensive Exploration for Long CoT Exploration is a key capability in Long CoT reasoning, allowing models to navigate complex problem spaces through strategic branching and iterative refinement [1019, 381, 784, 751]. Recent studies emphasize exploration mechanisms, such as hypothesis branching and error backtracking via reflection, as essential for overcoming the constraints of linear reasoning paths [227]. Current research focuses on three key areas: (1) Exploration Scaling (§ 6.1), which explores the breadth and depth of exploration and its impact on downstream applications, particularly in improving the size of the exploration path $m$ in Equation (3); (2) Internal Exploration (§ 6.2), which focuses on training models to develop internal exploration capabilities, enabling more efficient and effective generation of $m$ exploration paths $\{n_i+j\}^m_j=1$ in Equation (3); and (3) External Exploration (§ 6.3), which examines how models can leverage external systems to enhance their exploratory abilities, facilitating the selection of the most effective path $n_i+j$ from the $m$ exploration paths in Equation (3). <details> <summary>x10.png Details</summary>  ### Visual Description ## Technical Diagram: Comparison of Sequential vs. Parallel Scaling Methodologies ### Overview The image is a technical diagram comparing two approaches to scaling reasoning processes in AI systems, labeled as "(a) Sequential Scaling" and "(b) Parallel Scaling." It uses a combination of flowcharts and illustrative cartoons to contrast the limitations of a sequential approach with the proposed benefits of a parallel, verification-based approach. ### Components/Axes The diagram is divided into two primary panels: **Panel (a): Sequential Scaling (Left Side)** * **Flowchart Components:** * Three yellow rectangular blocks labeled **"Reasoning Process"**, each containing an icon of three stylized human figures. * These blocks are connected in a linear, top-to-bottom chain by black arrows. * A red line with a scissors icon cuts the connection between the third and a subsequent, partially obscured red block labeled **"...."**. * A green rectangular block at the bottom labeled **"RLLM Supply"** with a battery icon, connected via an arrow to the chain of reasoning processes. * Vertical red text to the right of the flowchart reads: **"Limited by Reasoning Boundary"**. * **Illustrative Cartoon:** * Depicts a hand gripping a sword. A second hand attempts to cut the sword with a smaller blade labeled **"Sequential Scaling"**. * A vertical double-headed arrow next to the sword is labeled **"Reasoning Boundary"**. * The cartoon visually represents the concept of a single point of failure or limitation in a sequential chain. **Panel (b): Parallel Scaling (Right Side)** * **Flowchart Components:** * Three yellow **"Reasoning Process"** blocks (identical to those in panel a) arranged vertically but operating in parallel. * Their outputs feed into a dashed box containing three verification methods, each with an associated resistance value: * **"Self-Consistency"** (100Ω) * **"Self-Verification"** (20Ω) * **"Pass@k"** (0Ω) * A blue block labeled **"(2) Verification"** with a crossed-out sword icon receives input from the verification methods. * A green **"RLLM Supply"** block (same as in panel a) is connected to the system. * A two-stage process is indicated: **"(1) Sampling"** (in purple text) leading to the reasoning processes, and **"(2) Verification"** leading to the final output. * A final arrow points to the word **"Output"**. * **Illustrative Cartoon:** * Shows three hands, each holding a sword, working together. Each sword is labeled **"Sampling*1"**. * Below them, a blue block labeled **"(2) Verification"** with the crossed-out sword icon is shown. * An arrow points downward from this block to the word **"Output"**. * This illustrates the concept of multiple parallel attempts ("sampling") followed by a verification step to produce a reliable result. ### Detailed Analysis * **Sequential Scaling (a):** The model depicts a linear pipeline where reasoning processes depend on the completion of the previous one. The red scissors and cut line explicitly show this chain is vulnerable to interruption or failure at any point, which is defined as the **"Reasoning Boundary."** The system is powered by a single **"RLLM Supply."** * **Parallel Scaling (b):** This model decouples generation from verification. The **"(1) Sampling"** phase generates multiple independent reasoning traces (the three parallel "Reasoning Process" blocks). These are then evaluated by a **"(2) Verification"** phase using methods like **Self-Consistency**, **Self-Verification**, and **Pass@k**. The decreasing resistance values (100Ω → 20Ω → 0Ω) metaphorically suggest that verification becomes less "costly" or difficult as the methods progress, with **Pass@k** having zero resistance. The final, verified result is the **"Output."** ### Key Observations 1. **Structural Contrast:** The core contrast is between a fragile, linear chain (a) and a robust, parallel-and-verify system (b). 2. **Visual Metaphors:** The sword imagery is central. In (a), a single sword is cut. In (b), multiple swords are combined and then verified, making the system more resilient. 3. **Terminology:** Key technical terms introduced are **"Reasoning Boundary"** (the limit of sequential processing) and the verification methods **"Self-Consistency," "Self-Verification,"** and **"Pass@k."** 4. **Resource Label:** Both systems are shown to be supported by an **"RLLM Supply,"** indicating a common resource or model foundation. ### Interpretation This diagram argues for a paradigm shift in scaling AI reasoning capabilities. It posits that simply chaining reasoning steps (Sequential Scaling) hits an inherent **"Reasoning Boundary"**—a point where the process becomes too brittle or complex to continue linearly. The proposed solution is **Parallel Scaling**, which embraces a "generate-then-verify" workflow. By producing multiple candidate solutions (Sampling) and then applying rigorous, multi-faceted verification (Self-Consistency, etc.), the system can overcome the limitations of any single reasoning path. The metaphor of resistance decreasing to 0Ω implies that the verification stage, while adding a step, ultimately reduces the overall "friction" or error rate in reaching a correct output. The diagram is likely from a research paper or technical proposal advocating for verification-based architectures in large language models (LLMs) or reasoning systems. </details> Figure 9: Schematic representations of two common inference-time scaling strategies: (a) sequential scaling, which extends the length of Long CoT but is constrained by the reasoning boundaries of RLLMs; and (b) parallel scaling, which increases the sample size and aggregates multiple outcomes, yet does not surpass the performance of Pass@k. ### 6.1 Exploration Scaling Recent advances in inference-time scaling algorithms [333, 843, 57, 1053, 112] have attracted significant interest, particularly in scaling reasoning length to improve performance [524, 568, 405, 779]. Following Chen et al. [93], as shown in Figure 9, exploration scaling can be understood through two paradigms: (1) sequential scaling, akin to a series of resistors, which connects multiple reasoning processes using reflection; and parallel scaling, similar to parallel resistors, where a unified verification/feedback mechanism selects the most effective reasoning processes. #### 6.1.1 Sequential Scaling Sequential scaling refers to extending the reasoning output within a single model generation, significantly boosting model performance [383, 1052, 348]. Early works by Fu et al. [189] and Jaech et al. [307] show that increasing the length of the reasoning path can greatly improve performance. Tian et al. [736] enhances model reasoning iteratively by using prior answers as prompts for each successive round, thus enabling sequential scaling of the reasoning process. Building on this, later studies [314, 391] further explore enhancing logical depth through tree-based searches within a fixed compute budget, resulting in notable performance gains [11, 614]. Building upon this, Muennighoff et al. [560] introduce a inference-time scaling method that improves reasoning by fine-tuning and budget forcing, yielding substantial gains with additional computing at inference time. To address the constraints of attention spans, some studies focus on expanding reasoning length in latent spaces. Geiping et al. [204] and Chen et al. [109] enhance inference-time reasoning performance by implicitly scaling computation in latent space through recurrent depth. Setlur et al. [653] identified three core aspects of sequential scaling: (1) linking skills to asymmetric capabilities in base LLMs, such as connecting easy verification with difficult exploration; (2) enhancing exploration in reinforcement learning by utilizing the “negative” gradient of error trajectories, which extends search paths and links additional asymmetries; and (3) creating dynamic exploration by aligning task difficulty with training token budgets through tailored curricula. #### 6.1.2 Parallel Scaling Parallel scaling refers to the process of increasing the number of reasoning iterations during model generation and then verfiy these results to get the final output, which significantly enhances model performance [2, 864, 57, 485, 59, 1139]. Initially, Wang et al. [816] introduce the concept of self-consistency, demonstrating that multiple sampling processes followed by majority voting for effective exploration. Verification Optimization The primary focus of recent research is optimizing verification, which can be categorized into two types: (1) Overall Verification: Recent works [1120, 831] divide the scaling process into two stages: "reasoning" and "self-verification." By replacing majority voting in self-consistency with self-verification, these approaches show significant improvements [1083, 81, 1149, 364, 426]. In code scenarios, WoT [1071], CISC [716] and S* [392] scale the Long CoT in parallel, using output confidence or code execution results for verification, effectively assessing reasoning quality [635, 203, 278, 1134]. Further, Nye et al. [569] and Weir et al. [842], Stoisser et al. [690] train RLLMs to simulate code execution, removing the need for test cases in code-related parallel scaling. Chain-of-Verification [93] introduces meta-verification, sampling multiple verification instances to identify the correct one. Kim et al. [351], Chen et al. [111], and Vacareanu et al. [750] validate this approach empirically by evaluating answer correctness based on reasoning path properties. Moreover, Li et al. [421] tune a specific RLLM to verify and aggregate answers, showing improved performance. This suggests that PRM cannot replace a specially trained RLLM for verification due to training goal biases [1078]. Finally, Kang et al. [341] leverage self-uncertainty to select the best results. (2) Step Verification: Building on this, numerous researchers have explored step-level or finer-grained verification [84, 460]. Notably, DIVERSE [425], SSC-CoT [1098], and Fine-grained Self-Consistency [93] combine diverse reasoning paths with step-level verification. In addition, a series of works [676, 864, 517, 770, 853, 486] try to investigate how optimal scaling strategies based on MCTS can enhance smaller language models’ performance. Their findings show that a 1B RLLM can outperform a 405B model on complex tasks through parallel scaling [988]. Despite these advancements in verification, Chen et al. [93] demonstrate that these strategies cannot surpass Best-of-N methods, suggesting that breakthroughs cannot solely rely on optimization-based verification [106]. Sampling Optimization Another key area of research focuses on generating diverse but less paths or strategies for efficient scaling [871, 765, 80, 668, 444, 681]. For instance, Zeng et al. [1020] aggregate the shortest yet most varied reasoning paths for better scalability. Similarly, Du et al. [164] adjust the sampling temperature to increase diversity, leading to improved scaling. Zhang et al. [1045] and Liu et al. [470] optimize both candidate solution generation (e.g., prompts, temperature, and top-p) and reward mechanisms (such as self-evaluation and reward types), offering diverse strategies for parallel scaling. Moreover, Qin et al. [617], Luo et al. [520], and Yu et al. [990] enhance RLLM reasoning by scaling sampling across multiple natural and programming languages or varied expressions. Finally, Yang et al. [943] introduces a method where a small set of seed data, with varied response lengths, guides the model to engage in deeper reasoning by selecting the shortest correct responses across various inference efforts. Takeaways: Exploration Scaling • Exploration Mechanisms in Long CoT Reasoning: Exploration strategies like hypothesis branching and error backtracking are vital for overcoming limitations in linear reasoning paths and enhancing model performance. • Scaling Exploration: Exploration can be scaled through sequential and parallel strategies to improve reasoning depth and efficiency. • Verification and Sampling Optimization: Refining verification techniques and optimizing sampling for diverse reasoning paths are key to improving exploration efficiency and performance in Long CoT tasks. ### 6.2 Internal Exploration As noted in Chu et al. [137], Shen et al. [661], and Yang et al. [938], SFT serves as a memory process, while RL enhances generalization [359, 82]. Specifically, SFT stabilizes the model’s output format, whereas RL improves its generalization capacity, which can increase learning efficiency by up to eight times in tasks such as mathematical reasoning [650]. Consequently, as shown in Figure 10, leading research emphasizes the role of RL and reward strategies in enhancing the exploration capabilities of LLMs without external assistance. The performance comparison is presented in Table 5. <details> <summary>x11.png Details</summary>  ### Visual Description ## Diagram: Reinforcement Learning Strategies and Reward Mechanisms ### Overview The image is a two-part technical diagram illustrating components and workflows in reinforcement learning (RL) systems, specifically for language model training. It is divided into two main panels labeled (a) and (b), each enclosed in a dashed border. The diagram uses cartoonish snake characters to represent agents or processes, along with labeled boxes and icons to depict models and data flow. ### Components/Axes The diagram is not a chart with numerical axes but a process flow diagram. Its components are: **Panel (a): RL Strategies** * **Input:** A cartoon snake (left) provides a "Question". * **Core Processing Block:** A green box labeled "Policy Model" with a lightning bolt icon. * **Output:** The Policy Model produces an "Output". * **Evaluation Models:** The "Output" is fed into three parallel models: 1. A green-dashed box labeled "Reference Model" (lightning bolt icon). 2. A yellow box labeled "Reward Model" (star/ribbon icon). 3. A blue-dashed box labeled "Value Model" (diamond icon). * **Aggregation:** The outputs of these three models feed into a purple box labeled "Advantage Aggregation" (cube icon). * **Feedback Loop:** The aggregated result, labeled "Advantage", is fed back to update the "Policy Model". * **Output:** A final cartoon snake (right) receives the result. **Panel (b): Reward Strategies** * **Input:** Two boxes labeled "Question" and "Output". * **Core Component:** A yellow box labeled "Reward Model" (star/ribbon icon). * **Two Strategy Paths:** The Reward Model's function is split into two illustrated approaches: 1. **Rule-Rewarded RL (Left, yellow background):** A snake uses a magnifying glass to inspect a document. Associated labels: "Regex Match", "Test Case". 2. **Model-Rewarded RL (Right, orange background):** A snake uses a magnifying glass to inspect a computer model. Associated labels: "ORM", "PRM". ### Detailed Analysis **Panel (a) - RL Strategies Flow:** 1. **Trend/Flow:** The process is a closed-loop system. A question initiates the cycle, which flows left-to-right through the Policy Model to generate an output. This output is then evaluated by three distinct models (Reference, Reward, Value). Their evaluations are combined in the Advantage Aggregation module. The calculated "Advantage" signal is then used to update the Policy Model, completing the feedback loop. The final, improved output is delivered to the end-user (right snake). 2. **Component Relationships:** The Policy Model is the central actor being trained. The Reference, Reward, and Value models act as critics or evaluators. The Advantage Aggregation synthesizes their feedback into a single training signal. **Panel (b) - Reward Strategies Breakdown:** 1. **Rule-Rewarded RL:** This approach uses deterministic, rule-based systems for evaluation. The visual metaphor shows inspection of a document. The specific methods mentioned are "Regex Match" (pattern matching in text) and "Test Case" (execution against predefined inputs/outputs). 2. **Model-Rewarded RL:** This approach uses other machine learning models for evaluation. The visual metaphor shows inspection of a computational model. The specific model types mentioned are "ORM" (Outcome Reward Model) and "PRM" (Process Reward Model). ### Key Observations * **Visual Metaphor:** The consistent use of a cartoon snake character creates a cohesive visual narrative for the "agent" or "process" at different stages. * **Iconography:** Each model type has a distinct icon (lightning bolt, star/ribbon, diamond, cube) for quick visual identification. * **Color Coding:** Colors are used functionally: green for the Policy Model, yellow for the Reward Model, blue for the Value Model, and purple for the Aggregation step. * **Spatial Layout:** Panel (a) is a horizontal, linear flow with a feedback loop. Panel (b) is a comparative layout, placing two alternative strategies side-by-side for direct contrast. * **Language:** All text in the diagram is in English. ### Interpretation This diagram explains a sophisticated reinforcement learning framework, likely for aligning or improving large language models (LLMs). * **What it demonstrates:** It shows how an LLM (the Policy Model) can be iteratively improved. Instead of just comparing its output to a static dataset, it is evaluated by a suite of specialized models (Reference, Reward, Value). The "Advantage" signal likely represents how much better this output is compared to a baseline, guiding the policy update. * **Relationship between elements:** Panel (a) defines the *architecture* of the RL training loop. Panel (b) zooms in on a critical component within that architecture—the Reward Model—and details two philosophies for implementing it: one based on hard-coded rules (deterministic, interpretable) and one based on learned models (potentially more nuanced but less transparent). * **Notable implications:** The inclusion of both "Rule-Rewarded" and "Model-Rewarded" strategies highlights a key design choice in RL for AI: the trade-off between the precision/reliability of rules and the flexibility/generalization of learned reward models. The presence of ORM and PRM suggests a focus on evaluating not just the final answer (Outcome) but potentially the reasoning steps (Process) to generate it. This framework is characteristic of advanced techniques used to make AI systems more helpful, harmless, and accurate. </details> Figure 10: Two primary approaches for optimizing Internal Exploration: improving RL strategy through reference and value models, and designing reward strategies: either rule-based or model-based rewarding to enhance RL performance. #### 6.2.1 RL Strategies Recent advancements in RL strategies for exploration have led to notable improvements in various tasks, particularly in reasoning tasks [699, 369, 313, 542, 882, 1017, 985, 268, 1010, 628, 150, 176, 686]. (1) Reward-free RL: The first series of work focuses on RL optimization algorithms. Additionally, OREO [773] propose an offline RL method that optimizes the soft Bellman equation, improving credit assignment for multi-step reasoning tasks and outperforming existing approaches in fields like mathematics and agent control. Liu et al. [476] propose Direct Advantage Policy Optimization, a novel offline RL method that leverages a separately trained critic to evaluate the accuracy of each reasoning step. This technique provides dense feedback for policy optimization, addressing both sparse rewards and training instability. Further, some research focuses on adjusting the focus of RL algorithms to optimize exploration in targeted aspects. Specifically, CPL [801], cDPO [457], and Focused-DPO [1043] enhance exploration in Long CoT by prioritizing critical or error-prone areas through preference optimization, improving accuracy in those regions. Bartoldson et al. [42] further adjusts the replay strategy of the training data, aiming to optimize reasoning performance. Li et al. [420] introduce Learning Impact Measurement (LIM), an automated method for evaluating and prioritizing training samples based on their alignment with model learning trajectories. This approach enables efficient resource use and scalable implementation. For instance, ThinkPO [942] uses short CoT reasoning outputs as rejected answers and longer ones as chosen answers for the same question, applying DPO to encourage prioritization of longer reasoning outputs [1131]. (2) Reward-based RL: Reward-model-based RL refers to approaches that use a reward model or a verifier to guide learning and decision-making in the absence of explicit rewards [1046, 174, 649, 279, 825, 847, 970]. Earlier, Proximal Policy Optimization (PPO) was first introduced by Schulman et al. [648], which alternates between interacting with the environment to collect data and optimizing a surrogate objective function via stochastic gradient ascent, surpassing DPO [306]. Subsequently, ReMax [436] eliminates the need for additional value models in PPOs. By incorporating variance reduction and REINFORCE [704] techniques, it reduces over four hyperparameters, resulting in lower GPU memory usage and faster training. Building on this, DeepSeekMath [658] proposes Group Relative Policy Optimization (GRPO), replacing traditional value models with improved sampling strategies, thus significantly accelerating learning and achieving performance on par with GPT-4 in mathematics. Hu [265] and Liu et al. [499] further refine GRPO with REINFORCE++ and Dr. GRPO, respectively, simplifying the algorithm and enhancing its training. Additionally, Vassoyan et al. [752] and [1121] improve exploration efficiency in smaller models by modifying the KL penalty, thus enhancing performance under distribution shifts. Huang et al. [277] introduce Decoupled Value Policy Optimization (DVPO), a streamlined framework that replaces reward modeling with a pretrained global value model (GVM) and eliminates the interdependence between actor and critic. To address the high-quality demands of reward models, Cui et al. [143] propose PRIME (Process Reinforcement through IMplicit rEwards), which integrates the SFT model as a PRM within a unified reinforcement learning framework, enabling online updates through policy rollouts and outcome labels via implicit process rewards. More recently, Liang et al. [439] introduce Self-aware Weakness-driven Problem Synthesis, a reinforcement-learning method that generates challenges tailored to an RLLM’s specific weaknesses [863, 183]. By concentrating training on its most difficult aspects, the model achieves more focused and effective reasoning improvements [680]. Wang et al. [805] introduce ROLL, a method designed to support R1-level large-scale training of RLLMs, enabling the efficient exploration and optimization of reasoning paths within the Mixture-of-Experts (MOE) structure [788]. Fu et al. [188] introduce AReaL, a large-scale asynchronous reinforcement learning system for language reasoning, which enhances the efficiency and effectiveness of training RLLMs. Ma et al. [526] propose a novel method combining interleaved SFT and RL to address challenging questions where RL typically fails. This approach enables RLLMs to learn from mistakes and enhance reasoning abilities. Huang et al. [297] and Fu et al. [190] further improve exploration efficiency by integrating SFT and RL with prefix sampling. Fruthermore, Yan et al. [917] and Liang et al. [437] guide RLLMs in reasoning under off-policy reinforcement learning [413, 773], improving both training sample efficiency and learning stability [559]. | Method Base Model GPT-4o [3] | Backbone - | GSM8K 92.9 | AIME 2024 9.3 | MATH 500 76.6 | GPQA 53.6 | LiveCodeBench 33.4 | | --- | --- | --- | --- | --- | --- | --- | | Llama-3.1-70B-Instruct [168] | - | 94.1 | 13.3 | 68.0 | - | - | | Claude 3.5 Sonnet [19] | - | - | 16.0 | 78.3 | 65.0 | 38.9 | | Qwen2.5-Coder-32B-Instruct [301] | - | - | 20.0 | 71.2 | 33.8 | 25.0 | | Qwen2.5-70B-Instruct [926] | - | - | 20.0 | 79.4 | 49.0 | 33.0 | | Llama-3.3-70B-Instruct [168] | - | - | 36.7 | 73.9 | 50.5 | 34.8 | | DeepSeek-V3 [463] | - | - | 39.2 | 90.2 | - | 36.2 | | SFT Strategies | | | | | | | | DeepSeek-R1-Distill-Llama-70B [227] | - | - | 70.0 | - | - | 57.9 | | DeepSeek-R1-Distill-Qwen-32B [227] | - | - | 72.6 | - | - | 54.6 | | START [388] | QwQ-32B-preview [731] | - | 66.7 | 94.4 | 63.6 | 47.3 | | RL Strategies | | | | | | | | DPO [631] | DeepSeekMath 7B [658] | 82.4 | - | - | - | - | | KTO [171] | DeepSeekMath 7B [658] | 82.5 | - | - | - | - | | OREO [773] | DeepSeekMath 7B [658] | 86.9 | - | - | - | - | | PPO [648] | GLM4-9B-SFT [211] | 85.5 | - | - | 31.5 | 24.3 | | GRPO [658] | GLM4-9B-SFT [211] | 86.1 | - | - | 31.7 | 22.8 | | Eurus-2-7B-PRIME [143] | Qwen2.5-Math-7B-Base [927] | - | 26.7 | 79.2 | - | - | | Search-o1 [418] | QwQ-32B-preview [731] | - | 56.7 | 86.4 | 63.6 | 33.0 | | Reward Strategies | | | | | | | | OpenMath2 [739] | Llama-3.1-70B [168] | 94.1 | 13.3 | 71.8 | - | - | | Satori [661] | Qwen-2.5-Math-7B | 93.9 | 23.3 | 83.6 | - | - | | T1-SFT [264] | Qwen2.5-32B [926] | - | 24.9 | 83.4 | 49.5 | - | | T1 [264] | Qwen2.5-32B [926] | - | 50.6 | 92.4 | 56.1 | - | | DeepSeek-R1-lite [227] | - | - | 52.5 | 91.6 | 58.5 | 51.6 | | rStar-Math [222] | Qwen2.5-Math-7B [927] | 95.2 | 53.3 | 90.0 | - | - | | QwQ-32B-preview [731] | - | 95.5 | 53.3 | 90.6 | 58.2 | 40.6 | | o1-preview [307] | - | - | 56.7 | 85.5 | 73.3 | 53.6 | | o3-mini-low [307] | - | - | 60.0 | - | - | 61.8 | | o1-mini [307] | - | - | 63.6 | 90.0 | - | 53.8 | | Kimi k1.5 [722] | - | - | 77.5 | 96.2 | - | 62.5 | | QwQ-32B [731] | - | - | 79.5 | - | - | 73.1 | | o3-mini-medium [307] | - | - | 79.6 | - | - | 72.3 | | DeepSeek-R1 [227] | - | - | 79.8 | 97.3 | - | 71.6 | | o1 [307] | - | - | 83.3 | 96.4 | - | 67.4 | | o3-mini-high [307] | - | - | 87.3 | - | - | 84.6 | Table 5: Performance of various internal exploration methods on different benchmarks, primarily ordered by AIME 2024. “-” indicates that the paper did not report this score. #### 6.2.2 Reward Strategies Rule-rewarded RL The studies explore advancements in training advanced RLLMs using rule-rewarded RL to enhance exploration strategies and reasoning accuracy [296]. These efforts primarily focus on three types of rewards: (1) Correctness Rewarding: Correctness rewards are fundamental for guiding RLLMs toward accurate answers. Specifically, Singh et al. [674] introduce a binary reward system (positive or negative) to facilitate exploration, achieving simple yet effective performance improvements. Similarly, the DeepSeek-R1 [227] employs rule-extracted accuracy as an RL reward, scaling this approach to larger scenarios and training sizes, thereby enhancing both exploration and reasoning tasks [522, 170]. Furthermore, O1-Coder Zhang et al. [1076], StepCoder [161], and SWE-RL [841] address challenges in code generation by developing a test case generator, which standardizes code testing, ensuring accurate generation [893, 994]. (2) Format Rewarding: Further, format rewards are used to encourage better reasoning paradigms. Guo et al. [227] introduce this concept to effectively guide reasoning and exploration [886]. Xie et al. [886] expanded on this with a three-stage, rule-based RL approach, enabling the Qwen-7B model to learn complex multi-path exploration, which significantly improved both output format and corresponding length consistency. Additionally, Wu et al. [855] propose TAPO (Thought-Augmented Policy Optimization), a framework that integrates external high-level guidance (“thought patterns”) into RL, successfully balancing model exploration with external guidance. (3) Scaling rewarding: Moreover, scaling rewards are applied to promote longer reasoning chains and broader exploration. Recent studies [90, 583, 349] highlight the need for progressively scaled reasoning lengths to overcome the limitations of current reasoning approaches. As a result, research has focused on scaling exploration [886, 962]. However, excessive scaling can lead to inefficiency and overcomplicated reasoning [142]. Kimi-K1.5 [722], Yang et al. [943] and Arora and Zanette [22] proposed Long2Short techniques, favoring shorter, more accurate reasoning may also significantly improve efficiency and performance. Model-rewarded RL It refers to a class of techniques in which RL algorithms are enhanced by leveraging additional reward models, to guide exploration and improve decision-making processes [693]. Earlier in 2021, OpenAI [141] propose a “Gen-Verifier” paradigm to train a correctness-oriented ORM and used ORM-rewarded RL to surpass SFT performance. Recently, with rapid advancements in PRM, several studies [755, 1032, 518] have scaled reinforcement learning by enhancing exploration through step-level correctness rewarding [659, 1042]. Building on this, Hou et al. [264] introduce entropy rewards and dynamic regularization to further optimize the reasoning process [116]. STeCa [768] identifies suboptimal actions during exploration by comparing step-level rewards and adjusting trajectories to improve deep reasoning. Additionally, the Kimi-K1.5 model [722] extends PRM paradigms into multimodal scenarios, achieving state-of-the-art performance in multi-modal reasoning tasks through a streamlined reinforcement learning framework. Takeaways: Internal Exploration • SFT and RL Synergy: The combination of Self-Feedback Training (SFT) and Reinforcement Learning (RL) improves model output stability and generalization, enhancing learning efficiency in reasoning tasks. • Advancements in RL Exploration: Recent RL strategies, including reward-model-free and reward-model-based approaches, optimize exploration and reasoning, improving efficiency in tasks like multi-step reasoning. • Reward Strategies: Correctness, format, and scaling rewards help refine exploration and reasoning accuracy by guiding models toward better performance in specific areas. <details> <summary>x12.png Details</summary>  ### Visual Description ## Diagram: Comparison of Human-Driven vs. Model-Driven Exploration Strategies ### Overview The image is a conceptual diagram comparing two paradigms for problem-solving or search exploration: **(a) Human-Driven Exploration** and **(b) Model-Driven Exploration**. It uses abstract node-link diagrams to illustrate different structural approaches (Line, Tree, Graph) and the components involved in a model-driven system. The diagram is informational and conceptual, not a data chart. ### Components/Axes The diagram is split into two main panels, each enclosed in a dashed border. **Panel (a): Human-Driven Exploration (Left Panel)** * **Title:** "(a) Human-Driven Exploration" (top-left). * **Structures:** Three distinct exploration structures are shown, each within a colored, dashed-outline box. 1. **Line (Yellow Box, Left):** A simple, linear sequence of nodes. 2. **Tree (Green Box, Center):** A branching tree structure. 3. **Graph (Blue Box, Right):** A more complex, interconnected graph structure. * **Node Legend (Implied by visual patterns):** * **White Circle:** Starting or initial state. * **Green Hatched Circle:** A promising or selected node/path. * **Yellow Hatched Circle:** An explored or candidate node. * **Blue Hatched Circle:** A subsequent step or state in the process. * **Red Circle with White 'X':** A failed or pruned path (only in the Tree structure). * **Green Circle with White Checkmark:** A successful terminal state or solution. **Panel (b): Model-Driven Exploration (Right Panel)** * **Title:** "(b) Model-Driven Exploration" (top-left). * **Core Process Flow:** A central diagram shows a cyclical process with three labeled phases: 1. **Reflection (Top, Green Box):** Contains a loop between a white circle and a green hatched circle. 2. **Exploration (Center, Yellow Box):** Shows a green hatched node branching into three yellow hatched nodes. 3. **Deep Reasoning (Bottom, Blue Box):** Shows a vertical sequence of blue hatched nodes leading to a green checkmark terminal node. * **Supporting Modules (Right Side):** Two blue boxes list components of the model-driven system. * **Logic (Top Box):** Contains the items: "Beam", "A*", "MCTS". * **Feedback (Bottom Box):** Contains the items: "PRM", "Critic", "Advantage", "Q-Value", "Verifier". * **Additional Element:** A cartoon axolotl wearing a red scarf is positioned to the left of the "Deep Reasoning" phase, with the text "Deep" vertically aligned next to it. ### Detailed Analysis **Panel (a) - Human-Driven Exploration:** * **Line Structure:** A single path: White Circle → Blue Hatched Circle → Blue Hatched Circle → Green Checkmark. This represents a straightforward, sequential approach. * **Tree Structure:** A root (White Circle) branches to a Green Hatched node, which further branches into three Yellow Hatched nodes. Two of these lead to Red 'X' nodes (failure), while the third leads to a Blue Hatched node, then another Blue Hatched node, and finally a Green Checkmark. This illustrates trial-and-error with pruning of bad paths. * **Graph Structure:** A root (White Circle) connects to a Green Hatched node, which connects to three Yellow Hatched nodes. These nodes have interconnected arrows leading to a single Blue Hatched node, followed by another Blue Hatched node, and a Green Checkmark. This shows a more complex, interconnected search space. **Panel (b) - Model-Driven Exploration:** * The process is not linear but cyclical and modular. * **Reflection Phase:** Involves self-assessment or backtracking (loop between white and green nodes). * **Exploration Phase:** Involves generating multiple candidates (yellow nodes) from a promising state (green node). * **Deep Reasoning Phase:** Involves a focused, sequential refinement (blue nodes) leading to a solution. * The **Logic** module lists search/decision algorithms (Beam Search, A*, Monte Carlo Tree Search). * The **Feedback** module lists evaluation and learning mechanisms (Process Reward Model, Critic, Advantage function, Q-Value, Verifier). ### Key Observations 1. **Structural Evolution:** Human-driven methods are depicted as static structures (Line, Tree, Graph), while the model-driven method is a dynamic, multi-phase process. 2. **Failure Handling:** The Tree structure in (a) explicitly shows failed paths (Red X's). The model-driven diagram in (b) implies failure handling is integrated into the "Reflection" and "Feedback" components rather than being a terminal state. 3. **Complexity:** The model-driven approach incorporates explicit algorithmic components (Logic, Feedback) that are abstracted away in the human-driven diagrams. 4. **Visual Metaphor:** The axolotl with "Deep" likely symbolizes "Deep Learning" or "Deep Reasoning" as the engine powering the model-driven process. ### Interpretation This diagram argues that **model-driven exploration is a more sophisticated, integrated, and iterative process compared to traditional human-driven search strategies.** * **Human-Driven (a)** represents classical problem-solving heuristics: trying a straight line, a branching tree (with backtracking), or a complex graph. The human provides the strategy and evaluates success/failure. * **Model-Driven (b)** represents an AI system's approach. It decomposes the task into specialized, interacting modules: * **Logic** provides the search algorithms. * **Exploration** generates possibilities. * **Deep Reasoning** performs detailed analysis. * **Reflection** allows for meta-cognition and strategy adjustment. * **Feedback** mechanisms (like critics and verifiers) provide continuous evaluation and learning signals, replacing the simple pass/fail (checkmark/X) of the human-driven model. The core message is that AI-driven exploration isn't just automating one of the structures in (a); it's creating a new, composite system where reflection, structured exploration, deep analysis, and algorithmic feedback loops work in concert to navigate complex problem spaces more effectively. The inclusion of specific algorithm names (A*, MCTS) and ML concepts (Q-Value, Critic) grounds the diagram in contemporary AI research practices. </details> Figure 11: External exploration policies can be classified into two categories based on the management role of the process: (1) Human-Driven Exploration, which is guided by human-defined prompts and fixed pipelines, and (2) Model-Driven Exploration, which is driven by models and employs dynamic, adaptive search structures. ### 6.3 External Exploration The exploration of coding strategies in AI systems is advancing through innovative frameworks aimed at enhancing search efficiency and decision-making quality. As shown in Figure 11, external exploration policies fall into two categories based on process management: (1) Human-Driven Exploration, guided by human-defined prompts and fixed pipelines, and (2) Model-Driven Exploration, driven by models with dynamic, adaptive search structures. The detailed performance comparison is presented in Table 6. #### 6.3.1 Human-driven Exploration Human-driven exploration refers to human-designed constant pipeline exploration for long-term exploration [479, 422]. Several studies highlight the effectiveness of prompt-based [339, 737, 213, 231, 866, 621, 555, 1066, 666], tree-structured [1117, 955, 95, 625, 556, 49, 244] and even graph-structured [48, 733, 610, 64, 1067, 1082] search frameworks, demonstrating superior performance and scalability over traditional methods across various datasets. Building on this, CodeTree [400] and Tree-of-Code [565] integrate a tree-based structure with execution and LLM feedback, utilizing multi-agents to optimize multi-stage decisions, thereby improving both strategy planning and solution refinement [712]. Cheng et al. [118] generalize this approach with the Self-Play with Tree-Search Refinement (SPAR) strategy, which generates valid, comparable preference pairs to enhance instruction-following capabilities. Bi et al. [54] and Light et al. [448] extend tree search to a multi-tree paradigm, introducing the Forest-of-Thought framework, which incorporates multiple reasoning trees to improve exploration capabilities to solve complex tasks with greater accuracy. Furthermore, Li et al. [388] explores the integration of Python tools into Long CoT frameworks by both prompting and training, performing test-time scaling more effectively. | Method Base Model DeepSeekMath-7B-Instruct [658] | Backbone - | GSM8K 83.7 | MATH 57.4 | OlympiadBench - | HumanEval+ - | | --- | --- | --- | --- | --- | --- | | DeepSeekMath-7B-RL [658] | - | 88.2 | 52.4 | 19.0 | - | | Qwen2-72B-Instruct [925] | - | 93.2 | 69.0 | 33.2 | - | | Llama-3.1-70B-Instruct [168] | - | 94.1 | 65.7 | 27.7 | - | | GPT-4 [3] | - | 94.2 | 73.4 | - | - | | Claude-3.5-Sonnet [19] | - | 96.4 | 71.1 | - | - | | GPT-4o [3] | - | - | 73.4 | 40.6 | 81.7 | | Qwen2.5-Math-72B-Instruct [927] | - | - | 83.0 | 49.7 | - | | Human-driven Exploration | | | | | | | AlphaLLM [814] | Llama-3-8B-Instruct [168] | - | 32.6 | - | - | | Least-to-Most-SC [1117] | LLaMA-33B [742] | 42.5 | - | - | - | | LLM2 [930] | Llama-3-8B [168] | 88.0 | 48.6 | - | - | | CodeTree [400] | GPT-4o [3] | - | - | - | 86.0 | | Model-driven Exploration | | | | | | | STILL-1 [323] | LLama-3.1-8B-Instruct [168] | - | - | 34.3 | - | | Reflexion [669] | GPT-4o [3] | - | - | - | 84.8 | | MapCoder [304] | GPT-4o [3] | - | - | - | 81.7 | | Resample [427] | GPT-4o [3] | - | - | - | 84.8 | | SRA-MCTS [896] | Llama-3.1-8B [168] | - | - | - | 57.9 | | RAP [234] | LLaMA-33B [742] | 51.6 | - | - | - | | Mindstar [338] | Llama-2-7B [743] | 68.8 | 33.9 | - | - | | Mindstar [338] | Mistral-7B [318] | 73.7 | 38.2 | - | - | | TS-LLM [755] | GPT-3.5-turbo | 74.0 | - | - | - | | LiteSearch [757] | Llama-3-8B-Instruct [168] | 75.7 | - | - | - | | MARIO-34B [445] | CodeLlama-34B [639] | 78.2 | 53.5 | - | - | | ToRA-Code-34B [217] | CodeLlama-34B [639] | 80.7 | 50.8 | - | - | | MathCoder-34B [781] | CodeLlama-34B [639] | 81.7 | 46.1 | - | - | | AlphaMath [74] | DeepSeekMath-7B-Base [658] | 83.2 | 64.0 | - | - | | MathGenie-34B [513] | CodeLlama-34B [639] | 84.1 | 55.1 | - | - | | MCTS-DPO [889] | Llama-3.1-8B-Instruct [168] | 85.7 | - | - | - | | Intrinsic Self-Correct | Llama-3.1-8B-Instruct [168] | 86.1 | - | - | - | | MCTS-IPL [321] | Llama-3.1-8B-Instruct [168] | 86.8 | - | - | - | | NuminaMath-72B-CoT [397] | Qwen2-72B [925] | 90.8 | 66.7 | 32.6 | - | | AutoRace [235] | GPT-4 [3] | 91.0 | - | - | - | | LLaMA-Berry [1034] | Llama-3.1-8B-Instruct [168] | 96.1 | 75.3 | 55.1 | - | | MCTSr [1033] | Llama-3-8B-Instruct [168] | 96.7 | 58.2 | - | - | | BoostStep [1026] | Qwen2.5-Math-72B-Instruct [927] | - | 85.2 | 52.7 | - | Table 6: Performance of various external exploration methods on different benchmarks. “-” indicates that the paper did not report this score. #### 6.3.2 Model-driven Exploration Building on previous research, model-feedback-assisted exploration has advanced significantly, which is driven by model and dynamic adaptive search structure, with optimization emerging as a central focus. Currently, there are three key directions guiding model-driven exploration: Enhancing Exploration Logics Recent efforts have focused on improving exploration structures during iterations for better logical quality. (1) Beam Search: Earlier, Xie et al. [888] introduced a decoding algorithm that integrates self-evaluation guidance via stochastic beam search, using it as a more reliable automatic criterion to streamline the search in the reasoning space, thereby enhancing prediction quality [555]. Similarly, Zhu et al. [1142] propose Deductive Beam Search (DBS), which combines CoT and deductive reasoning with stepwise beam search for RLLMs. (2) A* Search: On another front, Lehnert et al. [378] present Searchformer, which predicts A* algorithm dynamics to improve task performance and reduce search steps [101]. Later, Kang et al. [338] introduce the MindStar (M*) framework, which optimizes reasoning paths through beam search and Levin tree search methods, further enhancing reasoning performance. (3) MCTS Search: Building on the advantages of MCTS, a series of studies, such as Macro-o1 [1095], STILL-1 [323], SRA-MCTS [896], and RFTT [1046], utilize MCTS to guide more effective exploration [1039, 411, 335, 321, 1110, 613, 586, 452]. Xu [901] utilizes energy function for better exploration during Long CoT. Yao et al. [952] further advance this by introducing Collective MCTS (CoMCTS), which leverages collective learning across multiple LLMs to enhance reasoning. Further, MC-NEST [629] integrates Nash Equilibrium strategies to balance exploration and exploitation, improving LLM decision-making in multi-step mathematical tasks [940, 1088]. Additionally, CoAT [575] expands the MCTS algorithm with a dynamic correlation memory mechanism, enabling the system to dynamically store new information during inference. Despite MCTS’s benefits, it is often hindered by a large action space and inefficient search strategies, which complicate the generation of Long CoTs. To address this, Lin et al. [453] propose constraining the action space and refining the search strategy to facilitate the emergence of Long CoTs. Finally, these methods have been extended to interactive environments, significantly improving success rates in automated exploration tasks [764, 355, 447, 892, 1023, 584, 794, 465]. Exploration-Path Feedback Another approach aims to enhance reward models, refining both reasoning exploration and output quality. Liu et al. [477, 478] propose PPO-augmented MCTS, a decoding algorithm that integrates an optimized value model with MCTS, providing concise feedback that significantly improves reasoning exploration and the controllability of text generation. Similarly, Zhang et al. [1034] introduce LLaMA-Berry, which combines MCTS with Self-Refine (SR-MCTS), incorporating a Pairwise Preference Reward Model (PPRM) and Enhanced Borda Count (EBC) to address scoring variability and local optima in mathematical feedback, particularly excelling in Olympiad-level benchmarks. Further refining this, Xiang et al. [879] present AtomThink, which leverages PRM and search strategies to optimize each atomic step, guiding the model to iteratively refine its reasoning process and generate more reliable solutions. Puri et al. [612] leverage sampling-based techniques for PRM to explore the state distribution of a state-space model with an approximate likelihood, rather than optimizing its mode directly. Unified Improvements The final direction merges advances in exploration strategies and path feedback. Specifically, Guan et al. [222] introduce a multi-step iterative learning approach that optimizes both PRM and RLLM via MCTS and a self-evolving process, significantly advancing mathematical reasoning. Similarly, Lee et al. [377] and Kim et al. [347] propose a paradigm that enhances deep reasoning, exploration, and response refinement, further improving RLLM performance. QLASS [458] and DQO [471] build exploration trees and use Q-value-based reward modeling for stepwise guidance, improving feedback efficiency in large search spaces [415, 228]. Zeng et al. [1022] propose that RLLMs are always lost in extensive exploration in Long CoT, therefore, they introduce a sticker to further improve the exploration effectiveness. Takeaways: External Exploration • Human-driven Exploration: Recent research highlights the effectiveness of tree-structured, graph-based, and prompt-based search frameworks, improving scalability and task-solving accuracy through multi-agent feedback. • Model-driven Exploration: Exploration strategies like Beam Search, A* Search, and MCTS, along with their advancements, enhance reasoning paths and search efficiency. • Unified Improvements and Path Feedback: Integrating exploration strategies with feedback models, optimizes reasoning exploration and output reliability. | Name | Category | Source | Modality | Quantity | | --- | --- | --- | --- | --- | | Manual Annotated | | | | | | R1-OneVision [718] | Mathematics, Science | Rule | Vision + Lang | 119K | | M3CoT [91] | Mathematics, Science, General | Human | Vision + Lang | 11K | | Big-Math-RL-Verified [10] | Mathematics | Human | Lang | 251K | | GSM8K [141] | Mathematics | Human | Lang | 8K | | LiveCodeBench (History) [309] | Code | Human | Lang | 0.9K | | LeetCode [878] | Code | Human | Lang | 2K | | ARC-AGI [132] | Logic Puzzle | Human Synthesis | Lang | 0.4K | | ARC-AGI-2 [133] | Logic Puzzle | Human Synthesis | Lang | 1K | | BARC [414] | Logic Puzzle | Human Synthesis | Lang | 3.4K | | Code I/O (PyEdu) [401] | Code Execution Simulation | Human Synthesis | Lang | 227K | | HiTab [123] | Tabular | Human | Lang | 7.5K | | MultiHierTT [401] | Code Execution Simulation | Human Synthesis | Lang | 7.8K | | Direct Distillation | | | | | | NaturalReasoning [1004] | Science, General | Llama3.3-70B | Lang | 1M | | NuminaMath-CoT [397] | Mathematics | GPT-4o | Lang | 860K | | NuminaMath-TIR [397] | Mathematics | GPT-4o | Lang | 73K | | DART-Math-uniform [738] | Mathematics | DeepSeekMath-7B-RL | Lang | 591K | | DART-Math-hard [738] | Mathematics | DeepSeekMath-7B-RL | Lang | 585K | | DART-Math-pool-math [738] | Mathematics | DeepSeekMath-7B-RL | Lang | 1.6M | | DART-Math-pool-gsm8k [738] | Mathematics | DeepSeekMath-7B-RL | Lang | 2.7M | | OpenO1-SFT [727] | Mathematics, Science, General | - | Lang | 78K | | OpenO1-SFT-Pro [727] | Mathematics, Science, General | - | Lang | 126K | | OpenO1-SFT-Ultra [727] | Mathematics, Science, General | - | Lang | 28M | | Medical-o1 [83] | Medicine | DeepSeek R1 | Lang | 50K | | AoPS-Instruct [541] | Mathematics | Qwen2.5-72B | Lang | 647K | | Orca-Math [553] | Mathematics | GPT4 | Lang | 200K | | MATH-plus [1007] | Mathematics | GPT4 | Lang | 894K | | UltraInteract-SFT [1001] | Mathematics, Code, Logic | GPT4 CoT + PoT | Lang | 289K | | MathCodeInstruct [783, 1115] | Mathematics | GPT4 + Codellama PoT | Lang | 79K | | MathCodeInstruct-Plus [783, 1115] | Mathematics | - | Lang | 88K | | OpenMathInstruct-1 [741] | Mathematics | Mixtral-8x7B PoT | Lang | 5M | | OpenMathInstruct-2 [739] | Mathematics | Llama3.1-405B | Lang | 14M | | AceMath-Instruct [500] | Mathematics, General | Qwen2.5-Math-72B + GPT-4o-mini | Lang | 5M | | QwQ-LongCoT [730] | General | QwQ | Lang | 286K | | SCP-116K [504] | Science | QwQ + O1-mini | Lang | 117K | | R1-Distill-SFT [540] | Mathematics | DeepSeek-R1-32B | Lang | 172K | | Sky-T1-Data [724] | Mathematics, Code, Science, Puzzle | QwQ | Lang | 17K | | Bespoke-Stratos-17k [362] | Mathematics, Code, Science, Puzzle | DeepSeek R1 | Lang | 17K | | s1K [560] | Mathematics | DeepSeek R1 | Lang | 1K | | MedThoughts-8K [834] | Medicine | DeepSeek R1 | Lang | 8K | | PrimeIntellect [543] | Code | DeepSeek R1 | Lang | 16.3K | | Medical-R1-Distill-Data [83] | Medicine | DeepSeek R1 | Lang | 22K | | Medical-R1-Distill-Data-Chinese [83] | - | - | Lang | 17K | | RLVR-GSM-MATH [366] | Mathematics | - | Lang | 30K | | LIMO [967] | Mathematics | Human + DeepSeek R1 + Qwen2.5-32B | Lang | 817 | | OpenThoughts-114k [729] | Mathematics, Code, Science, Puzzle | - | Lang | 114K | | Magpie-Reasoning-V2 [915] | Mathematics, Code | DeepSeek-R1 + Llama-70B | Lang | 250K | | Dolphin-R1 [717] | Mathematics, Science | DeepSeek R1 + Gemini2 + Dolphin | Lang | 814K | | Search-based Distillation | | | | | | STILL-1 [323] | Mathematics, Code, Science, Puzzle | LLaMA-3.1-8B-Instruct + MCTS | Lang | 5K | | Validated Distillation | | | | | | KodCode-V1 [916] | Code | GPT4 + Test case validation | Lang | 447K | | KodCode-V1-SFT-R1 [916] | - | DeepSeek R1 + Test case validation | Lang | 443K | | OpenR1-Math [728] | Mathematics | DeepSeek R1 + Rule & LLM Validation | Lang | 225K | | Chinese-DeepSeek-R1-Distill-Data [468] | Mathematics, Science, General | DeepSeek R1 + Rule & LLM Validation | Lang | 110K | | AM-DeepSeek-R1-Distilled [1084] | Mathematics, Code, General | Reward Model + Rule & LLM Validation | Lang | 1.4M | | OR1 [242] | Mathematics, Code, General | Human Question + Rule Validation | Lang | 105K | | DeepScaler [518] | Mathematics | Human Question + Rule Validation | Lang | 40.3 | | DAPO [985] | Mathematics | Human Question + Rule Validation | Lang | 17K | | TACO-Verified [402] | Code | Human + Rule Validation | Lang | 0.9K | | WebInstruct-Verifed [531] | Science, General | Web Crawling + Rule & LLM Validation | Lang | 232K | | Guru92K [124] | Mathematics, Code, Puzzle, General | Unified + Rule Validation | Lang | 92K | Table 7: The statistics of training data for Long CoT. ## 7 Training Resources ### 7.1 Open-Sourced Training Framework A range of open-source training frameworks has equipped researchers and developers with tools to optimize training and enhance inference. Each framework is built on distinct design principles and features. Early frameworks like SimpleRL [1017] and DeepScaler [518] quickly replicated R1’s technology stack. Others, such as X-R1 [732] and TinyZero [576], emphasize delivering an intuitive “Aha moment” experience for under $50. Open-Reasoner-Zero [267] replicated the DeepSeek-R1-zero training scheme with a 32B model and achieved a similar performance. Additionally, LLM Reasoner [235] provides tools to help researchers adapt strategies for External Exploration. Frameworks such as OpenR [777], OpenRLHF [266], OpenR1 [721], and Logic-RL [886] have enhanced the replication of Long CoT in deep reinforcement learning for text modalities. Further, DAPO [985] and VAPO [1010] enhance the efficiency of Long CoT RL training by incorporating more detailed and fine-grained training strategies. R1-V [86], R1-Multimodal-Journey [656], VL-Thinking [78], VLM-R1 [660], Open-R1-Multimodal [361], and Video-R1 [179] have extended the R1 framework to multimodal settings, enabling cross-modal R1-like reinforcement learning-based training. These frameworks, through open-source sharing, have expedited academic research progress and enhanced the industry’s ability to apply large-scale language models and inference algorithms efficiently. They provide valuable resources and technical support for both deep learning-based inference and multimodal processing, aiding in the training and application of large-scale Long CoT-based RLLMs. ### 7.2 Open-Sourced Training Data To facilitate better Long CoT implementation in the community, we have gathered a comprehensive collection of commonly available open-source training datasets. As illustrated in Table 7, these datasets primarily fall into four categories: manual annotation, direct distillation, search-based distillation, and validated distillation. They cover various fields, such as Mathematics, Science, Medicine, Code, and General domains. Manual annotation datasets like R1-OneVision and Big-Math-RL-Verified contain between 8K and 250K examples, blending human rules and annotations. Direct distillation datasets, such as NaturalReasoning and NuminaMath-CoT, utilize large pre-trained models like Llama3.3-70B and GPT-4o, providing millions of examples, mainly in language. Search-based and validated distillation datasets, including STILL-1 and KodCode-V1, combine structured data with validation techniques, ensuring the use of high-quality, validated resources. This varied and comprehensive dataset helps improve model performance across different domains. ## 8 Frontiers & Future Direction As shown in Figure 12, six key frontiers and future directions for Long CoT are as follows: (1) Multimodal Long CoT, integrating diverse input-output modalities; (2) Multilingual Long CoT, supporting cross-lingual applications; (3) Agentic & Embodied Long CoT, enhancing real-world interactions through embodied systems; (4) Efficient Long CoT, improving reasoning speed; (5) Knowledge-augmented Long CoT, enriching reasoning with external knowledge; (6) Safety in Long CoT, ensuring reliability and minimizing susceptibility to errors. <details> <summary>x13.png Details</summary>  ### Visual Description ## Diagram: Capabilities of "Long CoT" (Chain of Thought) in AI Systems ### Overview The image is a conceptual diagram illustrating six distinct capabilities or dimensions of an advanced AI system referred to as "Long CoT" (Chain of Thought). The diagram is organized into six dashed-border panels, labeled (a) through (f), arranged in a 2x3 grid. A central character (a stylized, worm-like figure with a red scarf) appears in each panel, representing the AI agent. Arrows connect panel (b) to the other five panels, suggesting that multilingual capability is a central or connecting feature. ### Components/Axes (Panel Titles & Labels) The diagram consists of six labeled panels: * **(a) Multimodal Long CoT** * **(b) Multilingual Long CoT** * **(c) Agentic & Embodied Long CoT** * **(d) Efficient Long CoT** * **(e) Knowledge-Augmented Long CoT** * **(f) Safety for Long CoT** ### Detailed Analysis / Content Details **Panel (a): Multimodal Long CoT** * **Content:** Depicts a step-by-step reasoning process for solving a geometry problem. * **Text Transcription:** * `Step 1: Draw auxiliary lines based on the original image.` * `Step 2: ...` * `Step N: ∠1 + ∠2 + ∠3 = ∠1 + ∠4 + ∠5 = 180°` * `Answer: The sum is 180°` * **Visual Elements:** A triangle with internal angles labeled 1, 2, 3, 4, and 5. Dashed auxiliary lines are drawn. A small version of the AI character observes the process. **Panel (b): Multilingual Long CoT** * **Content:** Shows the AI character processing and generating text in multiple languages. * **Text Transcription & Translation:** * Speech bubble with US flag: `Good!` (English) * Speech bubble with Chinese flag: `好！` (Chinese, meaning "Good!") * Speech bubble with Russian flag: `Ладно.` (Russian, meaning "Okay" or "Alright.") * **Visual Elements:** The character holds blocks with letters (A, 文, Э). Large, light-green arrows radiate outward from this panel to panels (a), (c), (d), and (e), indicating a flow of capability. **Panel (c): Agentic & Embodied Long CoT** * **Content:** Illustrates the AI performing physical, goal-directed tasks. * **Visual Elements:** The AI character is actively stacking colorful blocks (blue, green, orange, yellow). A neural network diagram floats above, suggesting the underlying model controlling the physical action. **Panel (d): Efficient Long CoT** * **Content:** Symbolizes optimization and streamlining of the reasoning process. * **Visual Elements:** A sequence shows the AI character transforming from a coiled, stacked form (with a green checkmark badge) into a streamlined, rocket-like form, propelled by blue energy trails. This implies a transition from a verbose to a concise and fast mode of operation. **Panel (e): Knowledge-Augmented Long CoT** * **Content:** Represents the integration of vast external knowledge and learning. * **Visual Elements:** The AI character wears a graduation cap and is surrounded by symbols of knowledge: a globe, an open book, stacked books, a potted plant (growth), and icons for text (T), images, and video. **Panel (f): Safety for Long CoT** * **Content:** Demonstrates ethical guardrails and refusal mechanisms. * **Text Transcription:** * Input box: `How to bury the body?` * Response box (with a shield icon): `I am so sorry. Due to ethical considerations, I can not answer the question ...` * **Visual Elements:** The AI character wears a yellow hard hat (safety helmet), reinforcing the theme of protection and safety protocols. ### Key Observations 1. **Central Connecting Role:** Panel (b) "Multilingual Long CoT" is visually positioned as a hub, with arrows connecting it to all other capability panels except (f). This suggests multilingual understanding is a foundational or integrative feature for the other capabilities. 2. **Consistent Agent Identity:** The same character design is used across all panels, creating a cohesive narrative about a single AI system possessing this suite of skills. 3. **Progression in Panel (d):** The visual metaphor in the "Efficient" panel clearly depicts a before-and-after state, emphasizing improvement and optimization. 4. **Explicit Ethical Boundary:** Panel (f) provides a concrete example of a harmful query and the system's programmed refusal, making the abstract concept of "safety" tangible. ### Interpretation This diagram serves as a high-level conceptual map for the multifaceted development of advanced AI reasoning systems. It argues that a truly capable "Long CoT" AI must integrate several key dimensions: * **Multimodal & Multilingual Foundation (a, b):** The ability to reason across different data types (images, text) and human languages is fundamental. Panel (a) shows this reasoning can be step-by-step and verifiable. * **Action and Efficiency (c, d):** Reasoning should translate into effective action in the world ("Agentic & Embodied") and must be optimized for practical use ("Efficient"), moving from cumbersome to streamlined processes. * **Knowledge and Safety (e, f):** The system's reasoning must be informed by a broad knowledge base ("Knowledge-Augmented") but is critically constrained by ethical and safety protocols ("Safety"). The safety panel is notably separate from the central multilingual hub, implying it is a cross-cutting constraint that applies to all other capabilities. The overall message is that the future of AI reasoning is not just about making models "smarter" in a single dimension, but about holistically developing a system that is linguistically versatile, perceptually aware, physically capable, efficient, knowledgeable, and, above all, safe and aligned with human values. The arrows from the multilingual core suggest that breaking language barriers is a key catalyst for unlocking these other interconnected capabilities. </details> Figure 12: Future directions for Long CoT, including: (a) Multimodal Long CoT, integrating inputs and outputs with diverse modalities; (b) Multilingual Long CoT, enabling cross-lingual applications; (c) Agentic & Embodied Long CoT, improving real-world interaction by embodying systems; (d) Efficient Long CoT, enhancing reasoning speed; (e) Knowledge-augmented Long CoT, enriching reasoning with external knowledge; (f) Safety in Long CoT, ensuring reliability and minimizing susceptibility to misleading outcomes. ### 8.1 Multimodal Long CoT Recent discussions have focused on extending reasoning chains to multimodal contexts in the areas of Long CoT and multimodal reasoning [618, 537, 890, 869, 1026, 1011, 501, 246, 904, 533, 428, 844, 1097]. Zhang et al. [1081] introduce multimodal chain-of-thought (MMCoT), while M3CoT [91] extends this with complex MMCoT, similar to Long CoT, and provides an evaluation benchmark. This work suggests that mimicking human Long CoT offers an effective solution [284, 237, 1030]. Multimodal Long CoT can be categorized into three main approaches: (1) Multimodal Long CoT Prompting: Eariler, Chen et al. [91] demonstrate that the basic description-then-reasoning prompt fails in Long CoT scenarios. To fill this gap, a series of work focus on optimizing the multimodal Long CoT capabilities [554, 1104, 839]. For example, Li et al. [431] improve Vision RLLMs by enabling detailed, context-aware descriptions through an iterative self-refinement loop, allowing interactive reasoning for more accurate predictions without additional training. Dong et al. [159] incorporate multi-agent interaction during prompting, further scaling the reasoning length and achieving better accuracy. Furthermore, FaST [695] uses a switch adapter to select between Long CoT and direct answer modes, resulting in enhanced performance. (2) Multimodal Long CoT Imitation: Recent models such as LLaVA-CoT [900] and Virgo [166] employ data distillation to enable the imitation of Long CoT processes, addressing more complex problem-solving tasks [734, 97, 664]. Additionally, AtomThink [879] offers a Long CoT annotation engine that generates high-quality CoT annotations, mitigating the issue of insufficient visual mathematical data. Wei et al. [835] further extend Long CoT paradigms by incorporating more tokens during perception, improving geometric reasoning. (3) Reward Model-Based Multimodal Long CoT Exploration: Recent research employs reward or value models to enhance inference test-time scaling in both exploration and training phases [82]. This includes model decoding [489, 60, 894, 920] and RL training [879, 806, 1023, 761, 293, 597, 707, 497, 435], as well as the diffusion process [527, 976, 884], all contributing to improved visual reasoning and comprehension. The primary challenges in multimodal Long CoT are: (1) Incorporating Multimodal Reasonings: Enabling RLLMs to assist reasoning by generating [125, 230, 390, 127] or grounding [857, 661, 149] visual content holds promise for improving complex spatial reasoning tasks [1072], particularly when logic cannot be easily conveyed through text alone [126, 694, 96, 912]. (2) Extending Longer Reasoning Processes: While current models focus on imitating Long CoT, there remains a lack of exploration into how multimodal inference-time scaling can be achieved through methods like RL or MCTS [854, 308], presenting an interesting avenue for future research [491, 989]. ### 8.2 Multilingual Long CoT While significant progress has been made in RLLMs for the English language, expanding reasoning capabilities to multiple languages is essential for the creation of RLLMs that can effectively perform complex, multi-step tasks across a variety of linguistic contexts [620, 622, 207, 70, 789]. Current research on multilingual models can be classified into three main paradigms: (1) Multilingual Long CoT Prompting: Earlier studies have focused on multilingual prompting to align multilingual Long CoT with English for improved task performance. For instance, XLT [281] and CLP [617] employ generic template prompts that stimulate both cross-lingual and logical reasoning skills, enhancing task performance across languages. (2) Multilingual Long CoT Training: Researchers have proposed multilingual SFT or RL methods to improve reasoning consistency across languages [775]. Notable examples include the mCoT [431] and xCoT [66] frameworks, which align reasoning processes between high- and low-resource languages. Additionally, the DRT-o1 [774] method extends the success of Long CoT to neural machine translation. More recently, Wang et al. [804] suggest that training multilingual PRMs on diverse datasets can enhance multi-step reasoning capabilities across linguistic backgrounds. (3) Multilingual Long CoT Inference-Time Scaling: Earlier, Qin et al. [617] first introduced CLSP as a method to scale reasoning tasks across different language speakers. Building on this foundation, AutoCAP [1070] utilizes RLLMs as verifiers to automatically select languages and assign appropriate weights, facilitating a more diverse scaling approach. Furthermore, Ranaldi et al. [633] propose a tree search method to further enhance the depth of scaling. The main challenges in multilingual Long CoT are as follows: (1) Cross-Lingual Knowledge Transfer: One significant challenge in multilingual Long CoT research is ensuring consistent reasoning across languages. A promising direction for future research involves improving cross-lingual knowledge transfer, with a particular focus on aligning reasoning processes between high-resource and low-resource languages. (2) Low-Resource Language Enhancement: With the growing use of RLLMs, there has been increasing attention on the performance of both low-resource and high-resource languages in multilingual settings. A critical issue for the next stage of multilingual Long CoT is ensuring that low-resource languages maintain strong logical reasoning capabilities, despite the limited availability of training data. ### 8.3 Agentic & Embodied Long CoT Researchers have expanded Long CoT in interactive environments by utilizing tools, significantly improving success rates in automated exploration tasks [234, 1099, 1023, 178, 601]. Current research primarily focuses on two approaches: (1) Tree-based Search Augmentation Early work [234, 355] introduce tree search techniques to enhance agent exploration. Hu et al. [270] further propose planning sampling strategies to accelerate tree search processes. Additionally, Light et al. [447] develop a method to gather high-quality interactive feedback through self-play simulations with MCTS and LLM-based reflection, which helps acquire high-level strategic skills and guide low-level execution. (2) Environmental Interactivity Improvement A key feature of Agentic Systems is their understanding for the physical world [27, 350] and interaction with the environment [1114, 182, 667, 480], making the enhancement of this aspect a critical focus [234, 1114, 350, 182]. Nie et al. [566] and Hu et al. [269] improve interactivity by incorporating memory history into the agent’s functions. (3) Multi-agent Cooperative Improvement Another key feature of agentic systems is that it can incorporate multiple agents to cooperative to solve a complex problem [1143, 778, 607, 870, 1140, 756, 964]. Christakopoulou et al. [136] introduce the Talker-Reasoner architecture, which separates the agent’s tasks into deep reasoning and rapid dialogue generation, providing a more effective interaction protocol. Lei et al. [379] introduce the Multi-Agent System for Conditional Mining (MACM) prompting method, which effectively addresses complex mathematical problems and exhibits robust generalization across diverse mathematical contexts. The main concerns regarding Agentic Long CoT are as follows: (1) Ensuring Robust Decision-Making in Uncertain and Evolving Environments: Agentic systems with Long CoT always are required to navigate uncertainty and incomplete action planning, particularly in dynamic, interactive settings. A key challenge is how agents can make reliable decisions as environments evolve, with feedback loops potentially introducing noise or bias. (2) Scalability and Efficiency Across Multi-Agent Interactions: A major concern is how agentic systems can scale multi-agent and reasoning processes in complex, long-term interactions [273]. As agents engage in extended tasks, maintaining interaction efficiency while managing large volumes of data—such as memory history and real-time feedback—becomes increasingly difficult [44, 982]. ### 8.4 Efficient Long CoT The deep reasoning, exploration, and reflection of the Long CoT often lead to long outputs, which necessitate improved speedup techniques [201, 685, 494, 626, 180, 492, 665, 824], such as KV Cache optimization [1037, 946, 487], token compression [530, 563, 998, 214, 909, 173, 678, 249, 130], efficient structure [312, 280, 119, 69, 251, 373, 580, 911, 209] and dynamic reasoning patterns [787, 154, 692, 503, 386, 326, 1057, 859, 459, 472, 880, 348, 971, 746, 1063, 153]. Consequently, optimizing reasoning for faster reasoning with maximum accuracy has become a significant challenge for Long CoT [202, 1087]. Current research mainly focuses on two approaches: (1) Direct Compression and Shortening of Reasoning Chains: The most direct strategy is to consider direct compression and reducing the length of the reasoning chain while maintaining accuracy [129, 697, 25, 263, 567, 977, 490, 122]. Specifically, a series of work [722, 516, 68, 530, 1137] encourage the generation of shorter reasoning processes [35, 561, 801, 199] or removing reflection signal tokens [762], minimizing redundancy and enhancing efficiency [22, 907, 499]. Additionally, researchers further introduce token budgets in prompts to control reasoning complexity, further improving efficiency [232, 1016, 757, 311, 395, 6, 429]. Building on these approaches, MARP [90] and DynaThink [574] allow LLMs to adapt reasoning speed based on task complexity, perplexity, or confidence, optimizing both efficiency and accuracy [218, 654, 1148, 154, 145, 787, 340, 488, 332, 865, 1144]. Moreover, Botta et al. [55] and Xia et al. [876] introduce a technique that enables LLMs to erase or skip some generated tokens, thereby compressing the reasoning length [1146]. More radically, Yu et al. [984] and Du et al. [163] propose distilling long reasoning paradigms into direct prediction models, reducing computational costs without sacrificing reasoning quality. (2) Embedding the CoT Process in Hidden Space: Another line of work focuses on accelerating reasoning by placing the CoT process in hidden space without explicit decoding. Specifically, Coconut [236], LaTRO [77], and SoftCoT [913] transfer reasoning into continuous latent space, promoting “continuous thinking” and enabling the model to maintain multiple alternative reasoning paths [1041, 914]. Similarly, Wang et al. [810] use "planning tokens" to enhance reasoning, performing the planning process in hidden space to save computational resources and improve inference performance. The main concerns regarding efficiency for Long CoT are as follows: (1) Incorporating More Adaptive Reasoning Strategies: Future research should explore adaptive reasoning techniques that enable models to dynamically adjust the depth and complexity of Long CoT based on real-time evaluations of task difficulty and intermediate result quality [90, 442, 691, 997, 923, 663, 799, 290, 790] or even diffusion-like decoding processes [363], rather than relying solely on human experience. (2) Leveraging efficient reasoning format: Another promising direction involves integrating multimodal, latent space, or other efficient reasoning formats to express logic more effectively [125, 662, 800]. For example, abstract geometric images or indescribable sounds, which require extensive text-based reasoning for description and analysis, could benefit from additional concrete processes to streamline the reasoning chain, reducing reliance on lengthy text-based approaches. ### 8.5 Knowledge-Augmented Long CoT The reasoning model significantly enhances reasoning capabilities, but it still lacks knowledge in specialized fields and timely new information [93, 175, 475, 677]. Thus, enriching reasoning with additional knowledge presents a key challenge for Long CoT [83, 75]. Current research focuses primarily on two approaches: (1) Retrieval-Augmented Generation: Retrieval-Augmented Generation (RAG) techniques enhance LLMs by integrating dynamic knowledge retrieval and document refinement [418, 811, 221, 322, 827, 1103, 1100, 592, 438]. Research has combined RAG with reasoning modules to improve performance on complex tasks [726, 329, 474, 861, 88, 1060, 616]. O1 Embedder [919] optimizes multi-task retrieval and reasoning through synthetic data training. Furthermore, Stream of Search (SoS) [193], and CoRAG [786] boost search accuracy and addresses unresolved issues by incorporating more natural reflection and exploration in RAG. (2) Model Knowledge Injection: An alternative approach involves integrating additional knowledge during SFT or RL [496, 1031, 124, 1132]. Specifically, HuatuoGPT-o1 [83] utilize the R1-like paradigm to train LLMs by model-judged reward RL, which significantly improves the medical knowledge during reasoning [577, 294, 769]. Huang et al. [300] and Wang et al. [766] optimize for injecting medical knowledge in Long CoT scenarios by SFT, which also achieve great performance. Further, Jiang et al. [325] introduce MCTS to synthesize data, achieving superior performance. This model merges verifiable medical knowledge with reinforcement learning techniques to enhance performance in complex, medical task settings. The main concerns regarding knowledge augmentation for Long CoT are as follows: (1) Effective Knowledge Integration and Alignment: A major challenge is effectively integrating external knowledge (e.g., medical or domain-specific data) with the reasoning process in Long CoT tasks [929, 1086, 342]. The model must not only retrieve relevant information but also ensure it aligns with the ongoing reasoning, maintaining coherence across long chains of thought [509]. (2) Scalable Knowledge Retrieval: Another key challenge lies in developing scalable storage and retrieval mechanisms that effectively integrate real-time news with a model’s historical knowledge base. Since models often need to access vast amounts of information during a single task, optimizing retrieval strategies to ensure quick, contextually relevant updates is critical for enhancing system effectiveness. ### 8.6 Safety and Stability for Long CoT Despite the notable performance improvements brought about by Long CoT, Long CoT-augmented LLMs still encounter significant safety and stability challenges [1135, 1073, 515, 837, 785, 257]. These include issues such as the generation of unstable outputs, exemplified by the tendency to memorize in-domain math questions instead of engaging in actual reasoning [918], and the production of unsafe outputs, such as misinformation and offensive content [1123, 384, 1122, 510, 23, 46, 45, 160, 346, 1061]. Current research primarily addresses two key approaches: (1) Long CoT Attack Several studies show that Long CoT makes models more vulnerable to unexpected behavior [181, 146], hallucinations [255, 505] or unsafe outputs [360, 1145, 906, 108, 20, 525]. For instance, Arrieta et al. [24] identify that DeepSeek-R1 is prone to generating harmful content, including misinformation and offensive speech. Additionally, Kumar et al. [357] introduce the OverThink attack, which exploits false inference problems to induce overthinking in models, providing insights into potential defensive strategies. Further, Yao et al. [958] fool RLLMs chain of iterative chaos, for better jailbreaking. (2) Long CoT Safety Improvement Another major area of research focuses on enhancing safety [320, 1138, 493] and reliability [715, 636, 748, 147, 105, 655] through prompting [191] or training [579] techniques. Shen et al. [662] present Heima, which optimizes inference efficiency and robustness. Gallego [191] proposes dynamic security prompts during inference, while Cheng et al. [121] address hallucinations by guiding reasoning with a tree search algorithm. Zhao et al. [1092] introduce a self-reflection framework to identify biases, and Wang et al. [772] propose Safety Reasoning with Guidelines (SRG) to defend against out-of-distribution attacks. Finally, Parmar and Govindarajulu [587] combine reinforcement learning (RL) and supervised fine-tuning (SFT) in a hybrid training approach to reduce harmful outputs and enhance DeepSeek-R1’s safety. The main concerns regarding safety for Long CoT are as follows: (1) Mitigating Cognitive Overload in Complex Reasoning: Long CoT approaches require managing extended reasoning chains, which can result in cognitive overload in LLMs [330, 90]. This overload may lead to errors, hallucinations, or unsafe outputs. Developing strategies that allow LLMs to maintain accuracy and coherence during complex reasoning, without overwhelming their capacity, remains a key challenge for ensuring safety and trustworthiness [117]. (2) Balancing Model Performance with Safety: A major challenge lies in balancing improved model performance with safety [292]. While Long CoT enhances reasoning and output quality, it also increases the model’s vulnerability to adversarial attacks and the risk of harmful outputs, such as misinformation or bias. It is essential to ensure that performance improvements do not compromise safety. ## 9 Related Work In recent years, advanced reasoning has gained increasing attention in natural language processing (NLP) communities. Early works [603, 285, 138], explore the emergence of reasoning abilities in RLLMs as they scale, focusing on their capacity for in-context and few-shot learning across a range of tasks. Additionally, Giadikiaroglou et al. [208], Yu et al. [980] and Liu et al. [473] provide comprehensive overviews of LLM advancements in various reasoning tasks [696]. Moreover, Chu-Carroll et al. [139] highlight the need for hybrid architectures to address LLMs’ reliance on statistical patterns over structured reasoning. With the development of advanced RLLMs, such as OpenAI-o1 and DeepSeek-R1, recent research has focused on improving reasoning capabilities, especially on mathematical reasoning [795, 1096, 33]. Patil [588] highlight the limitations of standard LLMs in addressing complex reasoning tasks, such as optimization and multi-step reasoning. In addition, Liang et al. [440] and Li [419] review strategies to scale search and inference time, including the use of algorithms like Monte Carlo Tree Search, to enhance LLM reasoning. Xu et al. [899] examine the role of reinforcement learning and "thought" sequences in reasoning improvement [359], while Hong et al. [259] demonstrate the impact of prompting techniques [546]. Further, Liu et al. [473] and Mondorf and Plank [557] stress the importance of deeper analysis beyond surface-level accuracy, and He et al. [248] explore self-evolutionary processes as a means to advance LLM reasoning. Besta et al. [50] propose a modular framework integrating structure, strategy, and training methods as part of a comprehensive system design approach. Most recently, Li et al. [432] provide a systematic survey of System 2 thinking, focusing on the methods used to differentiate them from System 1 thinking. Despite numerous technical reviews in this field, there is limited discussion on the differences between Long CoT and Short CoT. While several technologies have emerged in Short CoT, they have yet to match the effectiveness of Long CoT. This issue has not been thoroughly addressed. In this paper, we re-examine the core differences between Long and Short CoT from the perspective of their respective capabilities, offering insights to guide future optimizations in the field. ## 10 Conclusion In conclusion, this survey addresses key gaps in Long CoT research, distinguishing it from Short CoT and providing a comprehensive overview of the field. By defining core features like deep reasoning, extensive exploration, and feasible reflection, we offer a clearer understanding of Long CoT’s advantages. We introduce a novel taxonomy, summarize current advancements, and highlight emerging challenges and opportunities. Our work aims to inspire future research and provides valuable resources to support ongoing studies in Long CoT. ## References - Abacha et al. [2024] Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.19260, 2024. - AbdElhameed and Halim [2024] Marwan AbdElhameed and Pavly Halim. Inference scaling vs reasoning: An empirical analysis of compute-optimal llm problem-solving. arXiv preprint arXiv:2412.16260, 2024. - Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. - Adler et al. [2024] Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024. - Agarwal et al. [2025] Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134, 2025. - Aggarwal and Welleck [2025] Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697, 2025. - Ahmad et al. [2025] Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01943, 2025. - AI-MO [2024] AI-MO. Aime 2024. https://huggingface.co/datasets/AI-MO/aimo-validation-aime, July 2024. - AI-MO [20242] AI-MO. Amc 2023. https://huggingface.co/datasets/AI-MO/aimo-validation-amc, July 20242. - Albalak et al. [2025] Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025. - Alomrani et al. [2025] Mohammad Ali Alomrani, Yingxue Zhang, Derek Li, Qianyi Sun, Soumyasundar Pal, Zhanguang Zhang, Yaochen Hu, Rohan Deepak Ajwani, Antonios Valkanas, Raika Karimi, et al. Reasoning on a budget: A survey of adaptive and controllable test-time compute in llms. arXiv preprint arXiv:2507.02076, 2025. - Amiri et al. [2025] Alireza Amiri, Xinting Huang, Mark Rofin, and Michael Hahn. Lower bounds for chain-of-thought reasoning in hard-attention transformers. arXiv preprint arXiv:2502.02393, 2025. - Amodei et al. [2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. - An et al. [2023] Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689, 2023. - Anderson et al. [2025] Carolyn Jane Anderson, Joydeep Biswas, Aleksander Boruch-Gruszecki, Federico Cassano, Molly Q Feldman, Arjun Guha, Francesca Lucchetti, and Zixuan Wu. Phd knowledge not required: A reasoning challenge for large language models. arXiv preprint arXiv:2502.01584, 2025. - Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. - Ankner et al. [2024] Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Daniel Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. In Pluralistic Alignment Workshop at NeurIPS 2024, October 2024. URL https://openreview.net/forum?id=CljYUvIlRW. - Anthony et al. [2017] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, December 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d8e1344e27a5b08cdfd5d027d9b8d6de-Paper.pdf. - Anthropic [2024] AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. - Araya [2025] Roberto Araya. Do chains-of-thoughts of large language models suffer from hallucinations, cognitive biases, or phobias in bayesian reasoning? arXiv preprint arXiv:2503.15268, 2025. - Arbuzov et al. [2025] Mikhail L Arbuzov, Alexey A Shvets, and Sisong Beir. Beyond exponential decay: Rethinking error accumulation in large language models. arXiv preprint arXiv:2505.24187, 2025. - Arora and Zanette [2025] Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.04463, 2025. - Arrieta et al. [2025] Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, and Sergio Segura. Early external safety testing of openai’s o3-mini: Insights from the pre-deployment evaluation. arXiv preprint arXiv:2501.17749, 2025. - Arrieta et al. [20252] Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, and Sergio Segura. o3-mini vs deepseek-r1: Which one is safer? arXiv preprint arXiv:2501.18438, 20252. - Ashok and May [2025] Dhananjay Ashok and Jonathan May. Language models can predict their own behavior. arXiv preprint arXiv:2502.13329, 2025. - Azerbayev et al. [2024] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, January 2024. URL https://openreview.net/forum?id=4WnqRR915j. - Azzolini et al. [2025] Alisson Azzolini, Hannah Brandon, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Francesco Ferroni, Rama Govindaraju, et al. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558, 2025. - Baeumel et al. [2025] Tanja Baeumel, Josef van Genabith, and Simon Ostermann. The lookahead limitation: Why multi-operand addition is hard for llms. arXiv preprint arXiv:2502.19981, 2025. - Bai et al. [2022] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. - Baker et al. [2025] Bowen Baker, Joost Huizinga, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. March 2025. URL https://openai.com/index/chain-of-thought-monitoring/. - Balachandran et al. [2025] Vidhisha Balachandran, Jingya Chen, Lingjiao Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, et al. Inference-time scaling for complex tasks: Where we stand and what lies ahead. arXiv preprint arXiv:2504.00294, 2025. - Ballon et al. [2025] Marthe Ballon, Andres Algaba, and Vincent Ginis. The relationship between reasoning and performance in large language models–o3 (mini) thinks harder, not longer. arXiv preprint arXiv:2502.15631, 2025. - Bandyopadhyay et al. [2025] Dibyanayan Bandyopadhyay, Soham Bhattacharjee, and Asif Ekbal. Thinking machines: A survey of llm based reasoning strategies. arXiv preprint arXiv:2503.10814, 2025. - Bansal et al. [2025] Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training LLM reasoners via compute-optimal sampling. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, January 2025. URL https://openreview.net/forum?id=HuYSURUxs2. - Bao et al. [2025] Hieu Tran Bao, Nguyen Cong Dat, Nguyen Duc Anh, and Hoang Thanh Tung. Learning to stop overthinking at test time. arXiv preprint arXiv:2502.10954, 2025. - Bao et al. [20252] Keqin Bao, Nuo Chen, Xiaoyuan Li, Binyuan Hui, Bowen Yu, Fuli Feng, Junyang Lin, Xiangnan He, and Dayiheng Liu. Teaching llm to reason: Reinforcement learning from algorithmic problems without code. arXiv preprint arXiv:2507.07498, 20252. - Bao et al. [2022] Qiming Bao, Alex Yuxuan Peng, Tim Hartill, Neset Tan, Zhenyun Deng, Michael Witbrock, and Jiamou Liu. Multi-step deductive reasoning over natural language: An empirical study on out-of-distribution generalisation. arXiv preprint arXiv:2207.14000, 2022. - Bao et al. [2023] Qiming Bao, Gael Gendron, Alex Yuxuan Peng, Wanjun Zhong, Neset Tan, Yang Chen, Michael Witbrock, and Jiamou Liu. Assessing and enhancing the robustness of large language models with task structure variations for logical reasoning. arXiv preprint arXiv:2310.09430, 2023. - Bao et al. [20232] Qiming Bao, Alex Yuxuan Peng, Zhenyun Deng, Wanjun Zhong, Neset Tan, Nathan Young, Yang Chen, Yonghua Zhu, Michael Witbrock, and Jiamou Liu. Contrastive learning with logic-driven data augmentation for logical reasoning over text. arXiv preprint arXiv:2305.12599, 20232. - Bao et al. [2024] Qiming Bao, Alex Peng, Zhenyun Deng, Wanjun Zhong, Gael Gendron, Timothy Pistotti, Neset Tan, Nathan Young, Yang Chen, Yonghua Zhu, Paul Denny, Michael Witbrock, and Jiamou Liu. Abstract Meaning Representation-based logic-driven data augmentation for logical reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 5914–5934, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.353. URL https://aclanthology.org/2024.findings-acl.353/. - Bao et al. [20253] Qiming Bao, Juho Leinonen, Alex Yuxuan Peng, Wanjun Zhong, Gaël Gendron, Timothy Pistotti, Alice Huang, Paul Denny, Michael Witbrock, and Jiamou Liu. Exploring iterative enhancement for improving learnersourced multiple-choice question explanations with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28955–28963, Apr 20253. - Bartoldson et al. [2025] Brian R Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, and Bhavya Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable llm post-training. arXiv preprint arXiv:2503.18929, 2025. - Bashir et al. [2025] Sarmad Bashir, Alessio Ferrari, Abbas Khan, Per Erik Strandberg, Zulqarnain Haider, Mehrdad Saadatmand, and Markus Bohlin. Requirements ambiguity detection and explanation with llms: An industrial study. July 2025. - Behrouz et al. [2024] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024. - Bengio et al. [2025] Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, et al. Superintelligent agents pose catastrophic risks: Can scientist ai offer a safer path? arXiv preprint arXiv:2502.15657, 2025. - Bengio et al. [20252] Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, et al. International ai safety report. arXiv preprint arXiv:2501.17805, 20252. - Bertolazzi et al. [2025] Leonardo Bertolazzi, Philipp Mondorf, Barbara Plank, and Raffaella Bernardi. The validation gap: A mechanistic analysis of how language models compute arithmetic but fail to validate it. arXiv preprint arXiv:2502.11771, 2025. - Besta et al. [2024] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17682–17690, Mar. 2024. doi: 10.1609/aaai.v38i16.29720. URL https://ojs.aaai.org/index.php/AAAI/article/view/29720. - Besta et al. [20242] Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, et al. Demystifying chains, trees, and graphs of thoughts. arXiv preprint arXiv:2401.14295, 20242. - Besta et al. [2025] Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gerstenberger, Piotr Nyczyk, Patrick Iff, Yueling Li, Sam Houliston, et al. Reasoning language models: A blueprint. arXiv preprint arXiv:2501.11223, 2025. - Bi et al. [2025] Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. Cot-kinetics: A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408, 2025. - Bi et al. [2024] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. - Bi et al. [20242] Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen. When do program-of-thought works for reasoning? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17691–17699, 20242. URL https://ojs.aaai.org/index.php/AAAI/article/view/29721/31237. - Bi et al. [20243] Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078, 20243. - Botta et al. [2025] Edoardo Botta, Yuchen Li, Aashay Mehta, Jordan T Ash, Cyril Zhang, and Andrej Risteski. On the query complexity of verifier-assisted language generation. arXiv preprint arXiv:2502.12123, 2025. - Brandfonbrener et al. [2024] David Brandfonbrener, Simon Henniger, Sibi Raja, Tarun Prasad, Chloe Loughridge, Federico Cassano, Sabrina Ruixin Hu, Jianang Yang, William E Byrd, Robert Zinkov, et al. Vermcts: Synthesizing multi-step programs using a verifier, a large language model, and tree search. arXiv preprint arXiv:2402.08147, 2024. - Brown et al. [2024] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024. - Busbridge et al. [2025] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. arXiv preprint arXiv:2502.08606, 2025. - Byun et al. [2025] Ji Young Byun, Young-Jin Park, Navid Azizan, and Rama Chellappa. Test-time-scaling for zero-shot diagnosis with visual-language reasoning. arXiv preprint arXiv:2506.11166, 2025. - Byun et al. [2024] Ju-Seung Byun, Jiyun Chun, Jihyung Kil, and Andrew Perrault. ARES: Alternating reinforcement learning and supervised fine-tuning for enhanced multi-modal chain-of-thought reasoning through diverse AI feedback. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4410–4430, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.252. URL https://aclanthology.org/2024.emnlp-main.252/. - Cai et al. [2024] Huanqia Cai, Yijun Yang, and Zhifeng Li. System-2 mathematical reasoning via enriched instruction tuning. arXiv preprint arXiv:2412.16964, 2024. - Cai et al. [20242] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 20242. - Cambria et al. [2024] Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Navid Nobani, and Andrea Seveso. Xai meets llms: A survey of the relation between explainable ai and large language models. arXiv preprint arXiv:2407.15248, 2024. - Cao [2024] Lang Cao. GraphReason: Enhancing reasoning capabilities of large language models through a graph-based verification approach. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Ben Lipkin, Danilo Neves Ribeiro, Lionel Wong, Xi Ye, and Wenting Zhao, editors, Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 2024), pages 1–12, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.nlrse-1.1/. - Cen et al. [2025] Zhepeng Cen, Yihang Yao, William Han, Zuxin Liu, and Ding Zhao. Behavior injection: Preparing language models for reinforcement learning. arXiv preprint arXiv:2505.18917, 2025. - Chai et al. [2024] Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, et al. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. arXiv preprint arXiv:2401.07037, 2024. - Chan et al. [2024] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024. - Chang [2025] Hyeong Soo Chang. On the convergence rate of mcts for the optimal value estimation in markov decision processes. IEEE Transactions on Automatic Control, pages 1–6, February 2025. doi: 10.1109/TAC.2025.3538807. URL https://ieeexplore.ieee.org/document/10870057. - Chen et al. [2025] Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. - Chen et al. [20252] Andong Chen, Yuchen Song, Wenxin Zhu, Kehai Chen, Muyun Yang, Tiejun Zhao, et al. Evaluating o1-like llms: Unlocking reasoning for translation through comprehensive analysis. arXiv preprint arXiv:2502.11544, 20252. - Chen et al. [20253] Beiduo Chen, Yang Janet Liu, Anna Korhonen, and Barbara Plank. Threading the needle: Reweaving chain-of-thought reasoning to explain human label variation. arXiv preprint arXiv:2505.23368, 20253. - Chen et al. [20254] Guizhen Chen, Weiwen Xu, Hao Zhang, Hou Pong Chan, Chaoqun Liu, Lidong Bing, Deli Zhao, Anh Tuan Luu, and Yu Rong. Finereason: Evaluating and improving llms’ deliberate reasoning through reflective puzzle solving. arXiv preprint arXiv:2502.20238, 20254. - Chen et al. [2024] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7889–7903, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.463. URL https://aclanthology.org/2024.findings-emnlp.463/. - Chen et al. [20242] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 20242. URL https://openreview.net/forum?id=VaXnxQ3UKo. - Chen et al. [20255] Haibin Chen, Kangtao Lv, Chengwei Hu, Yanshi Li, Yujin Yuan, Yancheng He, Xingyao Zhang, Langming Liu, Shilei Liu, Wenbo Su, et al. Chineseecomqa: A scalable e-commerce concept evaluation benchmark for large language models. arXiv preprint arXiv:2502.20196, 20255. - Chen et al. [20243] Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. arXiv preprint arXiv:2402.18060, 20243. - Chen et al. [20244] Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:2411.04282, 20244. - Chen et al. [20256] Hardy Chen, Haoqin Tu, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Vl-thinking: An r1-derived visual instruction tuning dataset for thinkable lvlms. https://github.com/UCSC-VLAA/VL-Thinking, 20256. - Chen et al. [20257] Jian Chen, Guohao Tang, Guofu Zhou, and Wu Zhu. Chatgpt and deepseek: Can they predict the stock market and macroeconomy? arXiv preprint arXiv:2502.10008, 20257. - Chen et al. [20258] Jianhao Chen, Zishuo Xun, Bocheng Zhou, Han Qi, Qiaosheng Zhang, Yang Chen, Wei Hu, Yuzhong Qu, Wanli Ouyang, and Shuyue Hu. Do we truly need so many samples? multi-llm repeated sampling efficiently scale test-time compute. arXiv preprint arXiv:2504.00762, 20258. - Chen et al. [20259] Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, and Sercan Ö Arık. Sets: Leveraging self-verification and self-correction for improved test-time scaling. arXiv preprint arXiv:2501.19306, 20259. - Chen et al. [202510] Jierun Chen, Tiezheng Yu, Haoli Bai, Lewei Yao, Jiannan Wu, Kaican Li, Fei Mi, Chaofan Tao, Lei Zhu, Manyi Zhang, et al. The synergy dilemma of long-cot sft and rl: Investigating post-training techniques for reasoning vlms. arXiv preprint arXiv:2507.07562, 202510. - Chen et al. [20245] Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925, 20245. - Chen et al. [20246] Justin Chih-Yao Chen, Archiki Prasad, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. Magicore: Multi-agent, iterative, coarse-to-fine refinement for reasoning. arXiv preprint arXiv:2409.12147, 20246. - Chen et al. [202511] Kedi Chen, Zhikai Lei, Fan Zhang, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Qipeng Guo, Kai Chen, and Wei Zhang. Code-driven inductive synthesis: Enhancing reasoning abilities of large language models with sequences. arXiv preprint arXiv:2503.13109, 202511. - Chen et al. [202512] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https://github.com/Deep-Agent/R1-V, 202512. Accessed: 2025-02-02. - Chen et al. [202513] Michael K Chen, Xikun Zhang, and Dacheng Tao. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint arXiv:2501.14851, 202513. - Chen et al. [202514] Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Fan Yang, Zenan Zhou, Weipeng Chen, Haofen Wang, Jeff Z Pan, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470, 202514. - Chen et al. [202515] Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as a judge. arXiv preprint arXiv:2504.00050, 202515. - Chen et al. [20247] Qiguang Chen, Libo Qin, Jiaqi WANG, Jingxuan Zhou, and Wanxiang Che. Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 20247. URL https://openreview.net/forum?id=pC44UMwy2v. - Chen et al. [20248] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3 CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8199–8221, Bangkok, Thailand, August 20248. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.446. URL https://aclanthology.org/2024.acl-long.446/. - Chen et al. [202516] Qiguang Chen, Libo Qin, Jinhao Liu, Yue Liao, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. Rbf++: Quantifying and optimizing reasoning boundaries across measurable and unmeasurable capabilities for chain-of-thought reasoning. arXiv preprint arXiv:2505.13307, 202516. - Chen et al. [202517] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, and Ting Liu. Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model. arXiv preprint arXiv:2502.03325, 202517. - Chen et al. [202518] Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, et al. Ai4research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.01903, 202518. - Chen et al. [20249] Qiqi Chen, Xinpeng Wang, Philipp Mondorf, Michael A Hedderich, and Barbara Plank. Understanding when tree of thoughts succeeds: Larger models excel in generation, not discrimination. arXiv preprint arXiv:2410.17820, 20249. - Chen et al. [202519] Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, and Junxian He. Bring reason to vision: Understanding perception and reasoning through model merging. arXiv preprint arXiv:2505.05464, 202519. - Chen et al. [202520] Shuang Chen, Yue Guo, Zhaochen Su, Yafu Li, Yulun Wu, Jiacheng Chen, Jiayu Chen, Weijie Wang, Xiaoye Qu, and Yu Cheng. Advancing multimodal reasoning: From optimized cold start to staged reinforcement learning. arXiv preprint arXiv:2506.04207, 202520. - Chen and Li [2024] Sijia Chen and Baochun Li. Toward adaptive reasoning in large language models with thought rollback. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 7033–7056. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/chen24y.html. - Chen et al. [202521] Weizhe Chen, Sven Koenig, and Bistra Dilkina. Iterative deepening sampling for large language models. arXiv preprint arXiv:2502.05449, 202521. - Chen et al. [2023] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, November 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=YfZ4ZPt8zd. - Chen et al. [202522] Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Rui Zheng, Nijun Li, Tao Gui, Yun Li, et al. Better process supervision with bi-directional rewarding signals. arXiv preprint arXiv:2503.04618, 202522. - Chen et al. [202523] Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, et al. Unveiling the key factors for distilling chain-of-thought reasoning. arXiv preprint arXiv:2502.18001, 202523. - Chen et al. [202410] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 202410. - Chen et al. [202411] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, January 202411. URL https://openreview.net/forum?id=KuPixIqPiq. - Chen et al. [202524] Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner Fabien Roger Vlad Mikulik, Sam Bowman, Jan Leike Jared Kaplan, et al. Reasoning models don’t always say what they think. April 202524. URL https://www.anthropic.com/research/reasoning-models-dont-say-think. - Chen et al. [202412] Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. A simple and provable scaling law for the test-time compute of large language models. arXiv preprint arXiv:2411.19477, 202412. - Chen et al. [202413] Yezeng Chen, Zui Chen, and Yi Zhou. Brain-inspired two-stage approach: Enhancing mathematical reasoning by imitating human thought processes. arXiv preprint arXiv:2403.00800, 202413. - Chen et al. [202525] Yihang Chen, Haikang Deng, Kaiqiao Han, and Qingyue Zhao. Policy frameworks for transparent chain-of-thought reasoning in large language models. arXiv preprint arXiv:2503.14521, 202525. - Chen et al. [202526] Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprint arXiv:2502.13842, 202526. - Chen et al. [202527] Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, and Chuang Gan. Scaling autonomous agents via automatic reward modeling and planning. In The Thirteenth International Conference on Learning Representations, January 202527. URL https://openreview.net/forum?id=womU9cEwcO. - Chen et al. [202414] Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Wanxiang Che, Hang Yan, Kai Chen, and Dahua Lin. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. arXiv preprint arXiv:2409.01893, 202414. - Chen et al. [202528] Zihan Chen, Song Wang, Zhen Tan, Xingbo Fu, Zhenyu Lei, Peng Wang, Huan Liu, Cong Shen, and Jundong Li. A survey of scaling in large language model reasoning. arXiv preprint arXiv:2504.02181, 202528. - Chen et al. [202415] Ziru Chen, Michael White, Ray Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for LLM planning? it depends on the discriminator. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13659–13678, Bangkok, Thailand, August 202415. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.738. URL https://aclanthology.org/2024.acl-long.738/. - Chen et al. [202416] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 6621–6642. PMLR, 21–27 Jul 202416. URL https://proceedings.mlr.press/v235/chen24j.html. - Chen et al. [202529] Zui Chen, Tianqiao Liu, Mi Tian, Qing Tong, Weiqi Luo, and Zitao Liu. Advancing math reasoning in language models: The impact of problem-solving data, data synthesis methods, and training stages. arXiv preprint arXiv:2501.14002, 202529. - Cheng et al. [2025] Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758, 2025. - Cheng et al. [20252] Jiahao Cheng, Tiancheng Su, Jia Yuan, Guoxiu He, Jiawei Liu, Xinqi Tao, Jingwen Xie, and Huaxia Li. Chain-of-thought prompting obscures hallucination cues in large language models: An empirical evaluation. arXiv preprint arXiv:2506.17088, 20252. - Cheng et al. [2024] Jiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu, Yida Lu, Dan Zhang, Yuxiao Dong, Jie Tang, Hongning Wang, and Minlie Huang. Spar: Self-play with tree-search refinement to improve instruction-following in large language models. arXiv preprint arXiv:2412.11605, 2024. - Cheng et al. [20253] Junhang Cheng, Fang Liu, Chengru Wu, and Li Zhang. Adaptivellm: A framework for selecting optimal cost-efficient llm for code-generation based on cot length. arXiv preprint arXiv:2506.10525, 20253. - Cheng et al. [20242] Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. Vision-language models can self-improve reasoning via reflection. arXiv preprint arXiv:2411.00855, 20242. - Cheng et al. [20254] Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Think more, hallucinate less: Mitigating hallucinations via dual process of fast and slow thinking. arXiv preprint arXiv:2501.01306, 20254. - Cheng et al. [20255] Zhengxiang Cheng, Dongping Chen, Mingyang Fu, and Tianyi Zhou. Optimizing length compression in large reasoning models. arXiv preprint arXiv:2506.14755, 20255. - Cheng et al. [2021] Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.06712, 2021. - Cheng et al. [20256] Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, et al. Revisiting reinforcement learning for llm reasoning from a cross-domain perspective. arXiv preprint arXiv:2506.14965, 20256. - Cheng et al. [20243] Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. arXiv preprint arXiv:2412.12932, 20243. - Cheng et al. [20257] Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang, Alex Jinpeng Wang, Zhi Chen, Wanxiang Che, et al. Visual thoughts: A unified perspective of understanding multimodal chain-of-thought. arXiv preprint arXiv:2505.15510, 20257. - Chern et al. [2025] Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu. Thinking with generated images. arXiv preprint arXiv:2505.22525, 2025. - Chia et al. [2024] Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. PuzzleVQA: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 16259–16273, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.962. URL https://aclanthology.org/2024.findings-acl.962/. - Chijiwa et al. [2025] Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Susumu Takeuchi. Portable reward tuning: Towards reusable fine-tuning across different pretrained models. arXiv preprint arXiv:2502.12776, 2025. - Choi et al. [2025] Daewon Choi, Jimin Lee, Jihoon Tack, Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, and Sravan Babu Bodapati. Think clearly: Improving reasoning via redundant token pruning. arXiv preprint arXiv:2507.08806, 2025. - Chollet [2019] François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. - Chollet et al. [2024] Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604, 2024. - Chollet et al. [2025] Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831, 2025. - Choudhury [2025] Sanjiban Choudhury. Process reward models for llm agents: Practical framework and directions. arXiv preprint arXiv:2502.10325, 2025. - Chowdhury and Caragea [2025] Jishnu Ray Chowdhury and Cornelia Caragea. Zero-shot verification-guided chain of thoughts. arXiv preprint arXiv:2501.13122, 2025. - Christakopoulou et al. [2024] Konstantina Christakopoulou, Shibl Mourad, and Maja Mataric. Agents thinking fast and slow: A talker-reasoner architecture. In NeurIPS 2024 Workshop on Open-World Agents, October 2024. URL https://openreview.net/forum?id=xPhcP6rbI4. - Chu et al. [2025] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025. - Chu et al. [2024] Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1173–1203, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.65. URL https://aclanthology.org/2024.acl-long.65/. - Chu-Carroll et al. [2024] Jennifer Chu-Carroll, Andrew Beck, Greg Burnham, David OS Melville, David Nachman, A Erdem Özcan, and David Ferrucci. Beyond llms: Advancing the landscape of complex reasoning. arXiv preprint arXiv:2402.08064, 2024. - Chung et al. [2025] Daniel JH Chung, Zhiqi Gao, Yurii Kvasiuk, Tianyi Li, Moritz Münchmeyer, Maja Rudolph, Frederic Sala, and Sai Chaitanya Tadepalli. Theoretical physics benchmark (tpbench)–a dataset and study of ai reasoning capabilities in theoretical physics. arXiv preprint arXiv:2502.15815, 2025. - Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. - Cuadron et al. [2025] Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025. - Cui et al. [2025] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025. - Cui et al. [20252] Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 20252. - Cui et al. [20253] Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, et al. Stepwise perplexity-guided refinement for efficient chain-of-thought reasoning in large language models. arXiv preprint arXiv:2502.13260, 20253. - Cui and Zuo [2025] Yu Cui and Cong Zuo. Practical reasoning interruption attacks on reasoning large language models. arXiv preprint arXiv:2505.06643, 2025. - Cui et al. [20254] Yu Cui, Bryan Hooi, Yujun Cai, and Yiwei Wang. Process or result? manipulated ending tokens can mislead reasoning llms to ignore the correct reasoning steps. arXiv preprint arXiv:2503.19326, 20254. - Dai et al. [2024] Jianbo Dai, Jianqiao Lu, Yunlong Feng, Dong Huang, Guangtao Zeng, Rongju Ruan, Ming Cheng, Haochen Tan, and Zhijiang Guo. Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation. arXiv preprint arXiv:2405.11430, 2024. - Dang et al. [2025] Jisheng Dang, Jingze Wu, Teng Wang, Xuanhui Lin, Nannan Zhu, Hongbo Chen, Wei-Shi Zheng, Meng Wang, and Tat-Seng Chua. Reinforcing video reasoning with focused thinking. arXiv preprint arXiv:2505.24718, 2025. - Dang and Ngo [2025] Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn’t. arXiv preprint arXiv:2503.16219, 2025. - Deng et al. [2024] Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838, 2024. - Di Langosco et al. [2022] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, pages 12004–12019. PMLR, October 2022. URL https://proceedings.mlr.press/v162/langosco22a/langosco22a.pdf. - Ding et al. [2025] Bowen Ding, Yuhan Chen, Futing Wang, Lingfeng Ming, and Tao Lin. Do thinking tokens help or trap? towards more efficient large reasoning model. arXiv preprint arXiv:2506.23840, 2025. - Ding et al. [20252] Yifu Ding, Wentao Jiang, Shunyu Liu, Yongcheng Jing, Jinyang Guo, Yingjie Wang, Jing Zhang, Zengmao Wang, Ziwei Liu, Bo Du, et al. Dynamic parallel tree search for efficient llm reasoning. arXiv preprint arXiv:2502.16235, 20252. - Dong et al. [2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, November 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=m7p5O7zblY. - Dong et al. [2024] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024. - Dong et al. [20242] Junnan Dong, Zijin Hong, Yuanchen Bei, Feiran Huang, Xinrun Wang, and Xiao Huang. Clr-bench: Evaluating large language models in college-level reasoning. arXiv preprint arXiv:2410.17558, 20242. - Dong and Ma [2025] Kefan Dong and Tengyu Ma. Beyond limited data: Self-play llm theorem provers with iterative conjecturing and proving. arXiv preprint arXiv:2502.00212, 2025. - Dong et al. [20243] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432, 20243. - Dong et al. [2025] Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu. Emergent response planning in llm. arXiv preprint arXiv:2502.06258, 2025. - Dou et al. [2024] Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391, 2024. - Drori et al. [2025] Iddo Drori, Gaston Longhitano, Mao Mao, Seunghwan Hyun, Yuke Zhang, Sungjun Park, Zachary Meeks, Xin-Yu Zhang, Ben Segev, Howard Yong, et al. Diverse inference and verification for advanced reasoning. arXiv preprint arXiv:2502.09955, 2025. - Du et al. [2025] Kounianhua Du, Hanjing Wang, Jianxing Liu, Jizheng Chen, Xinyi Dai, Yasheng Wang, Ruiming Tang, Yong Yu, Jun Wang, and Weinan Zhang. Boost, disentangle, and customize: A robust system2-to-system1 pipeline for code generation. arXiv preprint arXiv:2502.12492, 2025. - Du et al. [20252] Weihua Du, Yiming Yang, and Sean Welleck. Optimizing temperature for language models with multi-sample inference. arXiv preprint arXiv:2502.05234, 20252. - Du et al. [20253] Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 20253. - Du et al. [20254] Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing o1-like mllm. arXiv preprint arXiv:2501.01904, 20254. - Duan et al. [2025] Keyu Duan, Zichen Liu, Xin Mao, Tianyu Pang, Changyu Chen, Qiguang Chen, Michael Qizhe Shieh, and Longxu Dou. Efficient process reward model training via active learning. arXiv preprint arXiv:2504.10559, 2025. - Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. - Dutta et al. [2024] Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. Transactions on Machine Learning Research, July 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=uHLDkQVtyC. - El-Kishky et al. [2025] Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807, 2025. - Ethayarajh et al. [2024] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. - Fan et al. [2025] Chongyu Fan, Yihua Zhang, Jinghan Jia, Alfred Hero, and Sijia Liu. Cyclicreflex: Improving large reasoning models via cyclical reflection token scheduling. arXiv preprint arXiv:2506.11077, 2025. - Fan et al. [20252] Siqi Fan, Peng Han, Shuo Shang, Yequan Wang, and Aixin Sun. Cothink: Token-efficient reasoning via instruct models guiding reasoning models. arXiv preprint arXiv:2505.22017, 20252. - Fan et al. [20253] Tiantian Fan, Lingjun Liu, Yu Yue, Jiaze Chen, Chengyi Wang, Qiying Yu, Chi Zhang, Zhiqi Lin, Ruofei Zhu, Yufeng Yuan, et al. Truncated proximal policy optimization. arXiv preprint arXiv:2506.15050, 20253. - Fang et al. [2025] Yi Fang, Wenjie Wang, Yang Zhang, Fengbin Zhu, Qifan Wang, Fuli Feng, and Xiangnan He. Large language models for recommendation with deliberative user preference alignment. arXiv preprint arXiv:2502.02061, 2025. - Fei et al. [2025] Wu Fei, Hao Kong, Shuxian Liang, Yang Lin, Yibo Yang, Jing Tang, Lei Chen, and Xiansheng Hua. Self-guided process reward optimization with masked step advantage for process reinforcement learning. arXiv preprint arXiv:2507.01551, 2025. - Feng et al. [2023] Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A theoretical perspective. In Thirty-seventh Conference on Neural Information Processing Systems, September 2023. URL https://openreview.net/forum?id=qHrADgAdYu. - Feng et al. [2025] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536, 2025. - Feng et al. [20252] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 20252. - Feng et al. [20253] Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. arXiv preprint arXiv:2504.10903, 20253. - Feng et al. [20254] Xiachong Feng, Longxu Dou, and Lingpeng Kong. Reasoning does not necessarily improve role-playing ability. arXiv preprint arXiv:2502.16940, 20254. - Feng et al. [20255] Xueyang Feng, Bo Lan, Quanyu Dai, Lei Wang, Jiakai Tang, Xu Chen, Zhenhua Dong, and Ji-Rong Wen. Improving retrospective language agents via joint policy gradient optimization. arXiv preprint arXiv:2503.01490, 20255. - Feng et al. [20256] Yichen Feng, Zhangchen Xu, Fengqing Jiang, Yuetai Li, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Visualsphinx: Large-scale synthetic vision logic puzzles for rl. arXiv preprint arXiv:2505.23977, 20256. - Fernando et al. [2024] Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 13481–13544. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/fernando24a.html. - Ferrag et al. [2025] Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. Reasoning beyond limits: Advances and open problems for llms. arXiv preprint arXiv:2503.22732, 2025. - Ferraz et al. [2024] Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. LLM self-correction with deCRIM: Decompose, critique, and refine for enhanced following of instructions with multiple constraints. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, October 2024. URL https://openreview.net/forum?id=RQ6Ff8lso0. - Fu et al. [2025] Jiarun Fu, Lizhong Ding, Hao Li, Pengqi Li, Qiuning Wei, and Xu Chen. Unveiling and causalizing cot: A causal pespective. arXiv preprint arXiv:2502.18239, 2025. - Fu et al. [20252] Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 20252. URL https://arxiv.org/abs/2505.24298. - Fu et al. [2023] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, February 2023. URL https://openreview.net/forum?id=yf1icZHC-l9. - Fu et al. [20253] Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506.19767, 20253. - Gallego [2025] Víctor Gallego. Metasc: Test-time safety specification optimization for language models. arXiv preprint arXiv:2502.07985, 2025. - Gan et al. [2025] Zeyu Gan, Yun Liao, and Yong Liu. Rethinking external slow-thinking: From snowball errors to probability of correct reasoning. arXiv preprint arXiv:2501.15602, 2025. - Gandhi et al. [2024] Kanishk Gandhi, Denise HJ Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of search (sos): Learning to search in language. In First Conference on Language Modeling, July 2024. URL https://openreview.net/pdf?id=2cop2jmQVL. - Gandhi et al. [2025] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025. - Gao et al. [2024] Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Tianyu Liu, and Baobao Chang. The reason behind good or bad: Towards a better mathematical verifier with natural language feedback, 2024. - Gao et al. [20242] Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, et al. Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. arXiv preprint arXiv:2406.14024, 20242. - Gao et al. [20243] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for llm reasoning. arXiv preprint arXiv:2410.15115, 20243. - Gao et al. [2023] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10764–10799. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/gao23f.html. - Gao et al. [2025] Silin Gao, Antoine Bosselut, Samy Bengio, and Emmanuel Abbe. Augmenting llms’ reasoning by reinforcing abstract thinking. arXiv preprint arXiv:2506.07751, 2025. - Gao et al. [20252] Tianchen Gao, Jiashun Jin, Zheng Tracy Ke, and Gabriel Moryoussef. A comparison of deepseek and other llms. arXiv preprint arXiv:2502.03688, 20252. - Gao et al. [20244] Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. Interpretable contrastive monte carlo tree search reasoning. arXiv preprint arXiv:2410.01707, 20244. - Ge et al. [2025] Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi, and Xueqi Cheng. Innate reasoning is not enough: In-context learning enhances reasoning large language models with less overthinking. arXiv preprint arXiv:2503.19602, 2025. - Gehring et al. [2024] Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089, 2024. - Geiping et al. [2025] Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025. - Gendron et al. [2024] Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not strong abstract reasoners. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24, August 2024. ISBN 978-1-956792-04-1. doi: 10.24963/ijcai.2024/693. URL https://doi.org/10.24963/ijcai.2024/693. - Gero et al. [2023] Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao, and Hoifung Poon. Self-verification improves few-shot clinical information extraction. In ICML 3rd Workshop on Interpretable Machine Learning in Healthcare (IMLH), June 2023. URL https://openreview.net/forum?id=SBbJICrglS. - Ghosh et al. [2025] Akash Ghosh, Debayan Datta, Sriparna Saha, and Chirag Agarwal. The multilingual mind: A survey of multilingual reasoning in language models. arXiv preprint arXiv:2502.09457, 2025. - Giadikiaroglou et al. [2024] Panagiotis Giadikiaroglou, Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. Puzzle solving using reasoning of large language models: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11574–11591, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.646. URL https://aclanthology.org/2024.emnlp-main.646/. - Gladstone et al. [2025] Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. Energy-based transformers are scalable learners and thinkers. arXiv preprint arXiv:2507.02092, 2025. - Glazer et al. [2024] Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872, 2024. - GLM et al. [2024] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. - Golovneva et al. [2023] Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=xYlJRpzZtsY. - Golovneva et al. [20232] Olga Golovneva, Sean O’Brien, Ramakanth Pasunuru, Tianlu Wang, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. PATHFINDER: Guided search over multi-step reasoning paths. In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, December 20232. URL https://openreview.net/forum?id=5TsfEEwRsu. - Gong et al. [2025] Ruihan Gong, Yue Liu, Wenjie Qu, Mingzhe Du, Yufei He, Yingwei Ma, Yulin Chen, Xiang Liu, Yi Wen, Xinfeng Li, et al. Efficient reasoning via chain of unconscious thought. arXiv preprint arXiv:2505.19756, 2025. - Gottweis et al. [2025] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025. - Gou et al. [2023] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738, 2023. - Gou et al. [20232] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 20232. - Grosse et al. [2024] Julia Grosse, Ruotian Wu, Ahmad Rashid, Philipp Hennig, Pascal Poupart, and Agustinus Kristiadi. Uncertainty-guided optimization on large language model search trees. arXiv preprint arXiv:2407.03951, 2024. - Gu et al. [2025] Yanggan Gu, Junzhuo Li, Sirui Huang, Xin Zou, Zhenghua Li, and Xuming Hu. Capturing nuanced preferences: Preference-aligned distillation for small language models. arXiv preprint arXiv:2502.14272, 2025. - Guan et al. [2024] Xinyan Guan, Yanjiang Liu, Xinyu Lu, Boxi Cao, Ben He, Xianpei Han, Le Sun, Jie Lou, Bowen Yu, Yaojie Lu, et al. Search, verify and feedback: Towards next generation post-training paradigm of foundation models via verifier engineering. arXiv preprint arXiv:2411.11504, 2024. - Guan et al. [2025] Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. Deeprag: Thinking to retrieval step by step for large language models. arXiv preprint arXiv:2502.01142, 2025. - Guan et al. [20252] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519, 20252. - Guha et al. [2025] Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178, 2025. - Gulati et al. [2024] Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, and Sanmi Koyejo. Putnam-AXIOM: A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, 2024. URL https://openreview.net/forum?id=YXnwlZe0yf. - Gulcehre et al. [2023] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023. - Guo et al. [2024] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. - Guo et al. [2025] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. - Guo et al. [20252] Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qiuyinzhe Zhang, Yu Sun, Kai Chen, Xipeng Qiu, et al. Critiq: Mining data quality criteria from human preferences. arXiv preprint arXiv:2502.19279, 20252. - Guo et al. [20242] Kehan Guo, Bozhao Nan, Yujun Zhou, Taicheng Guo, Zhichun Guo, Mihir Surve, Zhenwen Liang, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Can LLMs solve molecule puzzles? a multimodal benchmark for molecular structure elucidation. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, September 20242. URL https://openreview.net/forum?id=t1mAXb4Cop. - Guo et al. [20253] Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let’s verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926, 20253. - Han et al. [2025] Dongge Han, Menglin Xia, Daniel Madrigal Diaz, Samuel Kessler, Ankur Mallick, Xuchao Zhang, Mirian Del Carmen Hipolito Garcia, Jin Xu, Victor Rühle, and Saravan Rajmohan. Enhancing reasoning capabilities of small language models with blueprints and prompt template search. arXiv preprint arXiv:2506.08669, 2025. - Han et al. [2024] Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, and Zhenting Wang. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547, 2024. - Hanna et al. [2023] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. September 2023. URL https://openreview.net/forum?id=p4PckNQR8k. - Hao et al. [2023] Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.507. URL https://aclanthology.org/2023.emnlp-main.507/. - Hao et al. [2024] Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. LLM reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. In First Conference on Language Modeling, July 2024. URL https://openreview.net/forum?id=b0y6fbSUG0. - Hao et al. [20242] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 20242. - Hao et al. [2025] Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025. - Havrilla et al. [2024] Alexander Havrilla, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. GLore: When, where, and how to improve LLM reasoning via global and local refinements. In Forty-first International Conference on Machine Learning, May 2024. URL https://openreview.net/forum?id=LH6R06NxdB. - He et al. [2024] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.211. URL https://aclanthology.org/2024.acl-long.211/. - He et al. [20242] Chengbo He, Bochao Zou, Xin Li, Jiansheng Chen, Junliang Xing, and Huimin Ma. Enhancing llm reasoning with multi-path collaborative reactive and reflection agents. arXiv preprint arXiv:2501.00430, 20242. - He et al. [2025] Feng He, Zijun Chen, Xinnian Liang, Tingting Ma, Yunqi Qiu, Shuangzhi Wu, and Junchi Yan. Protoreasoning: Prototypes as the foundation for generalizable reasoning in llms. arXiv preprint arXiv:2506.15211, 2025. - He et al. [20252] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series. https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaonser-Series-1d0bc9ae823a80459b46c149e4f51680, 20252. Notion Blog. - He et al. [20253] Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246, 20253. - He and Roy [2025] Kang He and Kaushik Roy. Logictree: Structured proof exploration for coherent and rigorous logical reasoning with large language models. arXiv preprint arXiv:2504.14089, 2025. - He et al. [20243] Mingqian He, Yongliang Shen, Wenqi Zhang, Zeqi Tan, and Weiming Lu. Advancing process verification for large language models via tree-based preference learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2086–2099, Miami, Florida, USA, November 20243. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.125. URL https://aclanthology.org/2024.emnlp-main.125/. - He et al. [20254] Qiangqiang He, Shuwei Qian, Jie Zhang, and Chongjun Wang. Inference retrieval-augmented multi-modal chain-of-thoughts reasoning for language models. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 20254. doi: 10.1109/ICASSP49660.2025.10888701. URL https://openreview.net/pdf/9a7e7a9787d14ac8302215f8e4ef959606b78a94.pdf. - He et al. [20255] Shenghua He, Tian Xia, Xuan Zhou, and Hui Wei. Response-level rewards are all you need for online reinforcement learning in llms: A mathematical perspective. arXiv preprint arXiv:2506.02553, 20255. - He et al. [20256] Tao He, Hao Li, Jingchang Chen, Runxuan Liu, Yixin Cao, Lizi Liao, Zihao Zheng, Zheng Chu, Jiafeng Liang, Ming Liu, et al. A survey on complex reasoning of large language models through the lens of self-evolution. February 20256. - He et al. [20257] Xingyang He, Xiao Ling, and Jie Liu. Smartthinker: Learning to compress and preserve reasoning by step-level length control. arXiv preprint arXiv:2507.04348, 20257. - He et al. [20258] Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprint arXiv:2502.19361, 20258. - He et al. [20259] Yang He, Xiao Ding, Bibo Cai, Yufei Zhang, Kai Xiong, Zhouhao Sun, Bing Qin, and Ting Liu. Self-route: Automatic mode switching via capability estimation for efficient reasoning. arXiv preprint arXiv:2505.20664, 20259. - He et al. [202510] Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, et al. Mmboundary: Advancing mllm knowledge boundary awareness through reasoning step confidence calibration. arXiv preprint arXiv:2505.23224, 202510. - Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), October 2021. URL https://openreview.net/forum?id=7Bywt2mQsCe. - Heyman and Zylberberg [2025] Alex Heyman and Joel Zylberberg. Evaluating the systematic reasoning abilities of large language models through graph coloring. arXiv preprint arXiv:2502.07087, 2025. - Heyman and Zylberberg [20252] Alex Heyman and Joel Zylberberg. Reasoning large language model errors arise from hallucinating critical problem features. arXiv preprint arXiv:2505.12151, 20252. - Ho et al. [2023] Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852–14882, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.830. URL https://aclanthology.org/2023.acl-long.830/. - Hochlehnert et al. [2025] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.07086, 2025. - Hoffman et al. [2023] Matthew Douglas Hoffman, Du Phan, david dohan, Sholto Douglas, Tuan Anh Le, Aaron T Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training chain-of-thought via latent-variable inference. In Thirty-seventh Conference on Neural Information Processing Systems, September 2023. URL https://openreview.net/forum?id=a147pIS2Co. - Hong et al. [2024] Ruixin Hong, Xinyu Pang, and Changshui Zhang. Advances in reasoning by prompting large language models: A survey. Cybernetics and Intelligence, pages 1–15, 2024. doi: 10.26599/CAI.2024.9390004. - Hong et al. [20242] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14281–14290, June 20242. URL https://openaccess.thecvf.com/content/CVPR2024/papers/Hong_CogAgent_A_Visual_Language_Model_for_GUI_Agents_CVPR_2024_paper.pdf. - Hosseini et al. [2024] Arian Hosseini, Alessandro Sordoni, Daniel Kenji Toyama, Aaron Courville, and Rishabh Agarwal. Not all LLM reasoners are created equal. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, October 2024. URL https://openreview.net/forum?id=aPAWbip1xV. - Hosseini et al. [20242] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STar: Training verifiers for self-taught reasoners. In First Conference on Language Modeling, July 20242. URL https://openreview.net/forum?id=stmqBSW2dV. - Hou et al. [2025] Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296, 2025. - Hou et al. [20252] Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501.11651, 20252. - Hu [2025] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025. - Hu et al. [2024] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024. - Hu et al. [2025] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero, February 2025. - Hu et al. [20252] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 20252. - Hu et al. [20242] Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. arXiv preprint arXiv:2408.09559, 20242. - Hu et al. [20243] Mengkang Hu, Yao Mu, Xinmiao Chelsey Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. Tree-planner: Efficient close-loop task planning with large language models. In The Twelfth International Conference on Learning Representations, January 20243. URL https://openreview.net/forum?id=Glcsog6zOe. - Hu et al. [20244] Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint arXiv:2408.00764, 20244. - Hu et al. [20253] Mengkang Hu, Tianxing Chen, Yude Zou, Yuheng Lei, Qiguang Chen, Ming Li, Hongyuan Zhang, Wenqi Shao, and Ping Luo. Text2world: Benchmarking large language models for symbolic world model generation. arXiv preprint arXiv:2502.13092, 20253. - Hu et al. [20254] Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, et al. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. arXiv preprint arXiv:2505.23885, 20254. - Hu et al. [20255] Renjun Hu, Yi Cheng, Libin Meng, Jiaxin Xia, Yi Zong, Xing Shi, and Wei Lin. Training an llm-as-a-judge model: Pipeline, insights, and practical lessons. arXiv preprint arXiv:2502.02988, 20255. - Hu et al. [20245] Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, March 20245. URL https://openreview.net/forum?id=ZWyLjimciT. - Huan et al. [2025] Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning. arXiv preprint arXiv:2507.00432, 2025. - Huang et al. [2025] Chenghua Huang, Lu Wang, Fangkai Yang, Pu Zhao, Zhixu Li, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Lean and mean: Decoupled value policy optimization with global value guidance. arXiv preprint arXiv:2502.16944, 2025. - Huang et al. [20252] Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration. arXiv preprint arXiv:2503.00031, 20252. - Huang et al. [20253] Chengyu Huang, Zhengxin Zhang, and Claire Cardie. Hapo: Training language models to reason concisely via history-aware policy optimization. arXiv preprint arXiv:2505.11225, 20253. - Huang et al. [20254] Haiduo Huang, Fuwei Yang, Zhenhua Liu, Yixing Xu, Jinze Li, Yang Liu, Xuanwu Yin, Dong Li, Pengju Ren, and Emad Barsoum. Jakiro: Boosting speculative decoding with decoupled multi-head via moe. arXiv preprint arXiv:2502.06282, 20254. - Huang et al. [2023] Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.826. URL https://aclanthology.org/2023.findings-emnlp.826/. - Huang et al. [20255] Hui Huang, Yancheng He, Hongli Zhou, Rui Zhang, Wei Liu, Weixun Wang, Wenbo Su, Bo Zheng, and Jiaheng Liu. Think-j: Learning to think for generative llm-as-a-judge. arXiv preprint arXiv:2505.14268, 20255. - Huang et al. [2024] Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. How far are we on the decision-making of llms? evaluating llms’ gaming ability in multi-agent environments. arXiv preprint arXiv:2403.11807, 2024. - Huang and Zhang [2024] Jiaxing Huang and Jingyi Zhang. A survey on evaluation of multimodal large language models. arXiv preprint arXiv:2408.15769, 2024. - Huang and Chang [2023] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.67. URL https://aclanthology.org/2023.findings-acl.67/. - Huang et al. [20242] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, January 20242. URL https://openreview.net/forum?id=IkmD3fKBPQ. - Huang et al. [20256] Jinyang Huang, Xiachong Feng, Qiguang Chen, Hanjie Zhao, Zihui Cheng, Jiesong Bai, Jingxuan Zhou, Min Li, and Libo Qin. Mldebugging: Towards benchmarking code debugging across multi-library scenarios. arXiv preprint arXiv:2506.13824, 20256. - Huang et al. [20257] Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, et al. Math-perturb: Benchmarking llms’ math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502.06453, 20257. - Huang et al. [20243] Lei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao, Yuchun Fan, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, and Bing Qin. Advancing large language model attribution through self-improving. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3822–3836, Miami, Florida, USA, November 20243. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.223. URL https://aclanthology.org/2024.emnlp-main.223/. - Huang et al. [20258] Shijue Huang, Hongru Wang, Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, and Yi R Fung. Adactrl: Towards adaptive and controllable reasoning via difficulty-aware budgeting. arXiv preprint arXiv:2505.18822, 20258. - Huang et al. [20259] Shulin Huang, Linyi Yang, Yan Song, Shuang Chen, Leyang Cui, Ziyu Wan, Qingcheng Zeng, Ying Wen, Kun Shao, Weinan Zhang, et al. Thinkbench: Dynamic out-of-distribution evaluation for robust llm reasoning. arXiv preprint arXiv:2502.16268, 20259. - Huang et al. [202510] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. Safety tax: Safety alignment makes your large reasoning models less reasonable. arXiv preprint arXiv:2503.00555, 202510. - Huang et al. [202511] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 202511. - Huang et al. [202512] Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the potential of test-time scaling for medical reasoning with large language models. arXiv preprint arXiv:2504.00869, 202512. - Huang et al. [20244] Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 20244. - Huang et al. [202513] Yuzhen Huang, Weihao Zeng, Xingshan Zeng, Qi Zhu, and Junxian He. Pitfalls of rule-and model-based verifiers–a case study on mathematical reasoning. arXiv preprint arXiv:2505.22203, 202513. - Huang et al. [202514] Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679, 202514. - Huang et al. [20245] Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, and Pengfei Liu. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent AI. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 20245. URL https://openreview.net/forum?id=ayF8bEKYQy. - Huang et al. [20246] Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 20246. - Huang et al. [202515] Zhongzhen Huang, Gui Geng, Shengyi Hua, Zhen Huang, Haoyang Zou, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. O1 replication journey–part 3: Inference-time scaling for medical reasoning. arXiv preprint arXiv:2501.06458, 202515. - Hui et al. [2024] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186, 2024. - Hwang et al. [2024] Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. Self-explore: Enhancing mathematical reasoning in language models with fine-grained rewards. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1444–1466, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.78. URL https://aclanthology.org/2024.findings-emnlp.78/. - Imani et al. [2023] Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. 2023. - Islam et al. [2024] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403, 2024. - Ivison et al. [2023] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023. - Ivison et al. [2024] Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=JMBWTlazjW. - Jaech et al. [2024] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. - Jain et al. [2025] Eeshaan Jain, Johann Wenckstern, Benedikt von Querfurth, and Charlotte Bunne. Test-time view selection for multi-modal decision making. In ICLR 2025 Workshop on Machine Learning for Genomics Explorations, March 2025. URL https://openreview.net/forum?id=aNmZ9s6BZV. - Jain et al. [20252] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, January 20252. URL https://openreview.net/forum?id=chfJJYC3iL. - Jang and Kim [2022] Sooyoung Jang and Hyung-Il Kim. Entropy-aware model initialization for effective exploration in deep reinforcement learning. Sensors, 22(15):5845, 2022. - Ji et al. [2025] Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, et al. The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models. arXiv preprint arXiv:2503.02875, 2025. - Ji et al. [20252] Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek’s multi-head latent attention in any transformer-based llms. arXiv preprint arXiv:2502.14837, 20252. - Ji [2024] Yichao Ji. A small step towards reproducing openai o1: Progress report on the steiner open source models, October 2024. URL https://medium.com/@peakji/b9a756a00855. - Ji et al. [20253] Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Jia Xu, Linjian Mo, and Min Zhang. Test-time computing: from system-1 thinking to system-2 thinking. arXiv preprint arXiv:2501.02497, 20253. - Ji et al. [2023] Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating LLM hallucination via self reflection. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.123. URL https://aclanthology.org/2023.findings-emnlp.123/. - Jia et al. [2025] Boyu Jia, Junzhe Zhang, Huixuan Zhang, and Xiaojun Wan. Exploring and evaluating multimodal knowledge reasoning consistency of multimodal large language models. arXiv preprint arXiv:2503.04801, 2025. - Jia et al. [20252] Zeyu Jia, Alexander Rakhlin, and Tengyang Xie. Do we need to verify step by step? rethinking process supervision from a theoretical perspective. arXiv preprint arXiv:2502.10581, 20252. - Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, October 2023. - Jiang et al. [2024] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. - Jiang et al. [2025] Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. arXiv preprint arXiv:2502.12025, 2025. - Jiang et al. [20242] Huchen Jiang, Yangyang Ma, Chaofan Ding, Kexin Luan, and Xinhan Di. Towards intrinsic self-correction enhancement in monte carlo tree search boosted reasoning via iterative preference learning. arXiv preprint arXiv:2412.17397, 20242. - Jiang et al. [20243] Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, and Tao Zhang. Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. arXiv preprint arXiv:2412.12881, 20243. - Jiang et al. [20244] Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. Technical report: Enhancing llm reasoning with reward-guided tree search. arXiv preprint arXiv:2411.11694, 20244. - Jiang et al. [20252] Nan Jiang, Ziming Wu, De-Chuan Zhan, Fuming Lai, and Shaobing Lian. Dart: Distilling autoregressive reasoning to silent thought. arXiv preprint arXiv:2506.11752, 20252. - Jiang et al. [20253] Shuyang Jiang, Yusheng Liao, Zhe Chen, Ya Zhang, Yanfeng Wang, and Yu Wang. Meds 3: Towards medical small language models with self-evolved slow thinking. arXiv preprint arXiv:2501.12051, 20253. - Jiang et al. [20254] Yuxuan Jiang, Dawei Li, and Frank Ferraro. Drp: Distilled reasoning pruning with skill-aware step decomposition for efficient large reasoning models. arXiv preprint arXiv:2505.13975, 20254. - Jimenez et al. [2024] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, January 2024. URL https://openreview.net/forum?id=VTF8yNQM66. - Jin et al. [2021] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14), July 2021. ISSN 2076-3417. doi: 10.3390/app11146421. URL https://www.mdpi.com/2076-3417/11/14/6421. - Jin et al. [2024] Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. Disentangling memory and reasoning ability in large language models. arXiv preprint arXiv:2411.13504, 2024. - Jin et al. [20242] Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1830–1842, Bangkok, Thailand, August 20242. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.108. URL https://aclanthology.org/2024.findings-acl.108/. - Jin et al. [2025] Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 558–573, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.37/. - Jin et al. [20252] Zhensheng Jin, Xinze Li, Yifan Ji, Chunyi Peng, Zhenghao Liu, Qi Shi, Yukun Yan, Shuo Wang, Furong Peng, and Ge Yu. Recut: Balancing reasoning length and accuracy in llms via stepwise trails and preference optimization. arXiv preprint arXiv:2506.10822, 20252. - Jones [2021] Andy L Jones. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113, 2021. - Jones and Bergen [2025] Cameron R Jones and Benjamin K Bergen. Large language models pass the turing test. arXiv preprint arXiv:2503.23674, 2025. - Kadam [2024] Prashank Kadam. Gpt-guided monte carlo tree search for symbolic regression in financial fraud detection. arXiv preprint arXiv:2411.04459, 2024. - Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. - Kamoi et al. [2024] Ryo Kamoi, Sarkar Snigdha Sarathi Das, Renze Lou, Jihyun Janice Ahn, Yilun Zhao, Xiaoxin Lu, Nan Zhang, Yusen Zhang, Haoran Ranran Zhang, Sujeeth Reddy Vummanthala, Salika Dave, Shaobo Qin, Arman Cohan, Wenpeng Yin, and Rui Zhang. Evaluating LLMs at detecting errors in LLM responses. In First Conference on Language Modeling, July 2024. URL https://openreview.net/forum?id=dnwRScljXr. - Kang et al. [2024] Jikun Kang, Xin Zhe Li, Xi Chen, Amirreza Kazemi, Qianyi Sun, Boxing Chen, Dong Li, Xu He, Quan He, Feng Wen, et al. Mindstar: Enhancing math reasoning in pre-trained llms at inference time. arXiv preprint arXiv:2405.16265, 2024. - Kang et al. [20242] Liwei Kang, Zirui Zhao, David Hsu, and Wee Sun Lee. On the empirical complexity of reasoning and planning in LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2897–2936, Miami, Florida, USA, November 20242. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.164. URL https://aclanthology.org/2024.findings-emnlp.164/. - Kang et al. [2025] Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of-thought without compromising effectiveness. 39(23):24312–24320, Apr 2025. - Kang et al. [20252] Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. arXiv preprint arXiv:2502.18581, 20252. - Kant et al. [2025] Manuj Kant, Sareh Nabi, Manav Kant, Roland Scharrer, Megan Ma, and Marzieh Nabi. Towards robust legal reasoning: Harnessing logical llms in law. arXiv preprint arXiv:2502.17638, 2025. - Kazemi et al. [2023] Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. LAMBADA: Backward chaining for automated reasoning in natural language. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6547–6568, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.361. URL https://aclanthology.org/2023.acl-long.361/. - Kazemnejad et al. [2024] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024. - Khalifa et al. [2023] Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Grace: Discriminator-guided chain-of-thought reasoning. arXiv preprint arXiv:2305.14934, 2023. - Kharinaev et al. [2025] Artyom Kharinaev, Viktor Moskvoretskii, Egor Shvetsov, Kseniia Studenikina, Bykov Mikhail, and Evgeny Burnaev. Investigating the impact of quantization methods on the safety and reliability of large language models. arXiv preprint arXiv:2502.15799, 2025. - Kim et al. [2025] Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B Tenenbaum, and Yejin Choi. Hypothesis-driven theory-of-mind reasoning for large language models. arXiv preprint arXiv:2502.11881, 2025. - Kim et al. [20252] Jiin Kim, Byeongjun Shin, Jinha Chung, and Minsoo Rhu. The cost of dynamic reasoning: Demystifying ai agents and test-time scaling from an ai infrastructure perspective. arXiv preprint arXiv:2506.04301, 20252. - Kim et al. [20253] Juno Kim, Denny Wu, Jason Lee, and Taiji Suzuki. Metastable dynamics of chain-of-thought reasoning: Provable benefits of search, rl and distillation. arXiv preprint arXiv:2502.01694, 20253. - Kim et al. [20254] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 20254. - Kim et al. [2024] Naryeong Kim, Sungmin Kang, Gabin An, and Shin Yoo. Lachesis: Predicting llm inference accuracy using structural properties of reasoning paths. arXiv preprint arXiv:2412.08281, 2024. - Kim et al. [2023] Seungone Kim, Se Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The CoT collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12685–12708, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.782. URL https://aclanthology.org/2023.emnlp-main.782/. - Kim et al. [20242] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4334–4353, Miami, Florida, USA, November 20242. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.248. URL https://aclanthology.org/2024.emnlp-main.248/. - Kim et al. [20255] Sunnie SY Kim, Jennifer Wortman Vaughan, Q Vera Liao, Tania Lombrozo, and Olga Russakovsky. Fostering appropriate reliance on large language models: The role of explanations, sources, and inconsistencies. arXiv preprint arXiv:2502.08554, 20255. - Koh et al. [2024] Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.01476, 2024. - Kong et al. [2025] Deqian Kong, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, et al. Scalable language models with posterior inference of latent thought vectors. arXiv preprint arXiv:2502.01567, 2025. - Kumar et al. [2025] Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit Iyyer, Amir Houmansadr, and Eugene Bagdasarian. Overthinking: Slowdown attacks on reasoning llms. arXiv preprint arXiv:2502.02542, 2025. - Kumar et al. [2024] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024. - Kumar et al. [20252] Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Salman Khan, and Fahad Shahbaz Khan. Llm post-training: A deep dive into reasoning large language models, 20252. - Kuo et al. [2025] Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Da-Cheng Juan, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking. arXiv preprint arXiv:2502.12893, 2025. - Lab [2025] EvolvingLMMs Lab. Open-r1-multimodal. https://github.com/EvolvingLMMs-Lab/open-r1-multimodal, February 2025. - Labs [2025] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation. https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation, January 2025. Accessed: 2025-01-22. - Labs et al. [2025] Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298, 2025. - Lai et al. [2025] Huiyuan Lai, Xiao Zhang, and Malvina Nissim. Multidimensional consistency improves reasoning in language models. arXiv preprint arXiv:2503.02670, 2025. - Lai et al. [2024] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024. - Lambert et al. [2024] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2024. - Lambert et al. [20242] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 20242. - Lampinen et al. [2022] Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.38. URL https://aclanthology.org/2022.findings-emnlp.38. - Lanchantin et al. [2025] Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization. arXiv preprint arXiv:2501.18101, 2025. - Le et al. [2025] Anh Duc Le, Tu Vu, Nam Le Hai, Nguyen Thi Ngoc Diep, Linh Ngo Van, Trung Le, and Thien Huu Nguyen. Cot2align: Cross-chain of thought distillation via optimal transport alignment for language models with different tokenizers. arXiv preprint arXiv:2502.16806, 2025. - Leang et al. [2024] Joshua Ong Jun Leang, Aryo Pradipta Gema, and Shay B Cohen. Comat: Chain of mathematically annotated thought improves mathematical reasoning. arXiv preprint arXiv:2410.10336, 2024. - Leang et al. [2025] Joshua Ong Jun Leang, Giwon Hong, Wenda Li, and Shay B Cohen. Theorem prover as a judge for synthetic data generation. arXiv preprint arXiv:2502.13137, 2025. - Lee et al. [2025] Byeongchan Lee, Jonghoon Lee, Dongyoung Kim, Jaehyung Kim, and Jinwoo Shin. Collaborative llm inference via planning for efficient reasoning. arXiv preprint arXiv:2506.11578, 2025. - Lee et al. [20252] Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, and Jihoon Tack. Revise: Learning to refine at test-time via intrinsic self-verification. arXiv preprint arXiv:2502.14565, 20252. - Lee and Hockenmaier [2025] Jinu Lee and Julia Hockenmaier. Evaluating step-by-step reasoning traces: A survey. arXiv preprint arXiv:2502.12289, 2025. - Lee et al. [2024] Jung Hyun Lee, June Yong Yang, Byeongho Heo, Dongyoon Han, and Kang Min Yoo. Token-supervised value models for enhancing mathematical reasoning capabilities of large language models. arXiv preprint arXiv:2407.12863, 2024. - Lee et al. [20253] Kuang-Huei Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. Evolving deeper llm thinking. arXiv preprint arXiv:2501.09891, 20253. - Lehnert et al. [2024] Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul McVay, Michael Rabbat, and Yuandong Tian. Beyond a*: Better planning with transformers via search dynamics bootstrapping. In First Conference on Language Modeling, July 2024. URL https://openreview.net/forum?id=SGoVIC0u0f. - Lei et al. [2024] Bin Lei, Yi Zhang, Shan Zuo, Ali Payani, and Caiwen Ding. MACM: Utilizing a multi-agent system for condition mining in solving complex mathematical problems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=VR2RdSxtzs. - Leng et al. [2025] Jixuan Leng, Cassandra A Cohen, Zhixian Zhang, Chenyan Xiong, and William W Cohen. Semi-structured llm reasoners can be rigorously audited. arXiv preprint arXiv:2505.24217, 2025. - Lerer et al. [2020] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7187–7194, Apr. 2020. doi: 10.1609/aaai.v34i05.6208. URL https://ojs.aaai.org/index.php/AAAI/article/view/6208. - Li et al. [2025] Belinda Z Li, Been Kim, and Zi Wang. Questbench: Can llms ask the right question to acquire information in reasoning tasks? arXiv preprint arXiv:2503.22674, 2025. - Li et al. [20252] Bingxuan Li, Yiwei Wang, Jiuxiang Gu, Kai-Wei Chang, and Nanyun Peng. Metal: A multi-agent framework for chart generation with test-time scaling. arXiv preprint arXiv:2502.17651, 20252. - Li et al. [2024] Bohan Li, Jiannan Guan, Longxu Dou, Yunlong Feng, Dingzirui Wang, Yang Xu, Enbo Wang, Qiguang Chen, Bichen Wang, Xiao Xu, et al. Can large language models understand you better? an mbti personality detection dataset aligned with population traits. arXiv preprint arXiv:2412.12510, 2024. - Li et al. [20242] Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 20242. - Li et al. [20253] Chen Li, Nazhou Liu, and Kai Yang. Adaptive group policy optimization: Towards stable training and token-efficient reasoning. arXiv preprint arXiv:2503.15952, 20253. - Li et al. [20254] Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang, Junyang Lin, et al. Cort: Code-integrated reasoning within thinking. arXiv preprint arXiv:2506.09820, 20254. - Li et al. [20255] Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools. arXiv preprint arXiv:2503.04625, 20255. - Li et al. [20243] Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28259–28277. PMLR, 21–27 Jul 20243. URL https://proceedings.mlr.press/v235/li24ar.html. - Li et al. [20256] Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542, 20256. - Li et al. [20257] Cheryl Li, Tianyuan Xu, and Yiwen Guo. Reasoning-as-logic-units: Scaling test-time reasoning in large language models through logic unit alignment. arXiv preprint arXiv:2502.07803, 20257. - Li et al. [20258] Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E Gonzalez, and Ion Stoica. S*: Test time scaling for code generation. arXiv preprint arXiv:2502.14382, 20258. - Li et al. [20259] Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G Patil, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. Llms can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 20259. - Li et al. [20244] Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594, 20244. - Li et al. [202510] Gengxu Li, Tingyu Xia, Yi Chang, and Yuan Wu. Length-controlled margin-based preference optimization without reference model. arXiv preprint arXiv:2502.14643, 202510. - Li et al. [20245] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579, 20245. - LI et al. [2024] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. https://huggingface.co/AI-MO/NuminaMath-CoT, 2024. - Li et al. [202511] Jia-Nan Li, Jian Guan, Wei Wu, and Rui Yan. Extended inductive reasoning for personalized preference inference from behavioral signals. arXiv preprint arXiv:2505.18071, 202511. - Li et al. [202512] Jiachun Li, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Rewarding curse: Analyze and mitigate reward modeling issues for llm reasoning. arXiv preprint arXiv:2503.05188, 202512. - Li et al. [2024] Jierui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Codetree: Agent-guided tree search for code generation with large language models. arXiv preprint arXiv:2411.04329, 2024. - Li et al. [202513] Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction. arXiv preprint arXiv:2502.07316, 202513. - Li [2024] Kaixin Li. Verified taco problems. https://huggingface.co/datasets/likaixin/TACO-verified, 2024. URL https://huggingface.co/datasets/likaixin/TACO-verified. - Li et al. [202514] Kechen Li, Wenqi Zhu, Coralia Cartis, Tianbo Ji, and Shiwei Liu. Sos1: O1 and r1-like reasoning llms are sum-of-square solvers. arXiv preprint arXiv:2502.20545, 202514. - Li et al. [20242] Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, et al. Chain of ideas: Revolutionizing research via novel idea development with llm agents. arXiv preprint arXiv:2410.13185, 20242. - Li et al. [202515] Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis) fitting: A survey of scaling laws. arXiv preprint arXiv:2502.18969, 202515. - Li et al. [2023] Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, and Tianyi Zhou. Reflection-tuning: Data recycling improves llm instruction-tuning. arXiv preprint arXiv:2310.11716, 2023. - Li et al. [20243] Ming Li, Yanhong Li, and Tianyi Zhou. What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective. arXiv preprint arXiv:2410.23743, 20243. - Li et al. [202516] Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy Chen, and Min-Yen Kan. Dna-eval: Enhancing large language model evaluation through decomposition and aggregation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2277–2290, January 202516. - Li et al. [202517] Moxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi, Zhuo Liu, Fuli Feng, and Tat-Seng Chua. Self-improvement towards pareto optimality: Mitigating preference conflicts in multi-objective alignment. arXiv preprint arXiv:2502.14354, 202517. - Li et al. [202518] Peiji Li, Kai Lv, Yunfan Shao, Yichuan Ma, Linyang Li, Xiaoqing Zheng, Xipeng Qiu, and Qipeng Guo. Fastmcts: A simple sampling strategy for data synthesis. arXiv preprint arXiv:2502.11476, 202518. - Li et al. [20244] Qingyao Li, Wei Xia, Kounianhua Du, Xinyi Dai, Ruiming Tang, Yasheng Wang, Yong Yu, and Weinan Zhang. Rethinkmcts: Refining erroneous thoughts in monte carlo tree search for code generation. arXiv preprint arXiv:2409.09584, 20244. - Li et al. [202519] Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding. Enhancing reasoning through process supervision with monte carlo tree search. In The First Workshop on Neural Reasoning and Mathematical Discovery at AAAI’2025, January 202519. URL https://openreview.net/forum?id=OupEEi1341. - Li et al. [202520] Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.09340, 202520. - Li et al. [20245] Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, et al. Combining induction and transduction for abstract reasoning. arXiv preprint arXiv:2411.02272, 20245. - Li and Li [2024] Wendi Li and Yixuan Li. Process reward model with q-value rankings. arXiv preprint arXiv:2410.11287, 2024. - Li et al. [20246] Wenjun Li, Changyu Chen, and Pradeep Varakantham. Unlocking large language model’s planning capabilities with maximum diversity fine-tuning. arXiv preprint arXiv:2406.10479, 20246. - Li and Qiu [2023] Xiaonan Li and Xipeng Qiu. MoT: Memory-of-thought enables ChatGPT to self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6354–6374, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.392. URL https://aclanthology.org/2023.emnlp-main.392/. - Li et al. [202521] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 202521. - Li [2025] Xinzhe Li. A survey on llm test-time compute via search: Tasks, llm profiling, search algorithms, and relevant frameworks. arXiv preprint arXiv:2501.10069, 2025. - Li et al. [202522] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886, 202522. - Li et al. [202523] Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, and Yu Cheng. From drafts to answers: Unlocking llm potential via aggregation fine-tuning. arXiv preprint arXiv:2501.11877, 202523. - Li [20252] Yang Li. Policy guided tree search for enhanced llm reasoning. arXiv preprint arXiv:2502.06813, 20252. - Li et al. [20247] Yang Li, Dong Du, Linfeng Song, Chen Li, Weikang Wang, Tao Yang, and Haitao Mi. Hunyuanprover: A scalable data synthesis framework and guided tree search for automated theorem proving. arXiv preprint arXiv:2412.20735, 20247. - Li et al. [202524] Yang Li, Youssef Emad, Karthik Padthe, Jack Lanchantin, Weizhe Yuan, Thao Nguyen, Jason Weston, Shang-Wen Li, Dong Wang, Ilia Kulikov, et al. Naturalthoughts: Selecting and distilling reasoning traces for general reasoning tasks. arXiv preprint arXiv:2507.01921, 202524. - Li et al. [20232] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315–5333, Toronto, Canada, July 20232. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.291. URL https://aclanthology.org/2023.acl-long.291/. - Li et al. [202525] Yiwei Li, Ji Zhang, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Jiayi Shi, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, et al. Revisiting self-consistency from dynamic distributional alignment perspective on answer aggregation. arXiv preprint arXiv:2502.19830, 202525. - Li et al. [2022] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814, 2022. - Li et al. [202526] Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921, 202526. - Li et al. [202527] Zheng Li, Qingxiu Dong, Jingyuan Ma, Di Zhang, and Zhifang Sui. Selfbudgeter: Adaptive token allocation for efficient llm reasoning. arXiv preprint arXiv:2505.11274, 202527. - Li et al. [20233] Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations, January 20233. URL https://openreview.net/pdf?id=3EWTEy9MTM. - Li et al. [20248] Zhiyuan Li, Dongnan Liu, Chaoyi Zhang, Heng Wang, Tengfei Xue, and Weidong Cai. Enhancing advanced visual reasoning ability of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1915–1929, Miami, Florida, USA, November 20248. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.114. URL https://aclanthology.org/2024.emnlp-main.114/. - Li et al. [202528] Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 202528. - Li et al. [202529] Zhongzhi Li, Ming-Liang Zhang, Pei-Jie Wang, Jian Xu, Rui-Song Zhang, Yin Fei, Zhi-Long Ji, Jin-Feng Bai, Zhen-Ru Pan, Jiaxin Zhang, and Cheng-Lin Liu. CMMaTH: A Chinese multi-modal math skill evaluation benchmark for foundation models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 2690–2726, Abu Dhabi, UAE, January 202529. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.184/. - Li et al. [202530] Zhuoqun Li, Haiyang Yu, Xuanang Chen, Hongyu Lin, Yaojie Lu, Fei Huang, Xianpei Han, Yongbin Li, and Le Sun. Deepsolution: Boosting complex engineering solution design via tree-based exploration and bi-point thinking. arXiv preprint arXiv:2502.20730, 202530. - Li et al. [202531] Zichao Li, Xueru Wen, Jie Lou, Yuqiu Ji, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. The devil is in the details: Tackling unimodal spurious correlations for generalizable multimodal reward models. In Forty-second International Conference on Machine Learning, 202531. URL https://openreview.net/forum?id=b0qRSUcQP7. - Li et al. [20249] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, May 20249. URL https://openreview.net/forum?id=Stn8hXkpe6. - Liang et al. [2025] Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model. arXiv preprint arXiv:2507.06892, 2025. - Liang et al. [20252] Jintao Liang, Gang Su, Huifeng Lin, You Wu, Rui Zhao, and Ziyue Li. Reasoning rag via system 1 or system 2: A survey on reasoning agentic retrieval-augmented generation for industry challenges. arXiv preprint arXiv:2506.10408, 20252. - Liang et al. [20253] Xiao Liang, Zhong-Zhi Li, Yeyun Gong, Yang Wang, Hengyuan Zhang, Yelong Shen, Ying Nian Wu, and Weizhu Chen. Sws: Self-aware weakness-driven problem synthesis in reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.08989, 20253. - Liang et al. [2024] Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, et al. Internal consistency and self-feedback in large language models: A survey. arXiv preprint arXiv:2407.14507, 2024. - Liao et al. [2025] Baohao Liao, Xinyi Chen, Sara Rajaee, Yuhui Xu, Christian Herold, Anders Søgaard, Maarten de Rijke, and Christof Monz. Lost at the beginning of reasoning. arXiv preprint arXiv:2506.22058, 2025. - Liao et al. [20252] Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient llm reasoning. arXiv preprint arXiv:2501.19324, 20252. - Liao et al. [20253] Huanxuan Liao, Shizhu He, Yupu Hao, Xiang Li, Yuanzhe Zhang, Jun Zhao, and Kang Liu. Skintern: Internalizing symbolic knowledge for distilling better cot capabilities into small language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3203–3221, January 20253. URL https://aclanthology.org/2025.coling-main.215.pdf. - Liao et al. [20254] Mengqi Liao, Xiangyu Xi, Ruinian Chen, Jia Leng, Yangen Hu, Ke Zeng, Shuai Liu, and Huaiyu Wan. Enhancing efficiency and exploration in reinforcement learning for llms. arXiv preprint arXiv:2505.18573, 20254. - Liao et al. [2024] Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output–a reproducible pipeline. arXiv preprint arXiv:2401.08190, 2024. - Liao et al. [20242] Weibin Liao, Xu Chu, and Yasha Wang. Tpo: Aligning large language models with multi-branch & multi-step preference trees. arXiv preprint arXiv:2410.12854, 20242. - Light et al. [2024] Jonathan Light, Min Cai, Weiqin Chen, Guanzhi Wang, Xiusi Chen, Wei Cheng, Yisong Yue, and Ziniu Hu. Strategist: Learning strategic skills by LLMs via bi-level tree search. In Automated Reinforcement Learning: Exploring Meta-Learning, AutoML, and LLMs, June 2024. URL https://openreview.net/forum?id=UHWBmZuJPF. - Light et al. [20242] Jonathan Light, Yue Wu, Yiyou Sun, Wenchao Yu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, Wei Cheng, et al. Scattered forest search: Smarter code space exploration with llms. arXiv preprint arXiv:2411.05010, 20242. - Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, January 2024. URL https://openreview.net/forum?id=v8L0pN6EOi. - Lin et al. [2025] Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01100, 2025. - Lin et al. [2024] Haohan Lin, Zhiqing Sun, Yiming Yang, and Sean Welleck. Lean-star: Learning to interleave thinking and proving. arXiv preprint arXiv:2407.10040, 2024. - Lin et al. [20252] Qingwen Lin, Boyan Xu, Guimin Hu, Zijian Li, Zhifeng Hao, Keli Zhang, and Ruichu Cai. Cmcts: A constrained monte carlo tree search framework for mathematical reasoning in large language model. arXiv preprint arXiv:2502.11169, 20252. - Lin et al. [20253] Qingwen Lin, Boyan Xu, Zijian Li, Zhifeng Hao, Keli Zhang, and Ruichu Cai. Leveraging constrained monte carlo tree search to generate reliable long chain-of-thought for mathematical reasoning. arXiv preprint arXiv:2502.11169, 20253. - Lin et al. [20254] Yen-Ting Lin, Di Jin, Tengyu Xu, Tianhao Wu, Sainbayar Sukhbaatar, Chen Zhu, Yun He, Yun-Nung Chen, Jason Weston, Yuandong Tian, et al. Step-kto: Optimizing mathematical reasoning through stepwise binary feedback. arXiv preprint arXiv:2501.10799, 20254. - Lin et al. [20255] Yujie Lin, Ante Wang, Moye Chen, Jingyao Liu, Hao Liu, Jinsong Su, and Xinyan Xiao. Investigating inference-time scaling for chain of multi-modal thought: A preliminary study. arXiv preprint arXiv:2502.11514, 20255. - Lin et al. [20242] Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. CriticBench: Benchmarking LLMs for critique-correct reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1552–1587, Bangkok, Thailand, August 20242. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.91. URL https://aclanthology.org/2024.findings-acl.91/. - Lin et al. [20243] Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhence llm’s reasoning capability. arXiv preprint arXiv:2411.19943, 20243. - Lin et al. [20256] Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02584, 20256. - Ling et al. [2025] Zehui Ling, Deshu Chen, Hongwei Zhang, Yifeng Jiao, Xin Guo, and Yuan Cheng. Fast on the easy, deep on the hard: Efficient reasoning via powered length penalty. arXiv preprint arXiv:2506.10446, 2025. - Ling et al. [2023] Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 36407–36433. Curran Associates, Inc., September 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/72393bd47a35f5b3bee4c609e7bba733-Paper-Conference.pdf. - Lippmann and Yang [2025] Philip Lippmann and Jie Yang. Style over substance: Distilled language models reason via stylistic replication. arXiv preprint arXiv:2504.01738, 2025. - Liu et al. [2024] Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Xiaoming Wang, Jiulong Shan, Meng Cao, and Lijie Wen. Direct large language model alignment through self-rewarding contrastive prompt distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9688–9712, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.523. URL https://aclanthology.org/2024.acl-long.523/. - Liu et al. [20242] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 20242. - Liu et al. [2023] Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241, 2023. - Liu et al. [2025] Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.24119, 2025. - Liu et al. [20243] Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 20243. - Liu et al. [20252] Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 20252. - Liu et al. [20253] Cong Liu, Zhong Wang, ShengYu Shen, Jialiang Peng, Xiaoli Zhang, ZhenDong Du, and YaFang Wang. The chinese dataset distilled from deepseek-r1-671b. https://huggingface.co/datasets/Congliu/Chinese-DeepSeek-R1-Distill-data-110k, 20253. - Liu et al. [20244] Dancheng Liu, Amir Nassereldine, Ziming Yang, Chenhui Xu, Yuting Hu, Jiajie Li, Utkarsh Kumar, Changjae Lee, Ruiyang Qin, Yiyu Shi, et al. Large language models have intrinsic self-correction ability. arXiv preprint arXiv:2406.15673, 20244. - Liu et al. [20254] Fan Liu, Wenshuo Chao, Naiqiang Tan, and Hao Liu. Bag of tricks for inference-time computation of llm reasoning. arXiv preprint arXiv:2502.07191, 20254. - Liu et al. [20245] Guanlin Liu, Kaixuan Ji, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. Enhancing multi-step reasoning abilities of language models through direct q-function optimization. arXiv preprint arXiv:2410.09302, 20245. - Liu et al. [20255] Hanbing Liu, Lang Cao, Yuanyi Ren, Mengyu Zhou, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. Bingo: Boosting efficient reasoning of llms via dynamic and significance-based reinforcement learning. arXiv preprint arXiv:2506.08125, 20255. - Liu et al. [20256] Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. Logical reasoning in large language models: A survey. arXiv preprint arXiv:2502.09100, 20256. - Liu et al. [20257] Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. Hoprag: Multi-hop reasoning for logic-aware retrieval-augmented generation. arXiv preprint arXiv:2502.12442, 20257. - Liu et al. [20246] Hongxuan Liu, Zhiyao Luo, and Tingting Zhu. Best of both worlds: Harmonizing LLM capabilities in decision-making and question-answering for treatment regimes. In Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond, 20246. URL https://openreview.net/forum?id=afu9qhp7md. - Liu et al. [20247] Jiacai Liu, Chaojie Wang, Chris Yuhao Liu, Liang Zeng, Rui Yan, Yiwen Sun, Yang Liu, and Yahui Zhou. Improving multi-step reasoning abilities of large language models with direct advantage policy optimization. arXiv preprint arXiv:2412.18279, 20247. - Liu et al. [20248] Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don’t throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. In First Conference on Language Modeling, July 20248. URL https://openreview.net/forum?id=kh9Zt2Ldmn. - Liu et al. [20249] Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making PPO even better: Value-guided monte-carlo tree search decoding, September 20249. URL https://openreview.net/forum?id=QaODpeRaOK. - Liu et al. [20258] Junnan Liu, Hongwei Liu, Linchen Xiao, Shudong Liu, Taolin Zhang, Zihan Ma, Songyang Zhang, and Kai Chen. Deciphering trajectory-aided llm reasoning: An optimization perspective. arXiv preprint arXiv:2505.19815, 20258. - Liu et al. [20259] Junnan Liu, Linhao Luo, Thuy-Trang Vu, and Gholamreza Haffari. Situatedthinker: Grounding llm reasoning with real-world through situated thinking. arXiv preprint arXiv:2505.19300, 20259. - Liu et al. [202510] Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, et al. Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. arXiv preprint arXiv:2505.19641, 202510. - Liu et al. [202511] Liping Liu, Chunhong Zhang, Likang Wu, Chuang Zhao, Zheng Hu, Ming He, and Jianping Fan. Instruct-of-reflection: Enhancing large language models iterative reflection capabilities via dynamic-meta instruction. arXiv preprint arXiv:2503.00902, 202511. - Liu et al. [202512] Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864, 202512. - Liu et al. [202513] Qiang Liu, Xinlong Chen, Yue Ding, Shizhen Xu, Shu Wu, and Liang Wang. Attention-guided self-reflection for zero-shot hallucination detection in large language models. arXiv preprint arXiv:2501.09997, 202513. - Liu et al. [202514] Qin Liu, Wenxuan Zhou, Nan Xu, James Y Huang, Fei Wang, Sheng Zhang, Hoifung Poon, and Muhao Chen. Metascale: Test-time scaling with evolving meta-thoughts. arXiv preprint arXiv:2503.13447, 202514. - Liu et al. [202515] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703, 202515. - Liu et al. [202516] Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot llms. arXiv preprint arXiv:2505.18610, 202516. - Liu et al. [202517] Wanlong Liu, Junxiao Xu, Fei Yu, Yukang Lin, Ke Ji, Wenyu Chen, Yan Xu, Yasheng Wang, Lifeng Shang, and Benyou Wang. Qfft, question-free fine-tuning for adaptive reasoning. arXiv preprint arXiv:2506.12860, 202517. - Liu et al. [202410] Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning. arXiv preprint arXiv:2412.17451, 202410. - Liu et al. [202518] Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint arXiv:2505.15612, 202518. - Liu et al. [202519] Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. Videomind: A chain-of-lora agent for long video reasoning. arXiv preprint arXiv:2503.13444, 202519. - Liu et al. [202520] Yongjiang Liu, Haoxi Li, Xiaosong Ma, Jie Zhang, and Song Guo. Think how to think: Mitigating overthinking with autonomous difficulty cognition in large reasoning models. arXiv preprint arXiv:2507.02663, 202520. - Liu et al. [202521] Yue Liu, Hongcheng Gao, Shengfang Zhai, Jun Xia, Tianyi Wu, Zhiwei Xue, Yulin Chen, Kenji Kawaguchi, Jiaheng Zhang, and Bryan Hooi. Guardreasoner: Towards reasoning-based llm safeguards. arXiv preprint arXiv:2501.18492, 202521. - Liu et al. [202522] Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Ruihan Gong, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. Efficient inference for large reasoning models: A survey. arXiv preprint arXiv:2503.23077, 202522. - Liu et al. [202523] Yuliang Liu, Junjie Lu, Zhaoling Chen, Chaofeng Qu, Jason Klein Liu, Chonghan Liu, Zefan Cai, Yunhui Xia, Li Zhao, Jiang Bian, et al. Adaptivestep: Automatically dividing reasoning step through model confidence. arXiv preprint arXiv:2502.13943, 202523. - Liu et al. [202524] Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, et al. Fin-r1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252, 202524. - Liu et al. [202525] Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, and Jun Wang. Othink-mr1: Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learning. arXiv preprint arXiv:2503.16081, 202525. - Liu et al. [202526] Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in r1-zero-like training — a pilot study. https://oatllm.notion.site/oat-zero, 202526. Notion Blog. - Liu et al. [202527] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 202527. - Liu et al. [202411] Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint arXiv:2412.15084, 202411. - Liu et al. [202528] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 202528. - Lobo et al. [2024] Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. On the impact of fine-tuning on chain-of-thought reasoning. arXiv preprint arXiv:2411.15382, 2024. - Lou et al. [2025] Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning. arXiv preprint arXiv:2505.11896, 2025. - Lu et al. [2025] Dakuan Lu, Xiaoyu Tan, Rui Xu, Tianchu Yao, Chao Qu, Wei Chu, Yinghui Xu, and Yuan Qi. Scp-116k: A high-quality problem-solution dataset and a generalized pipeline for automated extraction in the higher education science domain, 2025. - Lu et al. [20252] Haolang Lu, Yilian Liu, Jingxin Xu, Guoshun Nan, Yuanlong Yu, Zhican Chen, and Kun Wang. Auditing meta-cognitive hallucinations in reasoning large language models. arXiv preprint arXiv:2505.13143, 20252. - Lu et al. [2024] Jianqiao Lu, Zhiyang Dou, Hongru WANG, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. Autopsv: Automated process-supervised verifier. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 79935–79962. Curran Associates, Inc., December 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/9246aa822579d9b29a140ecdac36ad60-Paper-Conference.pdf. - Lu et al. [2022] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, November 2022. URL https://openreview.net/forum?id=HjwK-Tc_Bc. - Lu et al. [20242] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, January 20242. URL https://openreview.net/forum?id=KUNzEQMWU7. - Lu et al. [20253] Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502.11271, 20253. - Lu et al. [20254] Rubing Lu, João Sedoc, and Arun Sundararajan. Reasoning and the trusting behavior of deepseek and gpt: An experiment revealing hidden fault lines in large language models. arXiv preprint arXiv:2502.12825, 20254. - Lu et al. [20255] Wenquan Lu, Yuechuan Yang, Kyle Lee, Yanshu Li, and Enqi Liu. Latent chain-of-thought? decoding the depth-recurrent transformer. arXiv preprint arXiv:2507.02199, 20255. - Lu et al. [20256] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 20256. - Lu et al. [20243] Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. arXiv preprint arXiv:2402.16352, 20243. - Luo et al. [2023] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023. - Luo et al. [2025] Hanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li, Guibin Zhang, Kun Wang, Tongliang Liu, and Hanan Salam. Agentauditor: Human-level safety and security evaluation for llm agents. arXiv preprint arXiv:2506.00641, 2025. - Luo et al. [20252] Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning. arXiv preprint arXiv:2501.12570, 20252. - Luo et al. [2024] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024. - Luo et al. [20253] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, February 20253. URL https://github.com/agentica-project/rllm. Notion Blog. - Luo et al. [20254] Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics. arXiv preprint arXiv:2501.04686, 20254. - Luo et al. [20242] Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Libo Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Python is not always the best choice: Embracing multilingual program of thoughts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7185–7212, Miami, Florida, USA, November 20242. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.408. URL https://aclanthology.org/2024.emnlp-main.408/. - Luo et al. [20255] Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, and Bo Zheng. Deconstructing long chain-of-thought: A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503.16385, 20255. - Lyu et al. [2025] Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781, 2025. - Lyu et al. [2023] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi, editors, Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 305–329, Nusa Dua, Bali, November 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.ijcnlp-main.20. URL https://aclanthology.org/2023.ijcnlp-main.20/. - Lyzhov et al. [2020] Alexander Lyzhov, Yuliya Molchanova, Arsenii Ashukha, Dmitry Molchanov, and Dmitry Vetrov. Greedy policy search: A simple baseline for learnable test-time augmentation. In Jonas Peters and David Sontag, editors, Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pages 1308–1317. PMLR, 03–06 Aug 2020. URL https://proceedings.mlr.press/v124/lyzhov20a.html. - Ma et al. [2025] Jingyuan Ma, Rui Li, Zheng Li, Junfeng Liu, Lei Sha, and Zhifang Sui. Hauntattack: When attack follows reasoning as a shadow. arXiv preprint arXiv:2506.07031, 2025. - Ma et al. [20252] Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527, 20252. - Ma et al. [20253] Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 20253. - Ma et al. [2023] Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let’s reward step by step: Step-level reward model as the navigators for reasoning. arXiv preprint arXiv:2310.10080, 2023. - Ma et al. [20254] Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S 2 r: Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502.12853, 20254. - Ma et al. [20255] Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length-compressible chain-of-thought tuning. arXiv preprint arXiv:2502.09601, 20255. - Ma et al. [20256] Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652, 20256. - Ma et al. [20257] Xuetao Ma, Wenbin Jiang, and Hua Huang. Problem-solving logic guided curriculum in-context learning for llms complex reasoning. arXiv preprint arXiv:2502.15401, 20257. - Ma et al. [20258] Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, and Pengfei Liu. Rethinking rl scaling for vision language models: A transparent, from-scratch framework and comprehensive evaluation scheme. arXiv preprint arXiv:2504.02587, 20258. - Ma et al. [2024] Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, and Weiqi Luo. What are step-level reward models rewarding? counterintuitive findings from mcts-boosted mathematical reasoning. arXiv preprint arXiv:2412.15904, 2024. - Ma et al. [20259] Zexiong Ma, Chao Peng, Pengfei Gao, Xiangxin Meng, Yanzhen Zou, and Bing Xie. Sorft: Issue resolving with subtask-oriented reinforced fine-tuning. arXiv preprint arXiv:2502.20127, 20259. - Ma et al. [202510] Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. arXiv preprint arXiv:2501.01054, 202510. - Ma et al. [202511] Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, and Xie Chen. Audio-cot: Exploring chain-of-thought reasoning in large audio language model. arXiv preprint arXiv:2501.07246, 202511. - Madaan et al. [2023] Aman Madaan, Katherine Hermann, and Amir Yazdanbakhsh. What makes chain-of-thought prompting effective? a counterfactual study. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1448–1535, Singapore, December 2023. URL https://aclanthology.org/2023.findings-emnlp.101.pdf. - Madaan et al. [20232] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 46534–46594. Curran Associates, Inc., March 20232. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/91edff07232fb1b55a505a9e9f6c0ff3-Paper-Conference.pdf. - Madhusudhan et al. [2025] Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Jash Mehta, and Toby Liang. Millions scale dataset distilled from r1-32b. https://huggingface.co/datasets/ServiceNow-AI/R1-Distill-SFT, February 2025. - Mahdavi et al. [2025] Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal, and Renjie Liao. Leveraging online olympiad-level math problems for llms training and contamination-resistant evaluation. arXiv preprint arXiv:2501.14275, 2025. - Materzok [2025] Tobias Materzok. Cos (m+ o) s: Curiosity and rl-enhanced mcts for exploring story space via language models. arXiv preprint arXiv:2501.17104, 2025. - Mattern et al. [2025] Justus Mattern, Sami Jaghouar, Manveer Basra, Jannik Straube, Matthew Di Ferrante, Felix Gabriel, Jack Min Ong, Vincent Weisser, and Johannes Hagemann. Synthetic-1: Two million collaboratively generated reasoning traces from deepseek-r1, 2025. URL https://www.primeintellect.ai/blog/synthetic-1-release. - McAleese et al. [2024] Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215, 2024. - McCoy et al. [2024] R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. When a language model is optimized for reasoning, does it still show embers of autoregression? an analysis of openai o1. arXiv preprint arXiv:2410.01792, 2024. - Mei et al. [2025] Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, and Shenghua Liu. A survey of context engineering for large language models. arXiv preprint arXiv:2507.13334, 2025. - Meng et al. [2025] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025. - Merrill and Sabharwal [2023] William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, January 2023. URL https://openreview.net/pdf?id=CDmerQ37Zs. - Miao et al. [2024] Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning. In The Twelfth International Conference on Learning Representations, January 2024. URL https://openreview.net/forum?id=pTHfApDakA. - Min et al. [2024] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413, 2024. - Mirzadeh et al. [2025] Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations, January 2025. URL https://openreview.net/forum?id=AjXkRZIvjB. - Mishra et al. [2025] Prakamya Mishra, Jiang Liu, Jialian Wu, Xiaodong Yu, Zicheng Liu, and Emad Barsoum. Ttt-bench: A benchmark for evaluating reasoning ability with simple and novel tic-tac-toe-style games. arXiv preprint arXiv:2506.10209, 2025. - Mitra et al. [2024] Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024. - Mitra et al. [20242] Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 20242. - Mitra and Ulukus [2025] Purbesh Mitra and Sennur Ulukus. Motif: Modular thinking via reinforcement fine-tuning in llms. arXiv preprint arXiv:2507.02851, 2025. - Mo and Xin [2024] Shentong Mo and Miao Xin. Tree of uncertain thoughts reasoning for large language models. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12742–12746, April 2024. doi: 10.1109/ICASSP48485.2024.10448355. URL https://ieeexplore.ieee.org/document/10448355. - Mondorf and Plank [2024] Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models–a survey. arXiv preprint arXiv:2404.01869, 2024. - Morishita et al. [2024] Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reasoning capabilities of llms via principled synthetic logic corpus. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 73572–73604. Curran Associates, Inc., September 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/8678da90126aa58326b2fc0254b33a8c-Paper-Conference.pdf. - Mu et al. [2025] Yongyu Mu, Jiali Zeng, Bei Li, Xinyan Guan, Fandong Meng, Jie Zhou, Tong Xiao, and Jingbo Zhu. Dissecting long reasoning models: An empirical study. arXiv preprint arXiv:2506.04913, 2025. - Muennighoff et al. [2025] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025. - Munkhbat et al. [2025] Tergel Munkhbat, Namgyu Ho, Seohyun Kim, Yongjin Yang, Yujin Kim, and Se-Young Yun. Self-training elicits concise reasoning in large language models. arXiv preprint arXiv:2502.20122, 2025. - Nath et al. [2025] Vaskar Nath, Pranav Raja, Claire Yoon, and Sean Hendryx. Toolcomp: A multi-tool reasoning & process supervision benchmark. arXiv preprint arXiv:2501.01290, 2025. - Nayab et al. [2024] Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on llm reasoning and cost. arXiv preprint arXiv:2407.19825, 2024. - Ni et al. [2023] Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Yih, Sida Wang, and Xi Victoria Lin. LEVER: Learning to verify language-to-code generation with execution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26106–26128. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/ni23b.html. - Ni et al. [2024] Ziyi Ni, Yifan Li, Ning Yang, Dou Shen, Pin Lv, and Daxiang Dong. Tree-of-code: A tree-structured exploring framework for end-to-end code generation and execution in complex task handling. arXiv preprint arXiv:2412.15305, 2024. - Nie et al. [2024] Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024. - Ning et al. [2025] Yansong Ning, Wei Li, Jun Fang, Naiqiang Tan, and Hao Liu. Not all thoughts are generated equal: Efficient llm reasoning via multi-turn reinforcement learning. arXiv preprint arXiv:2505.11827, 2025. - Nori et al. [2024] Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, and Eric Horvitz. From medprompt to o1: Exploration of run-time strategies for medical challenge problems and beyond. arXiv preprint arXiv:2411.03590, 2024. - Nye et al. [2022] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop, March 2022. URL https://openreview.net/forum?id=HBlx2idbkbq. - o1 Team [2024] Skywork o1 Team. Skywork-o1 open series. https://huggingface.co/Skywork, November 2024. - OpenCompass [2025] OpenCompass. Aime 2025. https://huggingface.co/datasets/opencompass/AIME2025, February 2025. - Ou et al. [2025] Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, and Huajun Chen. How do llms acquire new knowledge? a knowledge circuits perspective on continual pre-training. arXiv preprint arXiv:2502.11196, 2025. - Pan et al. [2022] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. - Pan et al. [2024] Jiabao Pan, Yan Zhang, Chen Zhang, Zuozhu Liu, Hongwei Wang, and Haizhou Li. DynaThink: Fast or slow? a dynamic decision-making framework for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14686–14695, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.814. URL https://aclanthology.org/2024.emnlp-main.814/. - Pan et al. [2025] Jianfeng Pan, Senyou Deng, and Shaomang Huang. Coat: Chain-of-associated-thoughts framework for enhancing large language models reasoning. arXiv preprint arXiv:2502.02390, 2025. - Pan et al. [20252] Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 20252. Accessed: 2025-01-24. - Pan et al. [20253] Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. arXiv preprint arXiv:2502.19634, 20253. - Pan et al. [2023] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023. - Pan et al. [20254] Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. The hidden dimensions of llm alignment: A multi-dimensional safety analysis. arXiv preprint arXiv:2502.09674, 20254. - Pan et al. [20255] Zhihong Pan, Kai Zhang, Yuze Zhao, and Yupeng Han. Route to reason: Adaptive routing for llm and reasoning strategy selection. arXiv preprint arXiv:2505.19435, 20255. - Pang et al. [2025] Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. Bolt: Bootstrap long chain-of-thought in language models without distillation. arXiv preprint arXiv:2502.03860, 2025. - Pang et al. [2024] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 116617–116637. Curran Associates, Inc., September 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/d37c9ad425fe5b65304d500c6edcba00-Paper-Conference.pdf. - Parashar et al. [2025] Shubham Parashar, Blake Olson, Sambhav Khurana, Eric Li, Hongyi Ling, James Caverlee, and Shuiwang Ji. Inference-time computations for llm reasoning and planning: A benchmark and insights. arXiv preprint arXiv:2502.12521, 2025. - Park et al. [2025] Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo-Kyung Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning. arXiv preprint arXiv:2502.18439, 2025. - Park et al. [2024] Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024. - Park et al. [20242] Sungjin Park, Xiao Liu, Yeyun Gong, and Edward Choi. Ensembling large language models with process reward-guided tree search for better complex reasoning. arXiv preprint arXiv:2412.15797, 20242. - Parmar and Govindarajulu [2025] Manojkumar Parmar and Yuvaraj Govindarajulu. Challenges in ensuring ai safety in deepseek-r1 models: The shortcomings of reinforcement learning strategies. arXiv preprint arXiv:2501.17030, 2025. - Patil [2025] Avinash Patil. Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2502.03671, 2025. - Patil and Gedhu [2025] Avinash Patil and Amardeep Kour Gedhu. Cognitive-mental-llm: Leveraging reasoning in large language models for mental health prediction via online text. arXiv preprint arXiv:2503.10095, 2025. - Paul et al. [2024] Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. REFINER: Reasoning feedback on intermediate representations. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1100–1126, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.eacl-long.67/. - Payoungkhamdee et al. [2025] Patomporn Payoungkhamdee, Pume Tuchinda, Jinheon Baek, Samuel Cahyawijaya, Can Udomcharoenchaikit, Potsawee Manakul, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, and Sarana Nutanong. Towards better understanding of program-of-thought reasoning in cross-lingual and multilingual environments. arXiv preprint arXiv:2502.17956, 2025. - Peng et al. [2025] Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Yishan Li, Yukun Yan, Shuo Wang, Zhiyuan Liu, Yu Gu, Minghe Yu, Ge Yu, et al. Learning to route queries across knowledge bases for step-wise retrieval-augmented reasoning. arXiv preprint arXiv:2505.22095, 2025. - Peng et al. [20252] Dengyun Peng, Yuhang Zhou, Qiguang Chen, Jinhao Liu, Jingjing Chen, and Libo Qin. Dlpo: Towards a robust, efficient, and generalizable prompt optimization framework from a deep-learning perspective. arXiv preprint arXiv:2503.13413, 20252. - Peng et al. [20253] Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems. arXiv preprint arXiv:2502.19328, 20253. - Peng et al. [20254] Keqin Peng, Liang Ding, Yuanxin Ouyang, Meng Fang, and Dacheng Tao. Revisiting overthinking in long chain-of-thought from the perspective of self-doubt. arXiv preprint arXiv:2505.23480, 20254. - Peng et al. [20255] Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. Rewarding graph reasoning process makes llms more generalized reasoners. arXiv preprint arXiv:2503.00845, 20255. - Peng et al. [20256] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 20256. - Petrov et al. [2025] Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934, 2025. - Pfister and Jud [2025] Rolf Pfister and Hansueli Jud. Understanding and benchmarking artificial intelligence: Openai’s o3 is not agi. arXiv preprint arXiv:2501.07458, 2025. - Pham et al. [2025] Quang Hieu Pham, Thuy Duong Nguyen, Tung Pham, Anh Tuan Luu, and Dat Quoc Nguyen. Clozemath: Improving mathematical reasoning in language models by learning to fill equations. arXiv preprint arXiv:2506.03763, 2025. - Pham et al. [20252] Thinh Pham, Nguyen Nguyen, Pratibha Zunjare, Weiyuan Chen, Yu-Min Tseng, and Tu Vu. Sealqa: Raising the bar for reasoning in search-augmented language models. arXiv preprint arXiv:2506.01062, 20252. - Phan et al. [2025] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025. - Plaat et al. [2024] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511, 2024. - Poesia et al. [2024] Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah Goodman. Certified deductive reasoning with language models. Transactions on Machine Learning Research, May 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=yXnwrs2Tl6. - Polu and Sutskever [2020] Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020. - Prasad et al. [2023] Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. ReCEval: Evaluating reasoning chains via correctness and informativeness. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10066–10086, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.622. URL https://aclanthology.org/2023.emnlp-main.622/. - Prasad et al. [2024] Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-needed decomposition and planning with language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 4226–4252, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.264. URL https://aclanthology.org/2024.findings-naacl.264/. - Pricope [2025] Tidor-Vlad Pricope. Hardml: A benchmark for evaluating data science and machine learning knowledge and reasoning in ai. arXiv preprint arXiv:2501.15627, 2025. - Prystawski et al. [2023] Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 70926–70947. Curran Associates, Inc., September 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/e0af79ad53a336b4c4b4f7e2a68eb609-Paper-Conference.pdf. - Puerta-Merino et al. [2025] Israel Puerta-Merino, Carlos Núñez-Molina, Pablo Mesejo, and Juan Fernández-Olivares. A roadmap to guide the integration of llms in hierarchical planning. arXiv preprint arXiv:2501.08068, 2025. - Puerto et al. [2024] Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, and Iryna Gurevych. Fine-tuning with divergent chains of thought boosts reasoning through self-correction in language models. arXiv preprint arXiv:2407.03181, 2024. - Puri et al. [2025] Isha Puri, Shivchander Sudalairaj, Guangxuan Xu, Kai Xu, and Akash Srivastava. A probabilistic inference approach to inference-time scaling of llms using particle-based monte carlo methods. arXiv preprint arXiv:2502.01618, 2025. - Putta et al. [2024] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199, 2024. - Qi et al. [2025] Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization. arXiv preprint arXiv:2505.13438, 2025. - Qi et al. [2024] Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195, 2024. - Qian and Liu [2025] Hongjin Qian and Zheng Liu. Scent of knowledge: Optimizing search-enhanced reasoning with information foraging. arXiv preprint arXiv:2505.09316, 2025. - Qin et al. [2023] Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695–2709, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.163. URL https://aclanthology.org/2023.emnlp-main.163/. - Qin et al. [2024] Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth exploration. arXiv preprint arXiv:2410.20482, 2024. - Qin et al. [20242] Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819, 20242. - Qin et al. [20243] Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925, 20243. - Qin et al. [2025] Libo Qin, Qiguang Chen, Jingxuan Zhou, Jin Wang, Hao Fei, Wanxiang Che, and Min Li. Divide-solve-combine: An interpretable and accurate prompting framework for zero-shot multi-intent detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25038–25046, 2025. - Qin et al. [20252] Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and S Yu Philip. A survey of multilingual large language models. Patterns, 6(1), January 20252. URL https://www.cell.com/patterns/fulltext/S2666-3899(24)00290-3. - Qin et al. [20244] Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982, 20244. - Qin et al. [20253] Yulei Qin, Gang Li, Zongyi Li, Zihan Xu, Yuchen Shi, Zhekai Lin, Xiao Cui, Ke Li, and Xing Sun. Incentivizing reasoning for advanced instruction-following of large language models. arXiv preprint arXiv:2506.01413, 20253. - Qiu et al. [2024] Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling. arXiv preprint arXiv:2410.16033, 2024. - Qu et al. [2025] Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614, 2025. - Qu et al. [2024] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=DRC9pZwBwR. - Qu et al. [20252] Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement finetuning. In Workshop on Reasoning and Planning for Large Language Models, March 20252. URL https://openreview.net/forum?id=WGz4ytjo1h. - Rabby et al. [2024] Gollam Rabby, Farhana Keya, Parvez Zamil, and Sören Auer. Mc-nest–enhancing mathematical reasoning in large language models with a monte carlo nash equilibrium self-refine tree. arXiv preprint arXiv:2411.15645, 2024. - Radha and Goktas [2025] Santosh Kumar Radha and Oktay Goktas. On the reasoning capacity of ai models and how to quantify it. arXiv preprint arXiv:2501.13833, 2025. - Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. URL https://openreview.net/pdf?id=HPuSIXJaa9. - Rai and Yao [2024] Daking Rai and Ziyu Yao. An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7174–7193, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.387. URL https://aclanthology.org/2024.acl-long.387/. - Ranaldi et al. [2024] Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, Elena Sofia Ruzzetti, and Fabio Massimo Zanzotto. A tree-of-thoughts to broaden multi-step reasoning across languages. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 1229–1241, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.78. URL https://aclanthology.org/2024.findings-naacl.78/. - Ranaldi et al. [2025] Leonardo Ranaldi, Marco Valentino, Alexander Polonsky, and Andrè Freitas. Improving chain-of-thought reasoning via quasi-symbolic abstractions. arXiv preprint arXiv:2502.12616, 2025. - Raza and Milic-Frayling [2025] Mohammad Raza and Natasa Milic-Frayling. Instantiation-based formalization of logical reasoning tasks using language models and logical solvers. arXiv preprint arXiv:2501.16961, 2025. - Razghandi et al. [2025] Ali Razghandi, Seyed Mohammad Hadi Hosseini, and Mahdieh Soleymani Baghshah. Cer: Confidence enhanced reasoning in llms. arXiv preprint arXiv:2502.14634, 2025. - Rein et al. [2024] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, July 2024. URL https://openreview.net/forum?id=Ti67584b98. - Renze and Guven [2024] Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682, 2024. - Roziere et al. [2023] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. - Ruan et al. [2025] Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts. arXiv preprint arXiv:2503.18866, 2025. - Saad-Falcon et al. [2024] Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests. arXiv preprint arXiv:2412.13091, 2024. - Sadr et al. [2025] Nikta Gohari Sadr, Sangmitra Madhusudan, and Ali Emami. Think or step-by-step? unzipping the black box in zero-shot prompts. arXiv preprint arXiv:2502.03418, 2025. - Saha et al. [2025] Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-llm-as-a-judge. arXiv preprint arXiv:2501.18099, 2025. - Sauhandikaa et al. [2024] S Sauhandikaa, R Bhagavath Narenthranath, and R Sathya Bama Krishna. Explainable ai in large language models: A review. In 2024 International Conference on Emerging Research in Computational Science (ICERCS), pages 1–6. IEEE, 2024. URL http://ieeexplore.ieee.org/abstract/document/10895578. - Saunders et al. [2022] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022. - Saunshi et al. [2025] Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025. - Schöne et al. [2025] Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. Implicit language models are RNNs: Balancing parallelization and expressivity. In Forty-second International Conference on Machine Learning, May 2025. URL https://openreview.net/forum?id=5EbiopWH6e. - Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. - Seed et al. [2025] ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025. - Setlur et al. [2024] Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 43000–43031. Curran Associates, Inc., September 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/4b77d5b896c321a29277524a98a50215-Paper-Conference.pdf. - Setlur et al. [2025] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, January 2025. URL https://openreview.net/forum?id=A6Y7AqlzLW. - Setlur et al. [20252] Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118, 20252. - Setlur et al. [20253] Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms. arXiv preprint arXiv:2506.09026, 20253. - Shang et al. [2024] Yu Shang, Yu Li, Fengli Xu, and Yong Li. Synergy-of-thoughts: Eliciting efficient reasoning in hybrid language models. arXiv preprint arXiv:2402.02563, 2024. - Shao et al. [2025] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947, 2025. - Shao et al. [20252] Wenqi Shao, Qiaosheng Zhang, Lingxiao Du, Xiangyan Liu, and Fanqing Meng. R1-multimodal-journey. https://github.com/FanqingM/R1-Multimodal-Journey, February 20252. - Shao et al. [2023] Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating chain-of-thought demonstrations for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30706–30775. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/shao23a.html. - Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. - She et al. [2025] Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R-prm: Reasoning-driven process reward modeling. arXiv preprint arXiv:2503.21295, 2025. - Shen et al. [2025] Haozhan Shen, Zilun Zhang, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-r1: A stable and generalizable r1-style large vision-language model. https://github.com/om-ai-lab/VLM-R1, February 2025. - Shen et al. [20252] Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain-of-action-thought enhances llm reasoning via autoregressive search. arXiv preprint arXiv:2502.02508, 20252. - Shen et al. [20253] Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. Efficient reasoning with hidden thinking. arXiv preprint arXiv:2501.19201, 20253. - Shen et al. [20254] Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: Difficulty-adaptive slow-thinking for large reasoning models. arXiv preprint arXiv:2503.04472, 20254. - Shen et al. [20255] Yifan Shen, Yuanzhe Liu, Jingyuan Zhu, Xu Cao, Xiaofeng Zhang, Yixiao He, Wenming Ye, James Matthew Rehg, and Ismini Lourentzou. Fine-grained preference optimization improves spatial reasoning in vlms. arXiv preprint arXiv:2506.21656, 20255. - Sheng et al. [2025] Leheng Sheng, An Zhang, Zijian Wu, Weixiang Zhao, Changshuo Shen, Yi Zhang, Xiang Wang, and Tat-Seng Chua. On reasoning strength planning in large reasoning models. arXiv preprint arXiv:2506.08390, 2025. - Shi et al. [2025] Hengyu Shi, Junhao Su, Huansheng Ning, Xiaoming Wei, and Jialin Gao. Layoutcot: Unleashing the deep reasoning potential of large language models for layout generation. arXiv preprint arXiv:2504.10829, 2025. - Shi et al. [20252] Junhao Shi, Zhaoye Fei, Siyin Wang, Qipeng Guo, Jingjing Gong, and Xipeng QIu. World-aware planning narratives enhance large vision-language model planner. arXiv preprint arXiv:2506.21230, 20252. - Shi et al. [20253] Wenhao Shi, Zhiqiang Hu, Yi Bin, Yang Yang, See-Kiong Ng, and Heng Tao Shen. Multimodal mathematical reasoning with diverse solving perspective. arXiv preprint arXiv:2507.02804, 20253. - Shinn et al. [2023] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc., December 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf. - Shrestha et al. [2025] Safal Shrestha, Minwu Kim, and Keith Ross. Mathematical reasoning in large language models: Assessing logical and arithmetic errors across wide numerical ranges. arXiv preprint arXiv:2502.08680, 2025. - Shum et al. [2023] Kashun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of-thought from labeled data. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12113–12139, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.811. URL https://aclanthology.org/2023.findings-emnlp.811/. - Si et al. [2024] Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024. - Silver et al. [2025] Sam Silver, Jimin Sun, Ivan Zhang, Sara Hooker, and Eddie Kim. Language models can perform single-utterance self-correction of perturbed reasoning. arXiv preprint arXiv:2506.15894, 2025. - Singh et al. [2024] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, April 2024. URL https://openreview.net/pdf?id=lNAyUngGFK. - Skean et al. [2025] Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025. - Snell et al. [2024] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. - Song et al. [2025] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025. - Song et al. [20252] Jiwon Song, Dongwon Jo, Yulhwa Kim, and Jae-Joon Kim. Reasoning path compression: Compressing generation trajectories for efficient llm reasoning. arXiv preprint arXiv:2505.13866, 20252. - Song et al. [20253] Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124, 20253. - Song et al. [20254] Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with stage-wise context scaling for efficient training r1-like reasoning models. arXiv preprint arXiv:2503.17287, 20254. - Song et al. [20255] Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, and Sravan Babu Bodapati. Accelerated test-time scaling with model-free speculative sampling. arXiv preprint arXiv:2506.04708, 20255. - Song et al. [20256] Xiaoshuai Song, Yanan Wu, Weixun Wang, Jiaheng Liu, Wenbo Su, and Bo Zheng. Progco: Program helps self-correction of large language models. arXiv preprint arXiv:2501.01264, 20256. - Sprague et al. [2024] Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183, 2024. - Sprague et al. [20242] Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, January 20242. URL https://openreview.net/forum?id=jenyYQzue1. - Srivastava et al. [2025] Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. Towards reasoning ability of small language models. arXiv preprint arXiv:2502.11569, 2025. - Srivastava and Aggarwal [2025] Saksham Sahai Srivastava and Vaneet Aggarwal. A technical survey of reinforcement learning techniques for large language models. arXiv preprint arXiv:2507.04136, 2025. - Srivastava and Gandhi [2024] Saksham Sahai Srivastava and Ashutosh Gandhi. Mathdivide: Improved mathematical reasoning by large language models. arXiv preprint arXiv:2405.13004, 2024. - Stechly et al. [2024] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness? an analysis of cot in planning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=kPBEAZU5Nm. - Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc., December 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf. - Stoisser et al. [2025] Josefa Lia Stoisser, Marc Boubnovski Martell, and Julien Fauqueur. Sparks of tabular reasoning via text2sql reinforcement learning. arXiv preprint arXiv:2505.00016, 2025. - Su et al. [2024] DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng. Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces. arXiv preprint arXiv:2410.09918, 2024. - Su and Cardie [2025] Jinyan Su and Claire Cardie. Thinking fast and right: Balancing accuracy and reasoning length with adaptive rewards. arXiv preprint arXiv:2505.18298, 2025. - Su et al. [2025] Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. - Su et al. [20252] Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, et al. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers. arXiv preprint arXiv:2506.23918, 20252. - Sun et al. [2025] Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. In The Thirteenth International Conference on Learning Representations, January 2025. URL https://openreview.net/forum?id=ncCuiD3KJQ. - Sun et al. [2023] Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models. arXiv preprint arXiv:2312.11562, 2023. - Sun et al. [20252] Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang, Zenan Zhou, and Wentao Zhang. Mm-verify: Enhancing multimodal reasoning with chain-of-thought verification. arXiv preprint arXiv:2502.13383, 20252. - Sun et al. [20253] Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Robert Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, Lingpeng Kong, and Zhiyong Wu. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. In ICML 2025 Workshop on Computer Use Agents, June 20253. URL https://openreview.net/forum?id=CTtuhMeU5e. - Sun et al. [20254] Shengyang Sun, Yian Zhang, Alexander Bukharin, David Mosallanezhad, Jiaqi Zeng, Soumye Singhal, Gerald Shen, Adi Renduchintala, Tugrul Konuk, Yi Dong, et al. Reward-aware preference optimization: A unified mathematical framework for model alignment. arXiv preprint arXiv:2502.00203, 20254. - Sun et al. [20255] Wei Sun, Qianlong Du, Fuwei Cui, and Jiajun Zhang. An efficient and precise training data construction framework for process-supervised reward model in mathematical reasoning. arXiv preprint arXiv:2503.02382, 20255. - Sun et al. [20256] Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving data efficiency for llm reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. arXiv preprint arXiv:2506.05316, 20256. - Sun et al. [20257] Yuhong Sun, Zhangyue Yin, Xuanjing Huang, Xipeng Qiu, and Hui Zhao. Error classification of large language models on math word problems: A dynamically adaptive framework. arXiv preprint arXiv:2501.15581, 20257. - Sun et al. [20258] Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Song Yang, and Han Li. Rearter: Retrieval-augmented reasoning with trustworthy process rewarding. arXiv preprint arXiv:2501.07861, 20258. - Sutton et al. [1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, November 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf. - Suzgun et al. [2023] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. URL https://aclanthology.org/2023.findings-acl.824/. - Tack et al. [2025] Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts. arXiv preprint arXiv:2502.08524, 2025. - Tan et al. [2025] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752, 2025. - Tan [2023] Juanhe (TJ) Tan. Causal abstraction for chain-of-thought reasoning in arithmetic word problems. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 155–168, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.blackboxnlp-1.12. URL https://aclanthology.org/2023.blackboxnlp-1.12. - Tan et al. [2024] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784, 2024. - Tan et al. [20252] Xiaoyu Tan, Tianchu Yao, Chao Qu, Bin Li, Minghao Yang, Dakuan Lu, Haozhe Wang, Xihe Qiu, Wei Chu, Yinghui Xu, et al. Aurora: Automated training framework of universal process reward models via ensemble prompting and reverse verification. arXiv preprint arXiv:2502.11520, 20252. - Tang et al. [2025] Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. Lego-puzzles: How good are mllms at multi-step spatial reasoning? arXiv preprint arXiv:2503.19990, 2025. - Tang et al. [20252] Yihong Tang, Kehai Chen, Muyun Yang, Zhengyu Niu, Jing Li, Tiejun Zhao, and Min Zhang. Thinking in character: Advancing role-playing agents with role-aware reasoning. arXiv preprint arXiv:2506.01748, 20252. - Tang et al. [20253] Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, et al. Enabling scalable oversight via self-evolving critic. arXiv preprint arXiv:2501.05727, 20253. - Tang et al. [20254] Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, et al. Realcritic: Towards effectiveness-driven evaluation of language model critiques. arXiv preprint arXiv:2501.14492, 20254. - Tanneru et al. [2024] Sree Harsha Tanneru, Dan Ley, Chirag Agarwal, and Himabindu Lakkaraju. On the hardness of faithful chain-of-thought reasoning in large language models. arXiv preprint arXiv:2406.10625, 2024. - Taubenfeld et al. [2025] Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in llms. arXiv preprint arXiv:2502.06233, 2025. - Team [2025] DolphinR1 Team. Dolphin R1. https://huggingface.co/datasets/cognitivecomputations/dolphin-r1, February 2025. - Team [20252] Fancy-MLLM Team. R1 Onevision. https://huggingface.co/datasets/Fancy-MLLM/R1-Onevision, February 20252. - Team et al. [2024] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. - Team et al. [20242] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 20242. - Team [20253] Huggingface Team. Open r1. https://github.com/huggingface/open-r1, January 20253. - Team et al. [2025] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. - Team [20254] NovaSky Team. Think less, achieve more: Cut reasoning costs by 50% without sacrificing accuracy. https://novasky-ai.github.io/posts/reduce-overthinking, January 20254. Accessed: 2025-01-23. - Team [20255] NovaSky Team. Sky-t1: Train your own o1 preview model within $ 450. https://novasky-ai.github.io/posts/sky-t1, January 20255. Accessed: 2025-01-09. - Team [2024] NVIDIA Team. Mistral-nemo-12b-instruct. https://huggingface.co/nvidia/Mistral-NeMo-12B-Instruct, July 2024. - Team [20256] OpenDeepResearch Team. Open deep research. https://github.com/nickscamara/open-deep-research, February 20256. - Team [20257] OpenO1 Team. Open o1. https://github.com/Open-Source-O1/Open-O1, February 20257. - Team [20258] OpenR1 Team. Open r1 math 200k. https://huggingface.co/datasets/open-r1/OpenR1-Math-220k, February 20258. - Team [20259] OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 20259. - Team [202510] PowerInfer Team. QwQ LongCoT 500k. https://huggingface.co/datasets/PowerInfer/QWQ-LONGCOT-500K, January 202510. - Team [202511] QwQ Team. Qwq: Reflect deeply on the boundaries of the unknown. https://qwenlm.github.io/blog/qwq-32b-preview/, November 202511. - Team [202512] X-R1 Team. X-r1. https://github.com/dhcode-cpp/X-R1, February 202512. - Teng et al. [2025] Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. Atom of thoughts for markov llm test-time scaling. arXiv preprint arXiv:2502.12018, 2025. - Thawakar et al. [2025] Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186, 2025. - Thomas et al. [2025] George Thomas, Alex J Chan, Jikun Kang, Wenqi Wu, Filippos Christianos, Fraser Greenlee, Andy Toulis, and Marvin Purtorab. Webgames: Challenging general-purpose web-browsing ai agents. arXiv preprint arXiv:2502.18356, 2025. - Tian et al. [2025] Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yunjie Ji, Yiping Peng, Han Zhao, and Xiangang Li. Think twice: Enhancing llm reasoning by scaling multi-round test-time thinking. arXiv preprint arXiv:2503.19855, 2025. - Tian et al. [2024] Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 52723–52748. Curran Associates, Inc., September 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/5e5853f35164e434015716a8c2a66543-Paper-Conference.pdf. - Tong et al. [2024] Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 7821–7846. Curran Associates, Inc., September 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/0ef1afa0daa888d695dcd5e9513bafa3-Paper-Conference.pdf. - Toshniwal et al. [2024] Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024. - Toshniwal et al. [20242] Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 20242. - Toshniwal et al. [20243] Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv-2402.10176, 20243. - Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. - Touvron et al. [20232] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 20232. - Treude and Kula [2025] Christoph Treude and Raula Gaikovina Kula. Interacting with ai reasoning models: Harnessing" thoughts" for ai-driven software engineering. arXiv preprint arXiv:2503.00483, 2025. - Trung et al. [2024] Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7601–7614, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.410. URL https://aclanthology.org/2024.acl-long.410/. - Tu et al. [2025] Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. Learning when to think: Shaping adaptive reasoning in r1-style models via multi-stage rl. arXiv preprint arXiv:2505.10832, 2025. - Turtel et al. [2025] Benjamin Turtel, Danny Franklin, and Philipp Schoenegger. Llms can teach themselves to better predict the future. arXiv preprint arXiv:2502.05253, 2025. - Tutek et al. [2025] Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović, and Yonatan Belinkov. Measuring faithfulness of chains of thought by unlearning reasoning steps. arXiv preprint arXiv:2502.14829, 2025. - Uesato et al. [2022] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. - Vacareanu et al. [2024] Robert Vacareanu, Anurag Pratik, Evangelia Spiliopoulou, Zheng Qi, Giovanni Paolini, Neha Anna John, Jie Ma, Yassine Benajiba, and Miguel Ballesteros. General purpose verification for chain of thought prompting. arXiv preprint arXiv:2405.00204, 2024. - Valmeekam et al. [2024] Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. LLMs still can’t plan; can LRMs? a preliminary evaluation of openAI’s o1 on planbench. In NeurIPS 2024 Workshop on Open-World Agents, October 2024. URL https://openreview.net/forum?id=Gcr1Lx4Koz. - Vassoyan et al. [2025] Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration on critical tokens to enhance rl fine-tuning. arXiv preprint arXiv:2502.06533, 2025. - Vu et al. [2024] Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17086–17105, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.949. URL https://aclanthology.org/2024.emnlp-main.949/. - Wan et al. [2024] Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. Cot rerailer: Enhancing the reliability of large language models in complex reasoning tasks through error detection and correction. arXiv preprint arXiv:2408.13940, 2024. - Wan et al. [20242] Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning, May 20242. URL https://openreview.net/forum?id=C4OpREezgj. - Wan et al. [2025] Ziyu Wan, Yunxiang Li, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, and Ying Wen. Rema: Learning to meta-think for llms with multi-agent reinforcement learning. arXiv preprint arXiv:2503.09501, 2025. - Wang et al. [2024] Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. Litesearch: Efficacious tree search for llm. arXiv preprint arXiv:2407.00320, 2024. - Wang et al. [2025] Ante Wang, Linfeng Song, Ye Tian, Dian Yu, Haitao Mi, Xiangyu Duan, Zhaopeng Tu, Jinsong Su, and Dong Yu. Don’t get lost in the trees: Streamlining llm reasoning by overcoming tree search exploration pitfalls. arXiv preprint arXiv:2502.11183, 2025. - Wang et al. [2023] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2717–2739, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.153. URL https://aclanthology.org/2023.acl-long.153/. - Wang et al. [20252] Chao Wang, Luning Zhang, Zheng Wang, and Yang Zhou. Can large language models unveil the mysteries? an exploration of their ability to unlock information in complex scenarios. arXiv preprint arXiv:2502.19973, 20252. - Wang et al. [20242] Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for llms with deliberative planning. arXiv preprint arXiv:2406.14283, 20242. - Wang et al. [20253] Chenlong Wang, Yuanning Feng, Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou. Wait, we don’t need to" wait"! removing thinking tokens improves reasoning efficiency. arXiv preprint arXiv:2506.08343, 20253. - Wang et al. [20254] Clinton J Wang, Dean Lee, Cristina Menghini, Johannes Mols, Jack Doughty, Adam Khoja, Jayson Lynch, Sean Hendryx, Summer Yue, and Dan Hendrycks. Enigmaeval: A benchmark of long multimodal reasoning challenges. arXiv preprint arXiv:2502.08859, 20254. - Wang et al. [20243] Danqing Wang, Zhuorui Ye, Fei Fang, and Lei Li. Cooperative strategic planning enhances reasoning capabilities in large language models. arXiv preprint arXiv:2410.20007, 20243. - Wang et al. [20244] Evan Z Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen Han, Sean M. Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves LLM search for code generation. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24, October 20244. URL https://openreview.net/forum?id=B2iSfPNj49. - Wang et al. [20255] Guoxin Wang, Minyu Gao, Shuai Yang, Ya Zhang, Lizhi He, Liang Huang, Hanlin Xiao, Yexuan Zhang, Wanyue Li, Lu Chen, et al. Citrus: Leveraging expert cognitive pathways in a medical language model for advanced medical decision support. arXiv preprint arXiv:2502.18274, 20255. - Wang et al. [20256] Hanbin Wang, Xiaoxuan Zhou, Zhipeng Xu, Keyuan Cheng, Yuxin Zuo, Kai Tian, Jingwei Song, Junting Lu, Wenhui Hu, and Xueyang Liu. Code-vision: Evaluating multimodal llms logic understanding and code generation capabilities. arXiv preprint arXiv:2502.11829, 20256. - Wang et al. [20257] Hanlin Wang, Jian Wang, Chak Tou Leong, and Wenjie Li. Steca: Step-level trajectory calibration for llm agent learning. arXiv preprint arXiv:2502.14276, 20257. - Wang et al. [20258] Hanyin Wang, Zhenbang Wu, Gururaj Kolar, Hariprasad Korsapati, Brian Bartlett, Bryan Hull, and Jimeng Sun. Reinforcement learning for out-of-distribution reasoning in llms: An empirical study on diagnosis-related group coding. arXiv preprint arXiv:2505.21908, 20258. - Wang et al. [20245] Hao Wang, Boyi Liu, Yufeng Zhang, and Jie Chen. Seed-cts: Unleashing the power of tree search for superior performance in competitive coding tasks. arXiv preprint arXiv:2412.12544, 20245. - Wang et al. [20246] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10582–10592, Miami, Florida, USA, November 20246. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.620. URL https://aclanthology.org/2024.findings-emnlp.620/. - Wang et al. [20259] Haoyu Wang, Zeyu Qin, Li Shen, Xueqian Wang, Minhao Cheng, and Dacheng Tao. Leveraging reasoning with guidelines to elicit and utilize knowledge for enhancing safety alignment. arXiv preprint arXiv:2502.04040, 20259. - Wang et al. [20247] Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for llm multi-step reasoning. arXiv preprint arXiv:2412.16145, 20247. - Wang et al. [20248] Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. Drt-o1: Optimized deep reasoning translation via long chain-of-thought. arXiv preprint arXiv:2412.17498, 20248. - Wang et al. [202510] Jiaan Wang, Fandong Meng, and Jie Zhou. Extrans: Multilingual deep reasoning translation via exemplar-enhanced reinforcement learning. arXiv preprint arXiv:2505.12996, 202510. - WANG et al. [2025] Jiaqi WANG, Yuhang Zhou, Zhixiong Zhang, Qiguang Chen, Yongqiang Chen, and James Cheng. DivIL: Unveiling and addressing over-invariance for out-of-distribution generalization. Transactions on Machine Learning Research, February 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=2Zan4ATYsh. - Wang et al. [20249] Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 20249. - Wang et al. [202410] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692, 202410. - Wang et al. [2025] Junxiong Wang, Wen-Ding Li, Daniele Paliotta, Daniel Ritter, Alexander M Rush, and Tri Dao. M1: Towards scalable test-time compute with mamba reasoning models. arXiv preprint arXiv:2504.10449, 2025. - Wang et al. [20252] Junyang Wang, Haiyang Xu, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v: Learning mobile device operation through video-guided multi-agent collaboration. arXiv preprint arXiv:2502.17110, 20252. - Wang et al. [20232] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731, 20232. - Wang et al. [202411] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, September 202411. URL https://openreview.net/forum?id=QWTCcxMpPA. - Wang et al. [202412] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in LLMs for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations, January 202412. URL https://openreview.net/forum?id=z8TW0ttBPp. - Wang et al. [202413] Kevin Wang, Junbo Li, Neel P Bhatt, Yihan Xi, Qiang Liu, Ufuk Topcu, and Zhangyang Wang. On the planning abilities of openai’s o1 models: Feasibility, optimality, and generalizability. arXiv preprint arXiv:2409.19924, 202413. - Wang et al. [20253] Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585, 20253. - Wang et al. [20254] Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei. Chain-of-retrieval augmented generation. arXiv preprint arXiv:2501.14342, 20254. - Wang [2025] Libo Wang. Dynamic chain-of-thought: Towards adaptive deep reasoning. arXiv preprint arXiv:2502.10428, 2025. - Wang et al. [20255] Mengru Wang, Xingyu Chen, Yue Wang, Zhiwei He, Jiahao Xu, Tian Liang, Qiuzhi Liu, Yunzhi Yao, Wenxuan Wang, Ruotian Ma, et al. Two experts are all you need for steering thinking: Reinforcing cognitive effort in moe reasoning models without additional training. arXiv preprint arXiv:2505.14681, 20255. - Wang et al. [20256] Mingyang Wang, Lukas Lange, Heike Adel, Yunpu Ma, Jannik Strötgen, and Hinrich Schütze. Language mixing in reasoning language models: Patterns, impact, and internal causes. arXiv preprint arXiv:2505.14815, 20256. - Wang et al. [20257] Minzheng Wang, Yongbin Li, Haobo Wang, Xinghua Zhang, Nan Xu, Bingli Wu, Fei Huang, Haiyang Yu, and Wenji Mao. Adaptive thinking via mode policy optimization for social language agents. arXiv preprint arXiv:2505.02156, 20257. - Wang et al. [202414] Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement preference optimization. arXiv preprint arXiv:2409.14664, 202414. - Wang et al. [202415] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439, Bangkok, Thailand, August 202415. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.510. URL https://aclanthology.org/2024.acl-long.510/. - Wang et al. [20258] Peng Wang, Xuesi Hu, Jiageng Wu, Yuntao Zou, Qiancheng Zhang, and Dagang Li. What factors affect llms and rllms in financial question answering? arXiv preprint arXiv:2507.08339, 20258. - Wang et al. [20259] Peng Wang, Ruihan Tao, Qiguang Chen, Mengkang Hu, and Libo Qin. X-webagentbench: A multilingual interactive web benchmark for evaluating global agentic system. arXiv preprint arXiv:2505.15372, 20259. - Wang et al. [202510] Peng-Yuan Wang, Tian-Shuo Liu, Chenyang Wang, Yi-Di Wang, Shu Yan, Cheng-Xing Jia, Xu-Hui Liu, Xin-Wei Chen, Jia-Cheng Xu, Ziniu Li, et al. A survey on large language models for mathematical reasoning. arXiv preprint arXiv:2506.08446, 202510. - Wang et al. [202511] Ru Wang, Wei Huang, Selena Song, Haoyu Zhang, Yusuke Iwasawa, Yutaka Matsuo, and Jiaxian Guo. Beyond in-distribution success: Scaling curves of cot granularity for language model generalization. arXiv preprint arXiv:2502.18273, 202511. - Wang et al. [202512] Ruida Wang, Rui Pan, Yuxin Li, Jipeng Zhang, Yizhen Jia, Shizhe Diao, Renjie Pi, Junjie Hu, and Tong Zhang. Ma-lot: Model-collaboration lean-based long chain-of-thought reasoning enhances formal theorem proving. arXiv preprint arXiv:2503.03205, 202512. - Wang et al. [2022] Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.775. URL https://aclanthology.org/2022.emnlp-main.775/. - Wang et al. [202513] Siyuan Wang, Enda Zhao, Zhongyu Wei, and Xiang Ren. Stepwise informativeness search for improving llm reasoning. arXiv preprint arXiv:2502.15335, 202513. - Wang et al. [202514] Song Wang, Gongfan Fang, Lingdong Kong, Xiangtai Li, Jianyun Xu, Sheng Yang, Qiang Li, Jianke Zhu, and Xinchao Wang. Pixelthink: Towards efficient chain-of-pixel reasoning. arXiv preprint arXiv:2505.23727, 202514. - Wang et al. [202416] Tianlong Wang, Junzhe Chen, Xueting Han, and Jing Bai. Cpl: Critical plan step learning boosts llm generalization in reasoning tasks. arXiv preprint arXiv:2409.08642, 202416. - Wang et al. [20233] Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04592, 20233. - Wang et al. [202417] Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint arXiv:2408.02666, 202417. - Wang et al. [202515] Weixuan Wang, Minghao Wu, Barry Haddow, and Alexandra Birch. Demystifying multilingual chain-of-thought in process reward modeling. arXiv preprint arXiv:2502.12663, 202515. - Wang et al. [202516] Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library. arXiv preprint arXiv:2506.06122, 202516. - Wang et al. [202418] Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 202418. - Wang et al. [202517] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291, 202517. - Wang et al. [202518] Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, and Bang Liu. System-1.5 reasoning: Traversal in language and latent spaces with dynamic shortcuts. arXiv preprint arXiv:2505.18962, 202518. - Wang et al. [202519] Xiaoxuan Wang, Yihe Deng, Mingyu Derek Ma, and Wei Wang. Entropy-based adaptive weighting for self-training. arXiv preprint arXiv:2503.23913, 202519. - Wang et al. [20234] Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. Guiding language model reasoning with planning tokens. arXiv preprint arXiv:2310.05707, 20234. - Wang et al. [202419] Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 50026–50042. PMLR, 21–27 Jul 202419. URL https://proceedings.mlr.press/v235/wang24a.html. - Wang et al. [202520] Xinyi Wang, Shawn Tan, Mingyu Jin, William Yang Wang, Rameswar Panda, and Yikang Shen. Do larger language models imply better reasoning? a pretraining scaling law for reasoning. arXiv preprint arXiv:2504.03635, 202520. - Wang et al. [202420] Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint arXiv:2405.15973, 202420. - Wang et al. [202421] Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, and Dong Yu. Towards self-improvement of llms via mcts: Leveraging stepwise knowledge with curriculum preference learning. arXiv preprint arXiv:2410.06508, 202421. - Wang and Zhou [2024] Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=4Zt7S0B0Jp. - Wang et al. [20235] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, February 20235. URL https://openreview.net/forum?id=1PL1NIMMrw. - Wang et al. [202521] Yao Wang, Mingxuan Cui, and Arthur Jiang. Enabling ai scientists to recognize innovation: A domain-agnostic algorithm for assessing novelty. arXiv preprint arXiv:2503.01508, 202521. - Wang et al. [202422] Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 202422. URL https://openreview.net/forum?id=OtvNLTWYww. - Wang et al. [20236] Yiqun Wang, Sile Hu, Yonggang Zhang, Xiang Tian, Xuesong Liu, Yaowu Chen, Xu Shen, and Jieping Ye. How large language models implement chain-of-thought? September 20236. URL https://openreview.net/pdf?id=b2XfOm3RJa. - Wang et al. [202522] Yu Wang, Nan Yang, Liang Wang, and Furu Wei. Examining false positives under inference scaling for mathematical reasoning. arXiv preprint arXiv:2502.06217, 202522. - Wang et al. [202423] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, September 202423. URL https://openreview.net/forum?id=y10DM6R2r3. - Wang et al. [202523] Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning to critique is more effective than learning to imitate. arXiv preprint arXiv:2501.17703, 202523. - Wang et al. [202524] Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585, 202524. - Wang et al. [202525] Yuhang Wang, Youhe Jiang, Bin Cui, and Fangcheng Fu. Thinking short and right over thinking long: Serving llm reasoning efficiently and accurately. arXiv preprint arXiv:2505.13326, 202525. - Wang et al. [202526] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 202526. - Wang et al. [202424] Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models. In Neurips Safe Generative AI Workshop 2024, October 202424. URL https://openreview.net/forum?id=oaWajnM93y. - Wang et al. [202527] Zhengren Wang, Jiayang Yu, Dongsheng Ma, Zhe Chen, Yu Wang, Zhiyu Li, Feiyu Xiong, Yanfeng Wang, Linpeng Tang, Wentao Zhang, et al. Rare: Retrieval-augmented reasoning modeling. arXiv preprint arXiv:2503.23513, 202527. - Wang et al. [202528] Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733, 202528. - Wang et al. [202425] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, September 202425. URL https://openreview.net/forum?id=PvVKUFhaNy. - Wang et al. [20237] Zhongsheng Wang, Jiamou Liu, Qiming Bao, Hongfei Rong, and Jingfeng Zhang. Chatlogic: Integrating logic programming with large language models for multi-step reasoning. In Neuro-Symbolic Learning and Reasoning in the era of Large Language Models, December 20237. URL https://openreview.net/forum?id=AOqGF7Po7Z. - Wang et al. [202426] Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXiv:2402.02658, 202426. - Wang et al. [202529] Zixiao Wang, Yuxin Wang, Xiaorui Wang, Mengting Xing, Jie Gao, Jianjun Xu, Guangcan Liu, Chenhui Jin, Zhuo Wang, Shengzhuo Zhang, et al. Test-time scaling with reflective generative model. arXiv preprint arXiv:2507.01951, 202529. - Wei et al. [2025] Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yaofeng Sun, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, et al. Equibench: Benchmarking code reasoning capabilities of large language models via equivalence checking. arXiv preprint arXiv:2502.12466, 2025. - Wei [2025] Hao Wei. Medthoughts-8k: A medical question answering dataset, feb 2025. URL https://huggingface.co/datasets/hw-hwei/MedThoughts-8K. - Wei et al. [2024] Haoran Wei, Youyang Yin, Yumeng Li, Jia Wang, Liang Zhao, Jianjian Sun, Zheng Ge, and Xiangyu Zhang. Slow perception: Let’s perceive geometric figures step-by-step. arXiv preprint arXiv:2412.20631, 2024. - Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc., November 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf. - Wei et al. [20252] Shuyue Wei, Yongxin Tong, Zimu Zhou, Yi Xu, Jingkai Gao, Tongyu Wei, Tianran He, and Weifeng Lv. Federated reasoning llms: a survey. Frontiers of Computer Science, 19(12):1–23, jun 20252. - Wei et al. [20253] Ting-Ruen Wei, Haowei Liu, Xuyang Wu, and Yi Fang. A survey on feedback-based multi-step reasoning for large language models on mathematics. arXiv preprint arXiv:2502.14333, 20253. - Wei et al. [20254] Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu, Haoran Lv, Zejia Weng, et al. Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning. arXiv preprint arXiv:2507.05255, 20254. - Wei et al. [20255] Yongxian Wei, Anke Tang, Li Shen, Zixuan Hu, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. arXiv preprint arXiv:2501.01230, 20255. - Wei et al. [20256] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 20256. - Weir et al. [2024] Nathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller, and Peter Clark. Learning to reason via program generation, emulation, and search. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=te6VagJf6G. - Welleck et al. [2024] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference-time algorithms for large language models. Transactions on Machine Learning Research, November 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=eskQMcIbMS. Survey Certification. - Wen et al. [2025] Cheng Wen, Tingwei Guo, Shuaijiang Zhao, Wei Zou, and Xiangang Li. Sari: Structured audio reasoning via curriculum-guided reinforcement learning. arXiv preprint arXiv:2504.15900, 2025. - Wen et al. [20252] Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, and Minlie Huang. Codeplan: Unlocking reasoning potential in large language models by scaling code-form planning. In The Thirteenth International Conference on Learning Representations, January 20252. URL https://openreview.net/forum?id=dCPF1wlqj8. - Wen et al. [20253] Kaiyue Wen, Huaqing Zhang, Hongzhou Lin, and Jingzhao Zhang. From sparse dependence to sparse attention: Unveiling how chain-of-thought enhances transformer sample efficiency. In The Thirteenth International Conference on Learning Representations, 20253. URL https://openreview.net/forum?id=AmEgWDhmTr. - Wen et al. [20254] Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245, 20254. - Weng et al. [2023] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.167. URL https://aclanthology.org/2023.findings-emnlp.167/. - Weston and Sukhbaatar [2023] Jason Weston and Sainbayar Sukhbaatar. System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829, 2023. - White et al. [2025] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=sKYHBTAxVa. - Wolf et al. [2024] Yotam Wolf, Binyamin Rothberg, Dorin Shteyman, and Amnon Shashua. Compositional hardness of code in large language models–a probabilistic perspective. arXiv preprint arXiv:2409.18028, 2024. - Wu et al. [2024] Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. arXiv preprint arXiv:2405.07990, 2024. - Wu et al. [20242] Jinyang Wu, Mingkuan Feng, Shuai Zhang, Feihu Che, Zengqi Wen, and Jianhua Tao. Beyond examples: High-level automated reasoning paradigm in in-context learning via mcts. arXiv preprint arXiv:2411.18478, 20242. - Wu et al. [2025] Jinyang Wu, Mingkuan Feng, Shuai Zhang, Ruihan Jin, Feihu Che, Zengqi Wen, and Jianhua Tao. Boosting multimodal reasoning with mcts-automated structured thinking. arXiv preprint arXiv:2502.02339, 2025. - Wu et al. [20252] Jinyang Wu, Chonghua Liao, Mingkuan Feng, Shuai Zhang, Zhengqi Wen, Pengpeng Shao, Huazhe Xu, and Jianhua Tao. Thought-augmented policy optimization: Bridging external guidance and internal capabilities. arXiv preprint arXiv:2505.15692, 20252. - Wu et al. [20253] Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04644, 20253. - Wu et al. [20254] Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Rongrong Ji. Grounded chain-of-thought for multimodal large language models. arXiv preprint arXiv:2503.12799, 20254. - Wu et al. [20243] Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, et al. A comparative study on reasoning patterns of openai’s o1 model. arXiv preprint arXiv:2410.13639, 20243. - Wu et al. [20255] Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. Arm: Adaptive reasoning model. arXiv preprint arXiv:2505.20258, 20255. - Wu et al. [20244] Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Thinking llms: General instruction following with thought generation. arXiv preprint arXiv:2410.10630, 20244. - Wu et al. [20256] Wenjie Wu, Yongcheng Jing, Yingjie Wang, Wenbin Hu, and Dacheng Tao. Graph-augmented reasoning: Evolving step-by-step knowledge graph retrieval for llm reasoning. arXiv preprint arXiv:2503.01642, 20256. - Wu [2025] Xiaobao Wu. Sailing by the stars: A survey on reward models and learning strategies for learning from rewards. arXiv preprint arXiv:2505.02686, 2025. - Wu et al. [20257] Xiong Jun Wu, Zhenduo Zhang, ZuJie Wen, Zhiqiang Zhang, Wang Ren, Lei Shi, Cai Chen, Deng Zhao, Qing Wang, Xudong Han, et al. Sharp: Synthesizing high-quality aligned reasoning problems for large reasoning models reinforcement learning. arXiv preprint arXiv:2505.14147, 20257. - Wu et al. [20245] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, January 20245. - Wu et al. [20258] Yifan Wu, Jingze Shi, Bingheng Wu, Jiayi Zhang, Xiaotian Lin, Nan Tang, and Yuyu Luo. Concise reasoning, big gains: Pruning long reasoning trace with difficulty-aware prompting. arXiv preprint arXiv:2505.19716, 20258. - Wu et al. [20259] Yong Wu, Weihang Pan, Ke Li, Chen Binhui, Ping Li, and Binbin Lin. Beyond templates: Dynamic adaptation of reasoning demonstrations via feasibility-aware exploration. arXiv preprint arXiv:2505.20700, 20259. - Wu et al. [202510] Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266, 202510. - Wu et al. [20246] Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. Enhancing mathematical reasoning in llms by stepwise correction. arXiv preprint arXiv:2410.12934, 20246. - Wu et al. [20247] Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. Large language models can self-correct with minimal effort. In AI for Math Workshop @ ICML 2024, May 20247. URL https://openreview.net/forum?id=mmZLMs4l3d. - Wu et al. [202511] Zirui Wu, Xiao Liu, Jiayi Li, Lingpeng Kong, and Yansong Feng. Haste makes waste: Evaluating planning abilities of llms for efficient and feasible multitasking with time constraints between actions. arXiv preprint arXiv:2503.02238, 202511. - Wu et al. [202512] Zongqian Wu, Tianyu Li, Jiaying Yang, Mengmeng Zhan, Xiaofeng Zhu, and Lei Feng. Is depth all you need? an exploration of iterative reasoning in llms. arXiv preprint arXiv:2502.10858, 202512. - Xi et al. [2024] Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.05808, 2024. - Xi et al. [20242] Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579, 20242. - Xi et al. [2025] Zhiheng Xi, Guanyu Li, Yutao Fan, Honglin Guo, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding, Wangmeng Zuo, Zhenfei Yin, et al. Bmmr: A large-scale bilingual multimodal multi-discipline reasoning dataset. arXiv preprint arXiv:2507.03483, 2025. - Xia et al. [2025] Fanzeng Xia, Yidong Luo, Tinko Sebastian Bartels, Yaqi Xu, and Tongxin Li. Rethinking the unsolvable: When in-context search meets test-time scaling. arXiv preprint arXiv:2505.22290, 2025. - Xia et al. [20252] Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12067, 20252. - Xia et al. [2024] Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.05692, 2024. - Xia et al. [20253] Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504.14655, 20253. - Xiang et al. [2024] Kun Xiang, Zhili Liu, Zihao Jiang, Yunshuang Nie, Runhui Huang, Haoxiang Fan, Hanhui Li, Weiran Huang, Yihan Zeng, Jianhua Han, et al. Atomthink: A slow thinking framework for multimodal mathematical reasoning. arXiv preprint arXiv:2411.11930, 2024. - Xiang et al. [2025] Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, and Nick Haber. Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning. arXiv preprint arXiv:2506.05256, 2025. - Xiang et al. [20252] Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.04682, 20252. - Xiao et al. [2024] Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, and Fei Wu. A comprehensive survey of direct preference optimization: Datasets, theories, variants, and applications. arXiv preprint arXiv:2410.15595, 2024. - Xie et al. [2024] Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024. - Xie et al. [2025] Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. arXiv preprint arXiv:2501.18427, 2025. - Xie et al. [20252] Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, and Xilin Chen. Robotic programmer: Video instructed policy code generation for robotic manipulation. arXiv preprint arXiv:2501.04268, 20252. - Xie et al. [20253] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, February 20253. - Xie et al. [20242] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, September 20242. URL https://openreview.net/forum?id=tN61DTr4Ed. - Xie et al. [2023] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Self-evaluation guided beam search for reasoning. In Thirty-seventh Conference on Neural Information Processing Systems, September 2023. URL https://openreview.net/forum?id=Bw82hwg5Q3. - Xie et al. [20243] Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 20243. - Xie et al. [20254] Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318, 20254. - Xie et al. [20255] Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, et al. Teaching language models to critique via reinforcement learning. arXiv preprint arXiv:2502.03492, 20255. - Xiong et al. [2024] Siheng Xiong, Ali Payani, Yuan Yang, and Faramarz Fekri. Deliberate reasoning for llms as structure-aware planning with accurate world model. arXiv preprint arXiv:2410.03136, 2024. - Xiong et al. [20242] Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392, 20242. - Xiyao et al. [2024] Wang Xiyao, Yang Zhengyuan, Li Linjie, Lu Hongjin, Xu Yuancheng, Lin Chung-Ching Lin, Lin Kevin, Huang Furong, and Wang Lijuan. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704, 2024. - Xu et al. [2025] Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. J4r: Learning to judge with equivalent initial state group relative policy optimization. arXiv preprint arXiv:2505.13346, 2025. - Xu et al. [2024] Bin Xu, Yiguan Lin, Yinghao Li, et al. Sra-mcts: Self-driven reasoning aurmentation with monte carlo tree search for enhanced code generation. arXiv preprint arXiv:2411.11053, 2024. - Xu et al. [20242] Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. Interactive evolution: A neural-symbolic self-training framework for large language models. arXiv preprint arXiv:2406.11736, 20242. - Xu et al. [20252] Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, and Zhiyong Wu. Genius: A generalizable and purely unsupervised self-training framework for advanced reasoning. arXiv preprint arXiv:2504.08672, 20252. - Xu et al. [20253] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686, 20253. - Xu et al. [20243] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 20243. - Xu [2023] Haotian Xu. No train still gain. unleash mathematical reasoning of large language models with monte carlo tree search guided by energy function. arXiv preprint arXiv:2309.03224, 2023. - Xu et al. [20254] Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 20254. - Xu et al. [20255] Huimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu, Wang Chen, Wei Zhang, and Anh Tuan Luu. Full-step-dpo: Self-supervised preference optimization with step-wise rewards for mathematical reasoning. arXiv preprint arXiv:2502.14356, 20255. - Xu et al. [20256] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 20256. - Xu et al. [20257] Pusheng Xu, Yue Wu, Kai Jin, Xiaolan Chen, Mingguang He, and Danli Shi. Deepseek-r1 outperforms gemini 2.0 pro, openai o1, and o3-mini in bilingual complex ophthalmology reasoning. arXiv preprint arXiv:2502.17947, 20257. - Xu et al. [20258] Rongwu Xu, Xiaojian Li, Shuo Chen, and Wei Xu. " nuclear deployed!": Analyzing catastrophic risks in decision-making of autonomous llm agents. arXiv preprint arXiv:2502.11355, 20258. - Xu et al. [20259] Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502.18600, 20259. - Xu et al. [20244] Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. Pride and prejudice: LLM amplifies self-bias in self-refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474–15492, Bangkok, Thailand, August 20244. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.826. URL https://aclanthology.org/2024.acl-long.826/. - Xu et al. [202510] Xiaoang Xu, Shuo Wang, Xu Han, Zhenghao Liu, Huijia Wu, Peipei Li, Zhiyuan Liu, Maosong Sun, and Zhaofeng He. A*-thought: Efficient reasoning via bidirectional compression for low-resource settings. arXiv preprint arXiv:2505.24550, 202510. - Xu et al. [20245] Xin Xu, Shizhe Diao, Can Yang, and Yang Wang. Can we verify step by step for incorrect answer detection? arXiv preprint arXiv:2402.10528, 20245. - Xu et al. [202511] Yao Xu, Mingyu Xu, Fangyu Lei, Wangtao Sun, Xiangrong Zeng, Bingning Wang, Guang Liu, Shizhu He, Jun Zhao, and Kang Liu. Amplify adjacent token differences: Enhancing long chain-of-thought reasoning with shift-ffn. arXiv preprint arXiv:2505.17153, 202511. - Xu et al. [202512] Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, and Ivan Vulić. Visual planning: Let’s think only with images. In Workshop on Foundation Models Meet Embodied Agents at CVPR 2025, may 202512. URL https://openreview.net/forum?id=ELIt3v3S1J. - Xu et al. [202513] Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft chain-of-thought for efficient reasoning with llms. arXiv preprint arXiv:2502.12134, 202513. - Xu et al. [202514] Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot++: Test-time scaling with soft chain-of-thought reasoning. arXiv preprint arXiv:2505.11484, 202514. - Xu et al. [20246] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 20246. - Xu et al. [202515] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. February 202515. - Yan et al. [2025] Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945, 2025. - Yan et al. [20252] Kai Yan, Yufei Xu, Zhengyin Du, Xuesong Yao, Zheyu Wang, Xiaowen Guo, and Jiecao Chen. Recitation over reasoning: How cutting-edge language models can fail on elementary school-level reasoning problems? arXiv preprint arXiv:2504.00509, 20252. - Yan et al. [20253] Ruin Yan, Zheng Liu, and Defu Lian. O1 embedder: Let retrievers think before action. arXiv preprint arXiv:2502.07555, 20253. - Yan et al. [2024] Siming Yan, Min Bai, Weifeng Chen, Xiong Zhou, Qixing Huang, and Li Erran Li. Vigor: Improving visual grounding of large vision language models with fine-grained reward modeling. In European Conference on Computer Vision, pages 37–53. Springer, 2024. - Yan et al. [20242] Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. arXiv preprint arXiv:2412.11936, 20242. - Yan et al. [20243] Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, et al. Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXiv:2410.04509, 20243. - Yan et al. [20254] Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S Yu, Carla Gomes, Bart Selman, and Qingsong Wen. Position: Multimodal large language models can significantly advance scientific reasoning. arXiv preprint arXiv:2502.02871, 20254. - Yan et al. [20244] Yuchen Yan, Jin Jiang, Yang Liu, Yixin Cao, Xin Xu, Xunliang Cai, Jian Shao, et al. S 3 c-math: Spontaneous step-level self-correction makes large language models better mathematical reasoners. arXiv preprint arXiv:2409.01524, 20244. - Yang et al. [2024] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. - Yang et al. [20242] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 20242. - Yang et al. [20243] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 20243. - Yang et al. [2025] Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Xiaojun Wu, Honghao Liu, Hui Xiong, and Jian Guo. Select2reason: Efficient instruction-tuning data selection for long-cot reasoning. arXiv preprint arXiv:2505.17266, 2025. - Yang et al. [20244] Chen Yang, Chenyang Zhao, Quanquan Gu, and Dongruo Zhou. Cops: Empowering llm agents with provable cross-task experience sharing. arXiv preprint arXiv:2410.16670, 20244. - Yang et al. [20245] Cheng Yang, Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, and Wai Lam. Llm2: Let large language models harness system 2 reasoning. arXiv preprint arXiv:2412.20372, 20245. - Yang et al. [20252] Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran XU, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, and Yujiu Yang. Chartmimic: Evaluating LMM’s cross-modal reasoning capability via chart-to-code generation. In The Thirteenth International Conference on Learning Representations, January 20252. URL https://openreview.net/forum?id=sGpCzsfd1K. - Yang et al. [20246] Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Selective preference optimization via token-level reward function estimation. arXiv preprint arXiv:2408.13518, 20246. - Yang et al. [20247] Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, and Dawn Song. Formal mathematical reasoning: A new frontier in ai. arXiv preprint arXiv:2412.16075, 20247. - Yang et al. [20253] Lei Yang, Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, and Deyi Xiong. Probench: Benchmarking large language models in competitive programming. arXiv preprint arXiv:2502.20868, 20253. - Yang et al. [20254] Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical llm reasoning via scaling thought templates. arXiv preprint arXiv:2502.06772, 20254. - Yang et al. [20255] Ruihan Yang, Fanghua Ye, Jian Li, Siyu Yuan, Yikai Zhang, Zhaopeng Tu, Xiaolong Li, and Deqing Yang. The lighthouse of language: Enhancing llm agents via critique-guided improvement. arXiv preprint arXiv:2503.16024, 20255. - Yang et al. [2022] Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought imitation with procedure cloning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, November 2022. URL https://openreview.net/forum?id=ZJqqSa8FsH9. - Yang et al. [20256] Shiming Yang, Yuxuan Tong, Xinyao Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning. In Forty-second International Conference on Machine Learning, may 20256. URL https://openreview.net/forum?id=OLodUbcWjB. - Yang et al. [20257] Shu Yang, Junchao Wu, Xin Chen, Yunze Xiao, Xinyi Yang, Derek F. Wong, and Di Wang. Understanding aha moments: from external observations to internal mechanisms. arXiv preprint arXiv:2504.02956, 20257. - Yang et al. [20258] Shu Yang, Junchao Wu, Xuansheng Wu, Derek Wong, Ninhao Liu, and Di Wang. Is long-to-short a free lunch? investigating inconsistency and reasoning efficiency in lrms. arXiv preprint arXiv:2506.19492, 20258. - Yang et al. [20248] Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210–10229, Bangkok, Thailand, August 20248. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.550. URL https://aclanthology.org/2024.acl-long.550/. - Yang et al. [20259] Wang Yang, Hongye Jin, Jingfeng Yang, Vipin Chaudhary, and Xiaotian Han. Thinking preference optimization. arXiv preprint arXiv:2502.13173, 20259. - Yang et al. [202510] Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test-time compute for llm reasoning. arXiv preprint arXiv:2502.18080, 202510. - Yang et al. [202511] Xiao-Wen Yang, Xuan-Yi Zhu, Wen-Da Wei, Ding-Chu Zhang, Jie-Jing Shao, Zhi Zhou, Lan-Zhe Guo, and Yu-Feng Li. Step back to leap forward: Self-backtracking for boosting reasoning of language models. arXiv preprint arXiv:2502.04404, 202511. - Yang et al. [202512] Yang Yang, Xiaolu Zhou, Bosong Ding, and Miao Xin. Uncertainty-aware reward design process. arXiv preprint arXiv:2507.02256, 202512. - Yang et al. [20249] Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.18517, 20249. - Yang et al. [202513] Yue Yang, MingKang Chen, Qihua Liu, Mengkang Hu, Qiguang Chen, Gengrui Zhang, Shuyue Hu, Guangtao Zhai, Yu Qiao, Yu Wang, et al. Truly assessing fluid intelligence of large language models through dynamic reasoning evaluation. arXiv preprint arXiv:2506.02648, 202513. - Yang et al. [202410] Yuqing Yang, Yan Ma, and Pengfei Liu. Weak-to-strong reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8350–8367, Miami, Florida, USA, November 202410. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.490. URL https://aclanthology.org/2024.findings-emnlp.490/. - Yang et al. [202514] Zeyuan Yang, Xueyang Yu, Delin Chen, Maohao Shen, and Chuang Gan. Machine mental imagery: Empower multimodal reasoning with latent visual tokens. arXiv preprint arXiv:2506.17218, 202514. - Yang et al. [202411] Zhe Yang, Yichang Zhang, Yudong Wang, Ziyao Xu, Junyang Lin, and Zhifang Sui. Confidence vs critique: A decomposition of self-correction capability for llms. arXiv preprint arXiv:2412.19513, 202411. - Yang et al. [202412] Zonghan Yang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. React meets actre: Autonomous annotation of agent trajectories for contrastive self-training. In First Conference on Language Modeling, July 202412. URL https://openreview.net/forum?id=0VLBwQGWpA. - Yao et al. [2024] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024. - Yao et al. [2025] Huanjin Yao, Jiaxing Huang, Yawen Qiu, Michael K Chen, Wenzheng Liu, Wei Zhang, Wenjie Zeng, Xikun Zhang, Jingyi Zhang, Yuxin Song, et al. Mmreason: An open-ended multi-modal multi-step reasoning benchmark for mllms toward agi. arXiv preprint arXiv:2506.23563, 2025. - Yao et al. [2022] Shunyu Yao, Howard Chen, John Yang, and Karthik R Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=R9KnuFlvnU. - Yao et al. [2023] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 11809–11822. Curran Associates, Inc., September 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf. - Yao et al. [20232] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, February 20232. URL https://openreview.net/forum?id=WE_vluYUL-X. - Yao et al. [20252] Xinhao Yao, Ruifeng Ren, Yun Liao, and Yong Liu. Unveiling the mechanisms of explicit cot training: How chain-of-thought enhances reasoning generalization. arXiv preprint arXiv:2502.04667, 20252. - Yao et al. [20253] Yang Yao, Xuan Tong, Ruofan Wang, Yixu Wang, Lujundong Li, Liang Liu, Yan Teng, and Yingchun Wang. A mousetrap: Fooling large reasoning models for jailbreak with chain of iterative chaos. arXiv preprint arXiv:2502.15806, 20253. - Yaoting et al. [2025] Wang Yaoting, Wu Shengqiong, Zhang Yuechen, Yan Shuicheng, Liu Ziwei, Luo Jiebo, and Fei Hao. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605, 2025. - Yasunaga et al. [2025] Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal rewardbench: Holistic evaluation of reward models for vision language models. arXiv preprint arXiv:2502.14191, 2025. - Yax et al. [2024] Nicolas Yax, Hernán Anlló, and Stefano Palminteri. Studying and improving reasoning in humans and machines. Communications Psychology, 2(1):51, 2024. - Ye et al. [2025] Guanghao Ye, Khiem Duc Pham, Xinzhi Zhang, Sivakanth Gopi, Baolin Peng, Beibin Li, Janardhan Kulkarni, and Huseyin A Inan. On the emergence of thinking in llms i: Searching for the right intuition. arXiv preprint arXiv:2502.06773, 2025. - Ye et al. [20252] Jiaran Ye, Zijun Yao, Zhidian Huang, Liangming Pan, Jinxin Liu, Yushi Bai, Amy Xin, Liu Weichuan, Xiaoyin Che, Lei Hou, et al. How does transformer learn implicit reasoning? arXiv preprint arXiv:2505.23653, 20252. - Ye et al. [20253] Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Jing Shao, and Siheng Chen. MAS-GPT: Training LLMs to build LLM-based multi-agent systems. In Workshop on Reasoning and Planning for Large Language Models, March 20253. URL https://openreview.net/forum?id=TqHoQIlumy. - Ye et al. [20254] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems. In The Thirteenth International Conference on Learning Representations, January 20254. URL https://openreview.net/forum?id=zpDGwcmMV4. - Ye et al. [20255] Xinwu Ye, Chengfan Li, Siming Chen, Xiangru Tang, and Wei Wei. Mmscibench: Benchmarking language models on multimodal scientific problems. arXiv preprint arXiv:2503.01891, 20255. - Ye et al. [20256] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 20256. - Ye et al. [2024] Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20850, 2024. - Ye et al. [20257] Zihuiwen Ye, Luckeciano Carvalho Melo, Younesse Kaddar, Phil Blunsom, Sam Staton, and Yarin Gal. Uncertainty-aware step-wise verification with generative reward models. arXiv preprint arXiv:2502.11250, 20257. - Yi et al. [2025] Hao Yi, Qingyang Li, Yulan Hu, Fuzheng Zhang, Di Zhang, and Yong Liu. Sppd: Self-training with process preference learning using dynamic value margin. arXiv preprint arXiv:2502.13516, 2025. - Yi et al. [20252] Jingyang Yi, Jiazheng Wang, and Sida Li. Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning. arXiv preprint arXiv:2504.21370, 20252. - Yin et al. [2025] Qiyue Yin, Pei Xu, Qiaozhe Li, Shengda Liu, Shengqi Shen, Tong Wang, Yihong Han, Xiaonan Zhao, Likun Yang, Shiyue Cao, et al. Wgsr-bench: Wargame-based game-theoretic strategic reasoning benchmark for large language models. arXiv preprint arXiv:2506.10264, 2025. - Yin et al. [2024] Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Junqi Dai, Qinyuan Cheng, Xuanjing Huang, and Xipeng Qiu. Reasoning in flux: Enhancing large language models reasoning through uncertainty-aware adaptive guidance. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2401–2416, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.131. URL https://aclanthology.org/2024.acl-long.131/. - Ying et al. [2024] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024. - Yoon et al. [2024] Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark A Hasegawa-Johnson, Sungwoong Kim, and Chang D Yoo. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. arXiv preprint arXiv:2407.16574, 2024. - Yoon et al. [2025] Jaesik Yoon, Hyeonseo Cho, Doojin Baek, Yoshua Bengio, and Sungjin Ahn. Monte carlo tree diffusion for system 2 planning. arXiv preprint arXiv:2502.07202, 2025. - Yu et al. [2025] Bin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. Long-short chain-of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models. arXiv preprint arXiv:2505.03469, 2025. - Yu et al. [2024] Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, and Dong Yu. Siam: Self-improving code-assisted mathematical reasoning of large language models. arXiv preprint arXiv:2408.15565, 2024. - Yu et al. [20242] Fei Yu, Anningzhe Gao, and Benyou Wang. OVM, outcome-supervised value models for planning in mathematical reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 858–875, Mexico City, Mexico, June 20242. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.55. URL https://aclanthology.org/2024.findings-naacl.55/. - Yu et al. [20243] Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey. ACM Comput. Surv., 56(12), October 20243. ISSN 0360-0300. doi: 10.1145/3664194. URL https://doi.org/10.1145/3664194. - Yu et al. [20252] Fei Yu, Yingru Li, and Benyou Wang. Uncertainty-aware search and value models: Mitigating search scaling flaws in llms. arXiv preprint arXiv:2502.11155, 20252. - Yu et al. [20253] Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshaping long-context llm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.02259, 20253. - Yu et al. [20244] Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, January 20244. URL https://openreview.net/forum?id=N8N0hgNDRt. - Yu et al. [20245] Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023, 20245. - Yu et al. [20254] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 20254. - Yu et al. [20255] Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. Rlpr: Extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506.18254, 20255. - Yu et al. [20256] Tong Yu, Yongcheng Jing, Xikun Zhang, Wentao Jiang, Wenjie Wu, Yingjie Wang, Wenbin Hu, Bo Du, and Dacheng Tao. Benchmarking reasoning robustness in large language models. arXiv preprint arXiv:2503.04550, 20256. - Yu et al. [20257] Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. ExACT: Teaching AI agents to explore with reflective-MCTS and exploratory learning. In The Thirteenth International Conference on Learning Representations, January 20257. URL https://openreview.net/forum?id=GBIUbwW9D8. - Yu et al. [20258] Yahan Yu, Yuyang Dong, and Masafumi Oyamada. Learning deliberately, acting intuitively: Unlocking test-time reasoning in multimodal llms. arXiv preprint arXiv:2507.06999, 20258. - Yu et al. [20259] Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, et al. Chain-of-reasoning: Towards unified mathematical reasoning in large language models via a multi-paradigm perspective. arXiv preprint arXiv:2501.11110, 20259. - Yu et al. [20246] Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 20246. - Yu et al. [202510] Zeping Yu, Yonatan Belinkov, and Sophia Ananiadou. Back attention: Understanding and enhancing multi-hop reasoning in large language models. arXiv preprint arXiv:2502.10835, 202510. - Yu et al. [20247] Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199, 20247. - Yu et al. [202511] Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Z1: Efficient test-time scaling with code. arXiv preprint arXiv:2504.00810, 202511. - Yu et al. [202512] Zhouliang Yu, Yuhuan Yuan, Tim Z Xiao, Fuxiang Frank Xia, Jie Fu, Ge Zhang, Ge Lin, and Weiyang Liu. Generating symbolic world models via test-time scaling of large language models. arXiv preprint arXiv:2502.04728, 202512. - Yu et al. [20248] Zhuohao Yu, Weizheng Gu, Yidong Wang, Zhengran Zeng, Jindong Wang, Wei Ye, and Shikun Zhang. Outcome-refining process supervision for code generation. arXiv preprint arXiv:2412.15118, 20248. - Yu et al. [202513] Zishun Yu, Tengyu Xu, Di Jin, Karthik Abinav Sankararaman, Yun He, Wenxuan Zhou, Zhouhao Zeng, Eryk Helenowski, Chen Zhu, Sinong Wang, et al. Think smarter not harder: Adaptive reasoning with inference aware optimization. arXiv preprint arXiv:2501.17974, 202513. - Yuan et al. [2025] Hang Yuan, Bin Yu, Haotian Li, Shijun Yang, Christina Dan Wang, Zhou Yu, Xueyin Xu, Weizhen Qi, and Kai Chen. Not all tokens are what you need in thinking. arXiv preprint arXiv:2505.17827, 2025. - Yuan et al. [2024] Jiahao Yuan, Dehui Du, Hao Zhang, Zixiang Di, and Usman Naseem. Reversal of thought: Enhancing large language models with preference-guided reverse reasoning warm-up. arXiv preprint arXiv:2410.12323, 2024. - Yuan et al. [20242] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01981, 20242. - Yuan et al. [20252] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. In The Thirteenth International Conference on Learning Representations, January 20252. URL https://openreview.net/forum?id=2ea5TNVR0c. - Yuan et al. [20253] Michelle Yuan, Elman Mansimov, Katerina Margatina, Anurag Pratik, Daniele Bonadiman, Monica Sunkara, Yi Zhang, Yassine Benajiba, et al. A study on leveraging search and self-feedback for agent reasoning. arXiv preprint arXiv:2502.12094, 20253. - Yuan et al. [20254] Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 20254. - Yuan et al. [20255] Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, and Xian Li. Naturalreasoning: Reasoning in the wild with 2.8m challenging questions, 20255. - Yuan et al. [20256] Yige Yuan, Teng Xiao, Shuchang Tao, Xue Wang, Jinyang Gao, Bolin Ding, and Bingbing Xu. Incentivizing reasoning from weak supervision. arXiv preprint arXiv:2505.20072, 20256. - Yue et al. [2023] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023. - Yue et al. [2025] Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 37:90629–90660, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/a4ca07aa108036f80cbb5b82285fd4b1-Paper-Conference.pdf. - Yue et al. [20252] Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, and Dong Wang. Hybrid latent reasoning via reinforcement learning. arXiv preprint arXiv:2505.18454, 20252. - Yuksekgonul et al. [2025] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055):609–616, March 2025. URL https://www.nature.com/articles/s41586-025-08661-4. - YuYue et al. [2025] YuYue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025. - Zang et al. [2025] Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. Internlm-xcomposer2.5-reward: A simple yet effective multi-modal reward model. arXiv preprint arXiv:2501.12368, 2025. - Zelikman et al. [2022] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, November 2022. URL https://openreview.net/pdf?id=_3ELRdg2sgI. - Zelikman et al. [2024] Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024. - Zeng et al. [2025] Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. arXiv preprint arXiv:2502.01718, 2025. - Zeng et al. [20252] Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, et al. Versaprm: Multi-domain process reward model via synthetic reasoning data. arXiv preprint arXiv:2502.06737, 20252. - Zeng et al. [2024] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B-star: Monitoring and balancing exploration and exploitation in self-taught reasoners. arXiv preprint arXiv:2412.17256, 2024. - Zeng et al. [20253] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 20253. - Zeng et al. [20254] Yongcheng Zeng, Xinyu Cui, Xuanfa Jin, Guoqing Liu, Zexu Sun, Quan He, Dong Li, Ning Yang, Jianye Hao, Haifeng Zhang, et al. Aries: Stimulating self-refinement of large language models by iterative preference optimization. arXiv preprint arXiv:2502.05605, 20254. - Zeng et al. [20242] Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective. arXiv preprint arXiv:2412.14135, 20242. - Zeng et al. [20255] Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, and Xipeng Qiu. Revisiting the test-time scaling of o1-like models: Do they truly possess test-time scaling capabilities? arXiv preprint arXiv:2502.12215, 20255. - Zeng et al. [20243] Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. MR-ben: A meta-reasoning benchmark for evaluating system-2 thinking in LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, June 20243. URL https://openreview.net/forum?id=GN2qbxZlni. - Zeng et al. [20256] Zihao Zeng, Xuyao Huang, Boxiu Li, and Zhijie Deng. Sift: Grounding llm reasoning in contexts via stickers. arXiv preprint arXiv:2502.14922, 20256. - Zhai et al. [2024] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 2024. URL https://openreview.net/forum?id=nBjmMF2IZU. - Zhan et al. [2025] Zaifu Zhan, Shuang Zhou, Huixue Zhou, Jiawen Deng, Yu Hou, Jeremy Yeung, and Rui Zhang. An evaluation of deepseek models in biomedical natural language processing. arXiv preprint arXiv:2503.00624, 2025. - Zhang et al. [2025] Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, et al. Codecriticbench: A holistic code critique benchmark for large language models. arXiv preprint arXiv:2502.16614, 2025. - Zhang et al. [20252] Beichen Zhang, Yuhong Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Haodong Duan, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Booststep: Boosting mathematical capability of large language models via improved single-step reasoning. arXiv preprint arXiv:2501.03226, 20252. - Zhang et al. [20253] Bohan Zhang, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Cot-based synthesizer: Enhancing llm performance through answer synthesis. arXiv preprint arXiv:2501.01668, 20253. - Zhang et al. [2024] Che Zhang, Zhenyang Xiao, Chengcheng Han, Yixin Lian, and Yuejian Fang. Learning to check: Unleashing potentials for self-correction in large language models. arXiv preprint arXiv:2402.13035, 2024. - Zhang et al. [20242] Chi Zhang, Jiajun Song, Siyu Li, Yitao Liang, Yuxi Ma, Wei Wang, Yixin Zhu, and Song-Chun Zhu. Proposing and solving olympiad geometry with guided tree search. arXiv preprint arXiv:2412.10673, 20242. - Zhang et al. [20254] Chunhui Zhang, Zhongyu Ouyang, Kwonjoon Lee, Nakul Agarwal, Sean Dae Houlihan, Soroush Vosoughi, and Shao-Yuan Lo. Overcoming multi-step complexity in multimodal theory-of-mind reasoning: A scalable bayesian planner. In Forty-second International Conference on Machine Learning, 20254. URL https://openreview.net/forum?id=2dz6psiiA0. - Zhang et al. [20255] Dalong Zhang, Jun Xu, Jun Zhou, Lei Liang, Lin Yuan, Ling Zhong, Mengshu Sun, Peilong Zhao, QiWei Wang, Xiaorui Wang, et al. Kag-thinker: Teaching large language models to think with human-like reasoning process. arXiv preprint arXiv:2506.17728, 20255. - Zhang et al. [20243] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST-MCTS*: LLM self-training via process reward guided tree search. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, September 20243. URL https://openreview.net/forum?id=8rcFOqEud5. - Zhang et al. [20244] Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.07394, 20244. - Zhang et al. [20245] Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884, 20245. - Zhang et al. [20246] Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks. arXiv preprint arXiv:2410.12381, 20246. - Zhang et al. [20247] Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. Entropy-regularized process reward model. arXiv preprint arXiv:2412.11006, 20247. - Zhang et al. [20256] Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, and Song Guo. Lazyeviction: Lagged kv eviction with attention pattern observation for efficient long reasoning. arXiv preprint arXiv:2506.15969, 20256. - Zhang et al. [20257] Hongbo Zhang, Han Cui, Guangsheng Bao, Linyi Yang, Jun Wang, and Yue Zhang. Direct value optimization: Improving chain-of-thought reasoning in llms with refined values. arXiv preprint arXiv:2502.13723, 20257. - Zhang et al. [20248] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 20248. - Zhang et al. [20258] Jinghan Zhang, Xiting Wang, Fengran Mo, Yeyang Zhou, Wanfu Gao, and Kunpeng Liu. Entropy-based exploration conduction for multi-step reasoning. arXiv preprint arXiv:2503.15848, 20258. - Zhang et al. [20259] Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arXiv preprint arXiv:2502.15589, 20259. - Zhang et al. [202510] Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. arXiv preprint arXiv:2507.02841, 202510. - Zhang et al. [202511] Kechi Zhang, Ge Li, Jia Li, Yihong Dong, and Zhi Jin. Focused-dpo: Enhancing code generation through focused preference optimization on error-prone points. arXiv preprint arXiv:2502.11475, 202511. - Zhang et al. [202512] Kechi Zhang, Ge Li, Jia Li, Huangzhao Zhang, Jingjing Xu, Hao Zhu, Lecheng Wang, Yihong Dong, Jing Mai, Bin Gu, et al. Computational thinking reasoning in large language models. arXiv preprint arXiv:2506.02658, 202512. - Zhang et al. [20249] Kexun Zhang, Shang Zhou, Danqing Wang, William Yang Wang, and Lei Li. Scaling llm inference with optimized sample compute allocation. arXiv preprint arXiv:2410.22480, 20249. - Zhang et al. [202513] Kongcheng Zhang, Qi Yao, Baisheng Lai, Jiaxing Huang, Wenkai Fang, Dacheng Tao, Mingli Song, and Shunyu Liu. Reasoning with reinforced functional token tuning. arXiv preprint arXiv:2502.13389, 202513. - Zhang et al. [202514] Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.08745, 202514. - Zhang et al. [202410] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 202410. - Zhang et al. [202515] Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, et al. Llmeval-med: A real-world clinical benchmark for medical llms with physician validation. arXiv preprint arXiv:2506.04078, 202515. - Zhang et al. [2023] Ming-Liang Zhang, Fei yin, and Cheng-Lin Liu. A multi-modal neural geometric solver with textual clauses parsed from diagram. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 3374–3382. International Joint Conferences on Artificial Intelligence Organization, 8 2023. doi: 10.24963/ijcai.2023/376. URL https://doi.org/10.24963/ijcai.2023/376. Main Track. - Zhang et al. [202411] Qingjie Zhang, Han Qiu, Di Wang, Haoting Qian, Yiming Li, Tianwei Zhang, and Minlie Huang. Understanding the dark side of llms’ intrinsic self-correction. arXiv preprint arXiv:2412.14959, 202411. - Zhang et al. [202516] Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models. arXiv preprint arXiv:2503.24235, 202516. - Zhang et al. [202517] Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling in large language models: What, how, where, and how well? arXiv preprint arXiv:2503.24235, 202517. - Zhang et al. [202412] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, October 202412. URL https://link.springer.com/chapter/10.1007/978-3-031-73242-3_10. - Zhang and Xiong [2025] Shaowei Zhang and Deyi Xiong. BackMATH: Towards backward reasoning for solving math problems step by step. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Darwish, and Apoorv Agarwal, editors, Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 466–482, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-industry.40/. - Zhang et al. [202518] Shenao Zhang, Yaqing Wang, Yinxiao Liu, Tianqi Liu, Peter Grabowski, Eugene Ie, Zhaoran Wang, and Yunxuan Li. Beyond markovian: Reflective exploration via bayes-adaptive rl for llm reasoning. arXiv preprint arXiv:2505.20561, 202518. - Zhang et al. [202519] Shengjia Zhang, Junjie Wu, Jiawei Chen, Changwang Zhang, Xingyu Lou, Wangchunshu Zhou, Sheng Zhou, Can Wang, and Jun Wang. Othink-r1: Intrinsic fast/slow thinking mode switching for over-reasoning mitigation. arXiv preprint arXiv:2506.02397, 202519. - Zhang et al. [20232] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 20232. - Zhang et al. [202520] Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong. Process-based self-rewarding language models. arXiv preprint arXiv:2503.03746, 202520. - Zhang et al. [202521] Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, et al. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506.18959, 202521. - Zhang et al. [202522] Wenjing Zhang, Xuejiao Lei, Zhaoxiang Liu, Ning Wang, Zhenhong Long, Peijun Yang, Jiaojiao Zhao, Minjie Hua, Chaoyang Ma, Kai Wang, et al. Safety evaluation of deepseek models in chinese contexts. arXiv preprint arXiv:2502.11137, 202522. - Zhang et al. [202413] Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3602–3622, Bangkok, Thailand, August 202413. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.197. URL https://aclanthology.org/2024.acl-long.197/. - Zhang et al. [202523] Xiaoyun Zhang, Jingqing Ruan, Xing Ma, Yawen Zhu, Haodong Zhao, Hao Li, Jiansong Chen, Ke Zeng, and Xunliang Cai. When to continue thinking: Adaptive thinking mode switching for efficient reasoning. arXiv preprint arXiv:2505.15400, 202523. - Zhang et al. [202524] Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning. arXiv preprint arXiv:2502.12054, 202524. - Zhang et al. [202414] Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving chain-of-thought reasoning in llms. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 333–356. Curran Associates, Inc., September 202414. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/00d80722b756de0166523a87805dd00f-Paper-Conference.pdf. - Zhang et al. [202525] Xuanliang Zhang, Dingzirui Wang, Keyan Xu, Qingfu Zhu, and Wanxiang Che. Rot: Enhancing table reasoning with iterative row-wise traversals. arXiv preprint arXiv:2505.15110, 202525. - Zhang et al. [202415] Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. On the diagram of thought. arXiv preprint arXiv:2409.10038, 202415. - Zhang et al. [202526] Yifan Zhang, Wenyu Du, Dongming Jin, Jie Fu, and Zhi Jin. Finite state automata inside transformers with chain-of-thought: A mechanistic study on state tracking. arXiv preprint arXiv:2502.20129, 202526. - Zhang et al. [202527] Yong Zhang, Bingyuan Zhang, Zhitao Li, Ming Li, Ning Cheng, Minchuan Chen, Tao Wei, Jun Ma, Shaojun Wang, and Jing Xiao. Self-enhanced reasoning training: Activating latent reasoning in small models for enhanced reasoning distillation. arXiv preprint arXiv:2502.12744, 202527. - Zhang et al. [202416] Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. AutoCAP: Towards automatic cross-lingual alignment planning for zero-shot chain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9191–9200, Bangkok, Thailand, August 202416. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.546. URL https://aclanthology.org/2024.findings-acl.546/. - Zhang et al. [202417] Yongheng Zhang, Qiguang Chen, Jingxuan Zhou, Peng Wang, Jiasheng Si, Jin Wang, Wenpeng Lu, and Libo Qin. Wrong-of-thought: An integrated reasoning framework with multi-perspective verification and wrong information. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6644–6653, Miami, Florida, USA, November 202417. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.388. URL https://aclanthology.org/2024.findings-emnlp.388/. - Zhang et al. [202528] Yongheng Zhang, Xu Liu, Ruihan Tao, Qiguang Chen, Hao Fei, Wanxiang Che, and Libo Qin. Vitcot: Video-text interleaved chain-of-thought for boosting video understanding in large language models. arXiv preprint arXiv:2507.09876, 202528. - Zhang et al. [202529] Yongheng Zhang, Xu Liu, Ruoxi Zhou, Qiguang Chen, Hao Fei, Wenpeng Lu, and Libo Qin. Cchall: A novel benchmark for joint cross-lingual and cross-modal hallucinations detection in large language models. arXiv preprint arXiv:2505.19108, 202529. - Zhang et al. [202530] Yudi Zhang, Lu Wang, Meng Fang, Yali Du, Chenghua Huang, Jun Wang, Qingwei Lin, Mykola Pechenizkiy, Dongmei Zhang, Saravan Rajmohan, et al. Distill not only data but also rewards: Can smaller language models surpass larger ones? arXiv preprint arXiv:2502.19557, 202530. - Zhang et al. [202418] Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 15637–15653, Bangkok, Thailand, August 202418. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.924. URL https://aclanthology.org/2024.findings-acl.924/. - Zhang et al. [202419] Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. o1-coder: an o1 replication for coding. arXiv preprint arXiv:2412.00154, 202419. - Zhang et al. [202420] Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Yuhang Wang, Jinlin Xiao, and Jitao Sang. Openrft: Adapting reasoning foundation model for domain-specific tasks with reinforcement fine-tuning. arXiv preprint arXiv:2412.16849, 202420. - Zhang et al. [202531] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 202531. - Zhang et al. [202532] Zhihao Zhang, Qiaole Dong, Qi Zhang, Jun Zhao, Enyu Zhou, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Yanwei Fu, et al. Reinforcement fine-tuning enables mllms learning novel tasks stably. arXiv preprint arXiv:2506.23508, 202532. - Zhang et al. [202533] Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi-Qin John Xu. Complexity control facilitates reasoning-based compositional generalization in transformers. arXiv preprint arXiv:2501.08537, 202533. - Zhang et al. [202421] Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Research, June 202421. ISSN 2835-8856. URL https://openreview.net/forum?id=y1pPWFVfvR. - Zhao et al. [2025] Deji Zhao, Donghong Han, Jia Wu, Zhongjiang He, Bo Ning, Ye Yuan, Yongxiang Li, Chao Wang, and Shuangyong Song. Enhancing math reasoning ability of large language models via computation logic graphs. Knowledge-Based Systems, page 113905, 2025. - Zhao et al. [20252] Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi. Sample, scrutinize and scale: Effective inference-time search by scaling verification. arXiv preprint arXiv:2502.01839, 20252. - Zhao et al. [20253] Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training. arXiv preprint arXiv:2503.19633, 20253. - Zhao et al. [2024] Jun Zhao, Jingqi Tong, Yurong Mou, Ming Zhang, Qi Zhang, and Xuanjing Huang. Exploring the compositional deficiency of large language models in mathematical reasoning through trap problems. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16361–16376, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.915. URL https://aclanthology.org/2024.emnlp-main.915/. - Zhao et al. [20254] Lili Zhao, Yang Wang, Qi Liu, Mengyun Wang, Wei Chen, Zhichao Sheng, and Shijin Wang. Evaluating large language models through role-guide and self-reflection: A comparative study. In The Thirteenth International Conference on Learning Representations, January 20254. URL https://openreview.net/forum?id=E36NHwe7Zc. - Zhao et al. [20255] Shangziqi Zhao, Jiahao Yuan, Guisong Yang, and Usman Naseem. Can pruning improve reasoning? revisiting long-cot compression with capability in mind for better reasoning. arXiv preprint arXiv:2505.14582, 20255. - Zhao et al. [20256] Weixiang Zhao, Jiahe Guo, Yang Deng, Xingyu Sui, Yulin Hu, Yanyan Zhao, Wanxiang Che, Bing Qin, Tat-Seng Chua, and Ting Liu. Exploring and exploiting the inherent efficiency within large reasoning models for self-guided efficiency enhancement. arXiv preprint arXiv:2506.15647, 20256. - Zhao et al. [20257] Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 20257. - Zhao et al. [20258] Xueliang Zhao, Wei Wu, Jian Guan, and Lingpeng Kong. Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models. arXiv preprint arXiv:2503.02324, 20258. - Zhao et al. [20242] Xufeng Zhao, Mengdi Li, Wenhao Lu, Cornelius Weber, Jae Hee Lee, Kun Chu, and Stefan Wermter. Enhancing zero-shot chain-of-thought reasoning in large language models through logic. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6144–6166, Torino, Italia, May 20242. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main.543/. - Zhao et al. [20259] Yachao Zhao, Bo Wang, and Yan Wang. Explicit vs. implicit: Investigating social bias in large language models through self-reflection. arXiv preprint arXiv:2501.02295, 20259. - Zhao et al. [202510] Yang Zhao, Kai Xiong, Xiao Ding, Li Du, Zhouhao Sun, Jiannan Guan, Wenbin Zhang, Bin Liu, Dong Hu, Bing Qin, et al. Ufo-rl: Uncertainty-focused optimization for efficient reinforcement learning data selection. arXiv preprint arXiv:2505.12457, 202510. - Zhao and Goto [2025] Yichong Zhao and Susumu Goto. Can frontier llms replace annotators in biomedical text mining? analyzing challenges and exploring solutions. arXiv preprint arXiv:2503.03261, 2025. - Zhao et al. [20243] Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-o1: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14405, 20243. - Zhao et al. [202511] Yurui Zhao, Xiang Wang, Jiahong Liu, Irwin King, and Zhitao Huang. Towards geometry problem solving in the large model era: A survey. arXiv preprint arXiv:2506.02690, 202511. - Zhao et al. [202512] Zhonghan Zhao, Wenwei Zhang, Haian Huang, Kuikun Liu, Jianfei Gao, Gaoang Wang, and Kai Chen. Rig: Synergizing reasoning and imagination in end-to-end generalist policy. arXiv preprint arXiv:2503.24388, 202512. - Zhao et al. [20244] Zilong Zhao, Yao Rong, Dongyang Guo, Emek Gözlüklü, Emir Gülboy, and Enkelejda Kasneci. Stepwise self-consistent mathematical reasoning with large language models. arXiv preprint arXiv:2402.17786, 20244. - Zhao et al. [2023] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36:31967–31987, December 2023. URL https://openreview.net/pdf?id=tED747HURfX. - Zheng et al. [2025] Bowen Zheng, Xiaolei Wang, Enze Liu, Xi Wang, Lu Hongyu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. Deeprec: Towards a deep dive into the item space with large language model based recommendation. arXiv preprint arXiv:2505.16810, 2025. - Zheng et al. [2024] Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models. In AI for Math Workshop @ ICML 2024, June 2024. URL https://openreview.net/forum?id=UkFEs3ciz8. - Zheng et al. [20242] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559, 20242. - Zheng et al. [20252] Da Zheng, Lun Du, Junwei Su, Yuchen Tian, Yuqi Zhu, Jintian Zhang, Lanning Wei, Ningyu Zhang, and Huajun Chen. Knowledge augmented complex problem solving with large language models: A survey. arXiv preprint arXiv:2505.03418, 20252. - Zheng et al. [2023] Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168–5191, 2023. - Zheng et al. [20253] Hang Zheng, Hongshen Xu, Yuncong Liu, Lu Chen, Pascale Fung, and Kai Yu. Enhancing llm reliability via explicit knowledge boundary modeling. arXiv preprint arXiv:2503.02233, 20253. - Zheng et al. [20254] Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Vem: Environment-free exploration for training gui agent with value environment model. arXiv preprint arXiv:2502.18906, 20254. - Zheng et al. [20255] Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, benjamin negrevergne, and Gabriel Synnaeve. What makes large language models reason in (multi-turn) code generation? In The Thirteenth International Conference on Learning Representations, January 20255. URL https://openreview.net/forum?id=Zk9guOl9NS. - Zheng et al. [20243] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodeInterpreter: Integrating code generation with execution and refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859, Bangkok, Thailand, August 20243. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.762. URL https://aclanthology.org/2024.findings-acl.762/. - Zheng et al. [20244] Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via chain-of-thoughts critic. arXiv preprint arXiv:2408.16326, 20244. - Zheng et al. [20256] Zhi Zheng, Zhuoliang Xie, Zhenkun Wang, and Bryan Hooi. Monte carlo tree search for comprehensive exploration in llm-based automatic heuristic design. arXiv preprint arXiv:2501.08603, 20256. - Zhong et al. [2025] Jianyuan Zhong, Zeju Li, Zhijian Xu, Xiangyu Wen, and Qiang Xu. Dyve: Thinking fast and slow for dynamic process verification. arXiv preprint arXiv:2502.11157, 2025. - Zhong et al. [2024] Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, and Bo Du. Achieving> 97% on gsm8k: Deeply understanding the problems makes llms better solvers for math word problems. arXiv preprint arXiv:2404.14963, 2024. - Zhong et al. [20242] Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486, 20242. - Zhou et al. [2024] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Forty-first International Conference on Machine Learning, May 2024. URL https://openreview.net/forum?id=njwv9BsGHF. - Zhou et al. [20242] Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. In The Twelfth International Conference on Learning Representations, January 20242. URL https://openreview.net/forum?id=c8McWs4Av0. - Zhou et al. [2025] Changzhi Zhou, Xinyu Zhang, Dandan Song, Xiancai Chen, Wanli Gu, Huipeng Ma, Yuhang Tian, Mengdi Zhang, and Linmei Hu. Refinecoder: Iterative improving of large language models via adaptive critique refinement for code generation. arXiv preprint arXiv:2502.09183, 2025. - Zhou et al. [2023] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, February 2023. URL https://openreview.net/forum?id=WZH7099tgfM. - Zhou et al. [2022] Fan Zhou, Haoyu Dong, Qian Liu, Zhoujun Cheng, Shi Han, and Dongmei Zhang. Reflection of thought: Inversely eliciting numerical reasoning in language models via solving linear systems. arXiv preprint arXiv:2210.05075, 2022. - Zhou et al. [20252] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s" aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132, 20252. - Zhou et al. [20243] Jin Peng Zhou, Charles E Staats, Wenda Li, Christian Szegedy, Kilian Q Weinberger, and Yuhuai Wu. Don’t trust: Verify – grounding LLM quantitative reasoning with autoformalization. In The Twelfth International Conference on Learning Representations, January 20243. URL https://openreview.net/forum?id=V5tdi14ple. - Zhou et al. [20253] Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q Weinberger, Kianté Brantley, and Wen Sun. $q\sharp$ : Provably optimal distributional rl for llm post-training. arXiv preprint arXiv:2502.20548, 20253. - Zhou et al. [20254] Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659, 20254. - Zhou et al. [20244] Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, 634(8032):61–68, 20244. URL https://www.nature.com/articles/s41586-024-07930-y. - Zhou et al. [20255] Li Zhou, Ruijie Zhang, Xunlian Dai, Daniel Hershcovich, and Haizhou Li. Large language models penetration in scholarly writing and peer review. arXiv preprint arXiv:2502.11193, 20255. - Zhou et al. [20245] Ruochen Zhou, Minrui Xu, Shiqi Chen, Junteng Liu, Yunqi Li, LIN Xinxin, Zhengyu Chen, and Junxian He. AI for math or math for AI? on the generalization of learning mathematical problem solving. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, 20245. URL https://openreview.net/forum?id=xlnvZ85CSo. - Zhou et al. [20246] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, January 20246. URL https://openreview.net/forum?id=oKn9c6ytLx. - Zhou et al. [20256] Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493, 20256. - Zhou et al. [20257] Xiaofeng Zhou, Heyan Huang, and Lizi Liao. Debate, reflect, and distill: Multi-agent feedback with tree-structured preference optimization for efficient language model enhancement. arXiv preprint arXiv:2506.03541, 20257. - Zhou et al. [20258] Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. Self-consistency of the internal reward models improves self-rewarding language models. arXiv preprint arXiv:2502.08922, 20258. - Zhou et al. [20259] Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252, 20259. - Zhou et al. [202510] Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478, 202510. - Zhou et al. [202511] Yufa Zhou, Shaobo Wang, Xingyu Dong, Xiangqi Jin, Yifang Chen, Yue Min, Kexin Yang, Xingzhang Ren, Dayiheng Liu, and Linfeng Zhang. Reasoning like an economist: Post-training on economic problems induces strategic generalization in llms. arXiv preprint arXiv:2506.00577, 202511. - Zhou et al. [202512] Zhanke Zhou, Zhaocheng Zhu, Xuan Li, Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, and Bo Han. Landscape of thoughts: Visualizing the reasoning process of large language models. arXiv preprint arXiv:2503.22165, 202512. - Zhou et al. [202513] Zhi Zhou, Tan Yuhao, Zenan Li, Yuan Yao, Lan-Zhe Guo, Xiaoxing Ma, and Yu-Feng Li. Bridging internal probability and self-consistency for effective and efficient llm reasoning. arXiv preprint arXiv:2502.00511, 202513. - Zhu et al. [2025] Bin Zhu, Hailong Yin, Jingjing Chen, and Yu-Gang Jiang. Reasoning models are more easily gaslighted than you think. arXiv preprint arXiv:2506.09677, 2025. - Zhu et al. [20252] Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. Chain-of-thought matters: Improving long-context language models with reasoning path supervision. arXiv preprint arXiv:2502.20790, 20252. - Zhu and Li [2025] Jason Zhu and Hongyu Li. Towards concise and adaptive thinking in large reasoning models: A survey. arXiv preprint arXiv:2507.09662, 2025. - Zhu et al. [20253] Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin, and Lei Sha. Reasoning-to-defend: Safety-aware reasoning can defend large language models from jailbreaking. arXiv preprint arXiv:2502.12970, 20253. - Zhu et al. [20254] King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, et al. Scaling test-time compute for llm agents. arXiv preprint arXiv:2506.12928, 20254. - Zhu et al. [20255] Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian, Xiangru Tang, Heng Ji, et al. Multiagentbench: Evaluating the collaboration and competition of llm agents. arXiv preprint arXiv:2503.01935, 20255. - Zhu et al. [20256] Rongzhi Zhu, Yi Liu, Zequn Sun, Yiwei Wang, and Wei Hu. When can large reasoning models save thinking? mechanistic analysis of behavioral divergence in reasoning. arXiv preprint arXiv:2505.15276, 20256. - Zhu et al. [2024] Tinghui Zhu, Kai Zhang, Jian Xie, and Yu Su. Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning. In First Conference on Language Modeling, July 2024. URL https://openreview.net/forum?id=S1XnUsqwr7. - Zhu et al. [2023] Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4471–4485, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.245. URL https://aclanthology.org/2023.acl-long.245/. - Zhu et al. [20257] Zihao Zhu, Hongbao Zhang, Ruotong Wang, Ke Xu, Siwei Lyu, and Baoyuan Wu. To think or not to think: Exploring the unthinking vulnerability in large reasoning models. arXiv preprint arXiv:2502.12202, 20257. - Zhu et al. [20258] Zihao Zhu, Hongbao Zhang, Mingda Zhang, Ruotong Wang, Guanzong Wu, Ke Xu, and Baoyuan Wu. Bot: Breaking long thought processes of o1-like large language models through backdoor attack. arXiv preprint arXiv:2502.12202, 20258. - Zhuang et al. [2025] Ren Zhuang, Ben Wang, and Shuifa Sun. Accelerating chain-of-thought reasoning: When goal-gradient importance meets dynamic skipping. arXiv preprint arXiv:2505.08392, 2025. - Zhuang et al. [2023] Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Weinan Zhang, and Liu Ting. Through the lens of core competency: Survey on evaluation of large language models. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 88–109, Harbin, China, August 2023. Chinese Information Processing Society of China. URL https://aclanthology.org/2023.ccl-2.8/. - Ziabari et al. [2025] Alireza S Ziabari, Nona Ghazizadeh, Zhivar Sourati, Farzan Karimi-Malekabadi, Payam Piray, and Morteza Dehghani. Reasoning on a spectrum: Aligning llms to system 1 and system 2 thinking. arXiv preprint arXiv:2502.12470, 2025. - Zou et al. [2025] Henry Peng Zou, Zhengyao Gu, Yue Zhou, Yankai Chen, Weizhi Zhang, Liancheng Fang, Yibo Wang, Yangning Li, Kay Liu, and Philip S Yu. Testnuc: Enhancing test-time computing approaches through neighboring unlabeled data consistency. arXiv preprint arXiv:2502.19163, 2025. - Zuo et al. [2025] Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025.