2509.26626

# Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models Abstract Test-time scaling methods improve the capabilities of Large Language Models (LLMs) by increasing the amount of compute used during inference to make a prediction. Inference-time compute can be scaled in parallel by choosing among multiple independent solutions or sequentially through self-refinement. We propose Recursive Self-Aggregation (RSA), a test-time scaling method inspired by evolutionary methods that combines the benefits of both parallel and sequential scaling. Each step of RSA refines a population of candidate reasoning chains through aggregation of subsets to yield a population of improved solutions, which are then used as the candidate pool for the next iteration. RSA exploits the rich information embedded in the reasoning chains – not just the final answers – and enables bootstrapping from partially correct intermediate steps within different chains of thought. Empirically, RSA delivers substantial performance gains with increasing compute budgets across diverse tasks, model families and sizes. Notably, RSA enables Qwen3-4B-Instruct-2507 to achieve competitive performance with larger reasoning models, including DeepSeek-R1 and o3-mini (high), while outperforming purely parallel and sequential scaling strategies across AIME-25, HMMT-25, Reasoning Gym, LiveCodeBench-v6, and SuperGPQA. We further demonstrate that training the model to combine solutions via a novel aggregation-aware reinforcement learning approach yields significant performance gains. Code is available here. 1 Introduction Large language models (LLMs) demonstrate consistent improvements in performance with increasing training compute (Kaplan et al., 2020). Complementarily, test-time scaling strategies, i.e., those that increase compute at inference without altering model parameters, can deliver significant gains in performance (Snell et al., 2025; Jaech et al., 2024). Test-time scaling mechanisms for LLMs can broadly be characterised into two types (§ 2): those that use deeper model rollouts to iteratively improve solutions (e.g., Muennighoff et al., 2025; Zhang et al., 2025a) and those that branch to explore multiple solution paths, then filter or recombine them (e.g., Wang et al., 2023; Weng et al., 2023). We refer to these types as sequential and parallel scaling; some hybrid methods combine the strengths of both approaches (e.g., Yao et al., 2023; Meyerson et al., 2024; Lee et al., 2025). However, a universal and effective test-time-scaling method that allows reuse of promising fragments from multiple candidate solutions is lacking. Self-refinement methods – the quintessential form of sequential scaling – can improve a candidate solution by reusing its own correct parts, but do not leverage the information contained within other candidates. Similarly, parallel scaling methods such as verifier-guided Best-of-N selection can identify the best candidate from a batch, but do not recombine candidates to produce improved solutions. Existing hybrid approaches fail to solve this problem in a general way, often making strong assumptions on the form of reasoning chains (e.g., Meyerson et al., 2024; Hemberg et al., 2024) or requiring external verifiers (e.g., Novikov et al., 2025; Lee et al., 2025). Our work fills this gap in three ways, described in the following paragraphs. Self-aggregation. We study a general way to improve LLM reasoning chains through self-aggregation: providing the model with the query and a set of candidate solutions and prompting it to produce an improved solution. Such an approach, which relies on the implicit verification abilities of the model (Weng et al., 2023), can use the rich information contained within the reasoning chains: for example, a reasoning trace that results in an incorrect answer to a problem can have correct intermediate steps that can be reused in the aggregated solution (Appendix E). Such aggregation methods are explored in multiple concurrent works (e.g., Li et al., 2025; Wang et al., 2025; Zhao et al., 2025) and are promising directions for test-time scaling. Recursive self-aggregation. While self-aggregation can be used as a one-time procedure to combine candidate solutions, our proposed algorithm, Recursive Self-Aggregation (RSA, § 3), goes further: integrating aggregation steps into a self-improvement loop motivated by evolutionary algorithms. RSA maintains a population of candidate solutions and iteratively recombines subsets of the population to produce a new population of improved solutions (Fig. 3). This sequential refinement enables deeper reasoning by allowing the model to revisit its solutions and make multiple attempts at correcting errors. RSA maintains a candidate population larger than the aggregation set size and can therefore jointly consider significantly more proposals than single-step aggregation, which is constrained by the model’s effective context length. Unlike other evolutionary methods, RSA requires no external verification and can be seamlessly integrated into any LLM inference pipeline to improve reasoning. Aggregation-aware RL. During post-training, LLMs are trained with reinforcement learning (RL) to improve their reasoning ability (Jaech et al., 2024; Guo et al., 2025). RL training does not account for test-time scaling, which in this case is the task of aggregating multiple reasoning chains. In fact, we observe that standard RL fine-tuning can even degrade performance relative to the base model when combined with test-time aggregation (§ 5.4). To address this, we propose an aggregation-aware RL approach using a simple data-augmentation strategy to train LLMs to aggregate solutions (§ 4). We perform extensive experiments to demonstrate the effectiveness of RSA across diverse tasks, such as AIME-25, HMMT-25, LiveCodeBench, Reasoning Gym, and SuperGPQA with various base models (§ 5). RSA bridges the gap between the lightweight Qwen3-4B-Instruct-2507 and much larger reasoning models like DeepSeek-R1 and o3-mini (high) (Fig. 1). Our results also show that aggregation-aware RL significantly improves performance with RSA compared to naïve RL training (§ 5.4). We rigorously analyze the factors driving RSA performance and provide practical recommendations to enable deeper test-time thinking under compute constraints (§ 5.3). <details> <summary>2509.26626v1/x1.png Details</summary>  ### Visual Description # Technical Document Extraction: Image Analysis ## Chart 1: HMMMT-25 ### Components - **Title**: HMMMT-25 - **X-Axis Labels**: - o3-mini (high) - DeepSeek-R1 - Owen3-4B-Instruct (Base) - Base + RSA (Ours) - Base + RSA + RL (Ours) - **Y-Axis Label**: Pass@1 (0–80 range) - **Legend**: - Teal: Base - Orange: Base + RSA - **Data Points**: - o3-mini (high): 67.5 (Teal) - DeepSeek-R1: 41.7 (Teal) - Owen3-4B-Instruct (Base): 27.2 (Teal) - Base + RSA (Ours): 47.6 (Orange) - Base + RSA + RL (Ours): 55.5 (Orange) - **Trend**: Scores increase from Owen3-4B-Instruct (27.2) to Base + RSA + RL (55.5). ## Chart 2: Reasoning Gym Games ### Components - **Title**: Reasoning Gym Games - **X-Axis Labels**: - o3-mini (high) - DeepSeek-R1 - Owen3-4B-Instruct (Base) - Base + RSA (Ours) - Base + RSA + RL (Ours) - **Y-Axis Label**: Pass@1 (0–80 range) - **Legend**: - Teal: Base - Orange: Base + RSA - **Data Points**: - o3-mini (high): 69.9 (Teal) - DeepSeek-R1: 54.8 (Teal) - Owen3-4B-Instruct (Base): 53.9 (Teal) - Base + RSA (Ours): 69.0 (Orange) - Base + RSA + RL (Ours): 70.6 (Orange) - **Trend**: Scores rise from Owen3-4B-Instruct (53.9) to Base + RSA + RL (70.6). ## Chart 3: LiveCodeBench-v6 ### Components - **Title**: LiveCodeBench-v6 - **X-Axis Labels**: Pass@1 (0–100 range) - **Y-Axis Labels**: - Qwen3 Instruct (4B) - Qwen3 Instruct (30B) - GPT-OSS Medium (20B) - **Legend**: - Teal: Base - Orange: Base + RSA - **Data Points**: - Qwen3 Instruct (4B): - Base: 44.9 (Teal) - Base + RSA: +7.1 (Orange) - Qwen3 Instruct (30B): - Base: 60.0 (Teal) - Base + RSA: +7.1 (Orange) - GPT-OSS Medium (20B): - Base: 74.4 (Teal) - Base + RSA: +5.6 (Orange) - **Trend**: Base + RSA improves all models, with Qwen3 Instruct (4B) showing the largest absolute gain (+7.1). ## Chart 4: AIME-25 ### Components - **Title**: AIME-25 - **X-Axis Labels**: Pass@1 (0–100 range) - **Y-Axis Labels**: - Nemotron Nano (9B) - Qwen3 Instruct (4B) - Qwen3 Instruct (30B) - Qwen3 Thinking (4B) - GPT-OSS Medium (20B) - **Legend**: - Teal: Base - Orange: Base + RSA - **Data Points**: - Nemotron Nano (9B): - Base: 40.8 (Teal) - Base + RSA: +32.1 (Orange) - Qwen3 Instruct (4B): - Base: 44.9 (Teal) - Base + RSA: +29.9 (Orange) - Qwen3 Instruct (30B): - Base: 57.7 (Teal) - Base + RSA: +27.2 (Orange) - Qwen3 Thinking (4B): - Base: 65.0 (Teal) - Base + RSA: +19.4 (Orange) - GPT-OSS Medium (20B): - Base: 67.8 (Teal) - Base + RSA: +22.4 (Orange) - **Trend**: Base + RSA significantly boosts scores across all models, with Nemotron Nano (9B) showing the highest relative improvement (+32.1). ## Spatial Grounding - **Legend Position**: Bottom-right corner of all charts. - **Color Consistency**: - Teal consistently represents "Base" across all charts. - Orange consistently represents "Base + RSA" across all charts. ## Key Observations 1. **Model Performance**: - o3-mini (high) and Qwen3 Instruct (30B) consistently achieve the highest Pass@1 scores in their respective charts. 2. **RSA Impact**: - Base + RSA improves performance in all models, with the largest gains observed in AIME-25 (e.g., +32.1 for Nemotron Nano). 3. **RL Enhancement**: - Adding RL (Reinforcement Learning) further improves scores in HMMMT-25 and Reasoning Gym Games (e.g., +7.9 for Base + RSA + RL vs. Base + RSA in Reasoning Gym Games). ## Language Notes - All text is in English. No non-English content detected. </details> Figure 1: Recursive Self-Aggregation (RSA, § 3) substantially improves Pass@ $1$ across tasks and model architectures. RSA enables the much smaller Qwen3-4B-Instruct-2507 to match the performance of larger reasoning models such as DeepSeek-R1 and o3-mini (high). These gains are further amplified through our proposed aggregation-aware RL framework (§ 4). 2 A taxonomy of test-time scaling methods Test-time scaling refers to methods that obtain predictions using a static LLM with a larger number of model evaluations than that required by simply prompting for an answer. These methods significantly improve performance without modifying model weights (Snell et al., 2025; Zhang et al., 2025b), effectively using the model as a component in an external optimization or inference framework, at the cost of increased computation. A well-designed test-time scaling framework should yield monotonic improvements in performance as compute budgets increase, similar to scaling laws for pretraining (Kaplan et al., 2020; Snell et al., 2025). Most methods rely on some kind of verification, whether implicit or explicit, incorporated within a sequential or parallel control flow. In this section, we review the literature on test-time scaling in LLMs and provide a taxonomy of test-time scaling frameworks based on the verification strategy and control flow they employ, illustrated in Fig. 2. Building on this, we then introduce our proposed approach, Recursive Self-Aggregation (RSA) in § 3. See Appendix A for a discussion of broader related work. 2.1 Verification strategy External verification. Any external optimization procedure requires a mechanism to assess the quality of proposed solutions. In domains such as code or math, evaluation can often be performed exactly using external tools (e.g., compilation and execution (Gao et al., 2023b)). When such verifiers are unavailable, inference-time feedback is instead obtained via learned reward models, trained on preference data or correctness signals derived from reasoning chains (Cobbe et al., 2021; Ouyang et al., 2022; Snell et al., 2025). This verifier feedback, exact or learned, makes it possible to improve solution quality as more compute is allocated: a simple strategy is Best-of-N (Gao et al., 2023a), where $N$ candidates are generated and the highest-reward solution is selected. Self-verification. LLMs exhibit a generation-verification gap: they are more reliable at judging correctness of solutions than producing them (Li et al., 2024). This property can be exploited to enable test-time scaling by using the LLM as a verifier of its own outputs (e.g., Madaan et al., 2023; Weng et al., 2023). The LLM can also be further fine-tuned to enhance its verification ability (Zhang et al., 2025a), but we regard this as a form of external verification since it requires learning a verifier. Implicit verification. Some methods bypass explicit verification by relying on the LLM to generate improved solutions, effectively performing verification of solutions without scoring them. For example, majority voting (Wang et al., 2023) works on the assumption of self-consistency: that the model produces correct answers more consistently than incorrect ones. Similarly, frameworks for self-refinement (e.g., Madaan et al., 2023) iteratively refine a reasoning chain without being explicitly prompted for verification. RSA falls within this category: rather than explicitly verifying each solution, the model implicitly checks intermediate steps across multiple reasoning chains, allowing it to correct errors and generate improved solutions. 2.2 Reasoning control flow <details> <summary>2509.26626v1/x2.png Details</summary>  ### Visual Description # Technical Document Extraction: Scaling Methodologies Diagram ## Overview The image illustrates three scaling methodologies for processing prompts through neural network transformations. Each methodology is represented as a flowchart with distinct architectural patterns. --- ## 1. Parallel Scaling ### Components: - **Input**: Prompt → Neural Network (interconnected nodes) - **Transformations**: - τ₁, τ₂, τ₃ (parallel processing branches) - **Output**: Combined → τ (final output) ### Flow: 1. Prompt enters a neural network 2. Network output splits into three parallel transformation paths: - τ₁ (bottom) - τ₂ (middle) - τ₃ (top) 3. Results from all three transformations are combined 4. Final output labeled as τ --- ## 2. Sequential Scaling ### Components: - **Input**: Prompt → Neural Network - **Transformations**: - τ^(1), τ^(2), ..., τ^(T) (temporal sequence) - **Output**: Final τ^(T) ### Flow: 1. Prompt enters neural network 2. Sequential application of transformations: - τ^(1) → τ^(2) → ... → τ^(T) 3. Each transformation builds on previous output 4. Final output labeled as τ^(T) --- ## 3. Hybrid Scaling ### Components: - **Input**: Prompt → Neural Network - **Transformations**: - τ^(1), τ^(2), ..., τ^(T-1) (temporal sequence with parallel processing) - **Output**: Final τ^(T) ### Flow: 1. Prompt enters neural network 2. Temporal sequence of transformations: - Stage 1: τ^(1) applied to all three parallel paths (τ₁^(1), τ₂^(1), τ₃^(1)) - Stage 2: τ^(2) applied to all three parallel paths (τ₁^(2), τ₂^(2), τ₃^(2)) - ... - Stage T-1: τ^(T-1) applied to all three parallel paths 3. Final transformation τ^(T) applied to combined outputs 4. Final output labeled as τ^(T) --- ## Key Observations 1. **Notation Consistency**: - Lowercase τ (τ₁, τ₂, τ₃) = parallel processing units - Superscript τ^(n) = temporal sequence stages 2. **Architectural Patterns**: - Parallel: Horizontal processing - Sequential: Vertical stacking - Hybrid: Combination of both 3. **Temporal Progression**: - Sequential and Hybrid methods show explicit time-dependent progression (τ^(1) → τ^(T)) 4. **Combination Logic**: - Parallel method uses explicit "Combine" step - Hybrid method implies combination through sequential stages --- ## Diagram Structure - **X-Axis**: Represents processing stages (parallel branches or temporal sequence) - **Y-Axis**: Represents transformation depth (individual τ units or temporal stages) - **Color Coding**: No explicit color legend present; all components use default diagram colors --- ## Technical Implications 1. **Parallel Scaling**: - Best for independent processing paths - Requires explicit combination mechanism 2. **Sequential Scaling**: - Suitable for time-dependent transformations - Cumulative effect across stages 3. **Hybrid Scaling**: - Balances parallelism and temporal progression - Complex implementation but potentially optimal performance --- ## Limitations - No quantitative performance metrics provided - No implementation details for transformation functions - Assumes perfect parallelization capabilities </details> Figure 2: Overview of test-time scaling control flows. Parallel methods generate multiple candidates and select using a verification mechanism. Sequential methods iteratively refines a chain, correcting previous mistakes. Hybrid methods combine parallel branching with sequential refinement. Parallel scaling. These strategies generate multiple independent reasoning chains in parallel and then combine them to yield the final answer. Typical procedures for combination include majority voting, Best-of-N selection or single-step aggregation of the sampled proposals (Wang et al., 2023; Snell et al., 2025; Li et al., 2025). These strategies rely on the inherent diversity in sampling from the LLM, allowing parallel proposals to efficiently explore the search space and allow optimal GPU memory utilization. These algorithms embody the philosophy of breadth-first thinking. Sequential scaling. Purely parallel scaling sacrifices the ability to think deeply, which is often crucial for multi-step reasoning tasks that cannot be solved efficiently through guess-and-check. Sequential scaling instead increases the number of iterative model evaluations to produce higher-quality solutions, for example, by inducing a model to correct errors in its reasoning (Muennighoff et al., 2025) or simply increasing the number of latent reasoning tokens it can generate. While these strategies generally require more computation time than parallel ones (given sufficient memory budgets), they are well suited to complex reasoning problems requiring depth-first thinking. However, the lack of branching in such methods limits their ability to explore alternative continuations of promising solution paths, making the model prone to persisting in an unproductive reasoning chain. Sequential scaling also leaves excess GPU memory underutilized. Hybrid scaling. Sequential and parallel scaling strategies can be combined in hybrid frameworks that draw on the strengths of both paradigms. These methods make efficient use of GPU memory by evaluating many candidate solutions in parallel, while also incorporating sequential depth to iteratively refine and improve the batch of solutions. One strong class of hybrid methods uses LLMs as components within a genetic algorithm loop (e.g., Novikov et al., 2025; Lee et al., 2025; Meyerson et al., 2024), all using external verification to score candidates. Another example of hybrid test-time scaling is Mixture-of-Agents (Wang et al., 2024), where an ensemble of LLMs generates improved proposals that are aggregated by a strong model into the seed solution for the next iteration. Our method, RSA, is also a hybrid scaling algorithm: like Mixture-of-Agents, it relies on recursive aggregation, but it further maintains a population of candidate solutions larger than the aggregation batch size, similar to evolutionary algorithms, while only using a single LLM. By aggregating random subsets of this population, RSA preserves diversity in the candidate pool, which is critical when all proposals and aggregations are produced by the same model (as studied in § 5.3 and C). <details> <summary>2509.26626v1/x3.png Details</summary>  ### Visual Description # Technical Document Extraction: Image Analysis ## Diagram Description The image depicts a **flowchart** representing a multi-stage language model (LLM) processing pipeline. Below is a detailed breakdown of components, labels, and flow: ### Key Components 1. **Left Block**: Labeled `LLM` (Language Model) - Represents the initial processing unit. - Outputs a population `P_t` (temporal subscript indicates time step `t`). 2. **Central Aggregation Stage**: - **Input**: Population `P_t` from the left LLM. - **Process**: - Generates **Aggregation Sets** `S_t` (temporal subscript `t`). - Each set contains 8 candidate solutions (e.g., `τ₁^(t)`, `τ₂^(t)`, ..., `τ₈^(t)`). - **Output**: - A **Question** node (central yellow box). - The Question node feeds back into the LLM for iterative refinement. 3. **Right Block**: Labeled `LLM` (Language Model) - Receives the Question node as input. - Outputs a refined population `P_{t+1}` (temporal subscript `t+1`). ### Flowchart Arrows - **Left to Center**: - `P_t` → `Aggregation Sets S_t` (8 candidate solutions per set). - **Center to Right**: - `Question` → `P_{t+1}` (refined population). ### Labels and Annotations - **Axis Titles**: - `Population P_t` (left), `Aggregation Sets S_t` (center), `Population P_{t+1}` (right). - **Legend**: - No explicit legend present. Colors differentiate components: - **Yellow**: Question node. - **Blue**: Aggregation sets (`S_t`). - **Gray**: LLM blocks. - **White**: Population nodes (`P_t`, `P_{t+1}`). ## Text Box Content: Aggregation Prompt The right-side text box contains a **prompt** for aggregating candidate solutions. Transcribed verbatim: </details> Figure 3: RSA generates a population of $N$ solutions for a given prompt and recursively updates them over $T$ steps. Each update step subsamples $K$ distinct solutions from the current population and generates an improved solution with the aggregation prompt. See Appendix B for algorithm pseudo-code. 3 Evolving thoughts using recursive self-aggregation We present Recursive Self-Aggregation (RSA), a hybrid test-time scaling procedure designed to improve the model’s performance without complex scaffolding or using external verifiers. It frames reasoning as a form of evolutionary process where candidate reasoning chains are iteratively refined through self-aggregation, inspired by the crossover and mutation steps in genetic algorithms. RSA is simple to implement and leads to substantial improvements in reasoning abilities across different models and tasks, when compared to pure sequential or parallel scaling (§ 5). Fig. 3 illustrates the core components of RSA, which we describe below. The algorithm is also written in Appendix B. Given a query ${\mathbf{x}}$ and a pretrained LLM $p_{\theta_{\textrm{ref}}}$ , RSA maintains a population of $N$ candidate solutions ${\mathcal{P}}_{t}$ at each step $t$ . The model is provided with the question and a subset of $K$ solutions from this population, and prompted to produce an improved population of solutions ${\mathcal{P}}_{t+1}$ . The procedure is described in detail below: 1. Population of trajectories. At any given step $t$ , RSA maintains a population of $N$ independent candidate solutions ${\mathcal{P}}_{t}:=\{\tau_{1}^{(t)},...\tau_{N}^{(t)}\}$ . The initial population ${\mathcal{P}}_{1}$ is generated by sampling $N$ responses for query ${\mathbf{x}}$ using the LLM $p_{\theta_{\textrm{ref}}}$ : $$ \tau_{i}^{(1)}\sim p_{\theta_{\textrm{ref}}}(\,\cdot\mid{\mathbf{x}}),\quad\mathcal{P}_{1}=\{\tau_{1}^{(1)},\dots,\tau_{N}^{(1)}\}. \tag{1} $$ 1. Subsampling. We form $N$ aggregation sets of $K$ candidates, where each set is sampled uniformly without replacement from the population: $$ \mathcal{S}_{t}=\{S_{1}^{(t)},S_{2}^{(t)},\dots,S_{N}^{(t)}\},\quad S_{i}^{(t)}\subseteq\mathcal{P}_{t},\ |S_{i}^{(t)}|=K. \tag{2} $$ 1. Aggregation. Each set ${\mathcal{S}}_{i}^{(t)}$ and the query ${\mathbf{x}}$ is formatted using an aggregation prompt directing the LLM $p_{\theta_{\textrm{ref}}}$ to generate a refined response $\tau_{i}^{(t+1)}$ , forming a new population of candidates ${\mathcal{P}}_{t+1}$ : $$ \tau_{i}^{(t+1)}\sim p_{\theta_{\textrm{ref}}}(\,\cdot\mid S_{i}^{(t)},{\mathbf{x}}),\quad\mathcal{P}_{t+1}=\{\tau_{1}^{(t+1)},\dots,\tau_{N}^{(t+1)}\}. \tag{3} $$ RSA recursively updates the population ${\mathcal{P}}_{t}$ using (2) and (3) for $t=1,...,T-1$ . This sequential loop is expected to allow errors and inconsistencies to be gradually pruned away during aggregation, while preserving favorable reasoning patterns. Consequently, we expect overall diversity within the population to generally decrease as $t$ increases, accompanied by a monotonic improvement in success rate (See Appendix C). 1. Termination. Given the final population of candidate solutions ${\mathcal{P}}_{T}$ , the solution is obtained either by randomly sampling from this population or by majority voting. We use uniform random sampling in all our experiments, to evaluate our method without any special selection mechanism. Note that the intermediate trajectories $\tau_{i}^{(t)}$ are not required to terminate with complete answers; even partial reasoning chains can provide valuable signal during aggregation. Additionally, the choice of $K$ defines the number of alternative responses to consider for aggregation, with $K=1$ being equivalent to sequential self-refinement (Madaan et al., 2023). In § 5.3, we show that even setting $K=2$ leads to significant improvements over self-refinement, highlighting the importance of combining diverse solutions for improving reasoning performance. See Appendix E for an illustrative example of aggregation. An important consideration is that self-aggregation can lead to loss of diversity due to excessive reuse of reasoning patterns that occur in multiple trajectories in the population. Maintaining a large population size $N$ relative to the aggregation size $K$ helps ensure sufficient diversity for recombination. However, a very large $N$ relative to $K$ can lead to slow convergence of the population as a whole, since high-quality reasoning patterns will require more iterations to dominate the population. We study these tradeoffs in § 5.3. The aggregation prompts we use are provided in Appendix F. 4 Training aggregators with reinforcement learning In addition to the test-time strategies discussed thus far, a model’s reasoning ability can be improved by post-training it with reinforcement learning (RL) (Jaech et al., 2024; Guo et al., 2025). Standard RL post-training encourages a model to produce correct solutions, conditioned on the query (Trung et al., 2024; Lambert et al., 2025). While this improves the model’s ability to directly generate correct solutions, it does not explicitly teach the model how to aggregate multiple candidate solutions. As we show in § 5.4, this mismatch between the training objective and the test-time strategy can result in worse performance compared to the base (reference) model when using RSA. To better align training and inference, we formulate the task of aggregation as an RL problem. The reference model $p_{\theta_{\textrm{ref}}}$ generates a set of candidate reasoning chains given a problem. Next, the model is trained to produce a single correct reasoning chain given the problem and the set of candidate reasoning chains. To achieve this in practice, we create an aggregation-aware training dataset consisting of two types of prompts: (1) Standard prompts, containing only the problem, to train the model to propose good initial candidate reasoning chains; and (2) aggregation prompts, which include the problem along with $K$ candidate solutions from the reference model, formatted with the same aggregation prompt used for RSA; see Appendix F. Consider problem-solution pairs sampled from some dataset $({\mathbf{x}},{\mathbf{y}})\sim{\mathcal{D}}$ , and candidate solutions $\tau$ generated by the model conditioned on the problems. Training with the standard prompts described above corresponds to the standard RL training of LLMs that optimizes the following objective: $$ \displaystyle\max_{\theta}\mathbb{E}_{({\mathbf{x}},{\mathbf{y}})\sim{\mathcal{D}}}\left[\mathbb{E}_{\tau\sim p_{\theta}(\cdot\mid{\mathbf{x}})}\left[r(\tau,{\mathbf{y}})\right]-\beta\,\text{KL}\left(p_{\theta}(\cdot\mid{\mathbf{x}})\,\|\,p_{\theta_{\textrm{ref}}}(\cdot\mid{\mathbf{x}})\right)\right], \tag{4} $$ where $\beta$ controls the optional KL regularization with the reference policy $p_{\theta_{\textrm{ref}}}$ . For the aggregation prompts, we additionally sample $K$ candidates from $p_{\theta_{\textrm{ref}}}$ to construct the aggregation set $S_{0}$ , resulting in the following objective: $$ \displaystyle\max_{\theta}\mathbb{E}_{({\mathbf{x}},{\bm{y}})\sim{\mathcal{D}},S_{0}\sim p_{\theta_{\textrm{ref}}}(\cdot\mid{\mathbf{x}})}\left[\mathbb{E}_{\tau\sim p_{\theta}(\cdot\mid{\mathbf{x}},S_{0})}\left[r(\tau,{\bm{y}})\right]-\beta\,\text{KL}\left(p_{\theta}(\cdot\mid{\mathbf{x}},S_{0})\,\|\,p_{\theta_{\textrm{ref}}}(\cdot\mid{\mathbf{x}},S_{0})\right)\right]. \tag{5} $$ This objective can be optimized using any off-the-shelf policy gradient algorithm, such as PPO (Ouyang et al., 2022), GRPO (Shao et al., 2024), or RLOO (Ahmadian et al., 2024), initializing $\theta$ with a copy of the base model parameters $\theta_{\rm ref}$ or using a parameter-efficient fine-tuning technique. We use RLOO in all our experiments (§ 5.4) for its simplicity and good empirical performance. <details> <summary>2509.26626v1/x4.png Details</summary>  ### Visual Description # Model Performance Comparison Across Benchmarks ## Chart Overview This bar chart compares the performance of various AI models across multiple benchmarks, measured by **Pass@1** (percentage of correct answers). The chart is segmented by **benchmark groups** (e.g., AIME-25, Reasoning Gym Cognition + ARC) and **models** (e.g., Nemotron Nano-v2, Qwen3 Instruct). Each bar is further segmented into **components** (Base, Base + RSA, DeepSeek-R1, o3-mini high), with colors mapped to these components via a legend at the bottom. --- ## Key Components ### **Legend** - **Base**: Green (solid) - **Base + RSA**: Orange (solid) - **DeepSeek-R1**: Blue (solid) - **o3-mini (high)**: Dark Blue (solid) **Legend Position**: Bottom of the chart. --- ## Axes - **X-axis**: Models (e.g., Nemotron Nano-v2, Qwen3 Instruct, GPT-OSS Medium, DeepSeek-R1, o3-mini high). - **Y-axis**: Pass@1 (percentage, 0–100%). --- ## Benchmark Groups and Data ### **1. AIME-25** - **Models and Pass@1**: - **Nemotron Nano-v2 (9B)**: +32.1% (Base: 32.1%, Base + RSA: 29.8%, DeepSeek-R1: 27.2%, o3-mini high: 22.4%) - **Qwen3 Instruct (4B)**: +27.2% (Base: 27.2%, Base + RSA: 25.1%, DeepSeek-R1: 22.4%, o3-mini high: 19.4%) - **Qwen3 Instruct (30B)**: +19.4% (Base: 19.4%, Base + RSA: 17.2%, DeepSeek-R1: 14.8%, o3-mini high: 12.6%) - **GPT-OSS Medium (20B)**: +22.4% (Base: 22.4%, Base + RSA: 20.6%, DeepSeek-R1: 18.9%, o3-mini high: 16.3%) - **DeepSeek-R1**: -1% (Base: -1%, Base + RSA: -2.3%, DeepSeek-R1: -3.5%, o3-mini high: -4.7%) - **o3-mini (high)**: +22.4% (Base: 22.4%, Base + RSA: 20.6%, DeepSeek-R1: 18.9%, o3-mini high: 16.3%) **Trend**: - **o3-mini (high)** and **Nemotron Nano-v2** show the highest Pass@1, while **DeepSeek-R1** underperforms with negative values. --- ### **2. Reasoning Gym Cognition + ARC** - **Models and Pass@1**: - **GPT-OSS Medium (20B)**: +1.6% (Base: 1.6%, Base + RSA: 0.8%, DeepSeek-R1: 0.4%, o3-mini high: -0.2%) - **Nemotron Nano-v2 (9B)**: +5.6% (Base: 5.6%, Base + RSA: 4.2%, DeepSeek-R1: 3.0%, o3-mini high: 1.8%) - **Qwen3 Instruct (4B)**: +9.7% (Base: 9.7%, Base + RSA: 7.5%, DeepSeek-R1: 5.3%, o3-mini high: 3.1%) - **Qwen3 Instruct (30B)**: +8.2% (Base: 8.2%, Base + RSA: 6.4%, DeepSeek-R1: 4.7%, o3-mini high: 2.9%) - **DeepSeek-R1**: -1% (Base: -1%, Base + RSA: -0.5%, DeepSeek-R1: -0.3%, o3-mini high: -0.1%) - **o3-mini (high)**: +8.2% (Base: 8.2%, Base + RSA: 6.4%, DeepSeek-R1: 4.7%, o3-mini high: 2.9%) **Trend**: - **Qwen3 Instruct (30B)** and **Qwen3 Instruct (4B)** show the highest Pass@1, while **DeepSeek-R1** again underperforms. --- ### **3. LiveCodeBench-v6** - **Models and Pass@1**: - **Qwen3 Instruct (4B)**: +7.1% (Base: 7.1%, Base + RSA: 5.3%, DeepSeek-R1: 3.8%, o3-mini high: 2.1%) - **Qwen3 Instruct (30B)**: +7.1% (Base: 7.1%, Base + RSA: 5.3%, DeepSeek-R1: 3.8%, o3-mini high: 2.1%) - **GPT-OSS Medium (20B)**: +5.6% (Base: 5.6%, Base + RSA: 4.1%, DeepSeek-R1: 2.9%, o3-mini high: 1.7%) - **o3-mini (high)**: +5.6% (Base: 5.6%, Base + RSA: 4.1%, DeepSeek-R1: 2.9%, o3-mini high: 1.7%) **Trend**: - **Qwen3 Instruct (4B/30B)** and **o3-mini (high)** show similar performance, while **GPT-OSS Medium** lags. --- ### **4. HMMT-25** - **Models and Pass@1**: - **Qwen3 Instruct (4B)**: +20.4% (Base: 20.4%, Base + RSA: 18.2%, DeepSeek-R1: 15.6%, o3-mini high: 12.8%) - **Nemotron Nano-v2 (9B)**: +25.9% (Base: 25.9%, Base + RSA: 23.7%, DeepSeek-R1: 21.4%, o3-mini high: 19.2%) - **Qwen3 Instruct (30B)**: +25.1% (Base: 25.1%, Base + RSA: 22.9%, DeepSeek-R1: 20.6%, o3-mini high: 18.3%) - **GPT-OSS Medium (20B)**: +12.6% (Base: 12.6%, Base + RSA: 10.8%, DeepSeek-R1: 9.4%, o3-mini high: 7.2%) - **DeepSeek-R1**: -1% (Base: -1%, Base + RSA: -0.5%, DeepSeek-R1: -0.3%, o3-mini high: -0.1%) - **o3-mini (high)**: +35.0% (Base: 35.0%, Base + RSA: 32.8%, DeepSeek-R1: 30.4%, o3-mini high: 28.1%) **Trend**: - **o3-mini (high)** achieves the highest Pass@1, while **DeepSeek-R1** again underperforms. --- ### **5. Reasoning Gym Games** - **Models and Pass@1**: - **Nemotron Nano-v2 (9B)**: +8.8% (Base: 8.8%, Base + RSA: 7.2%, DeepSeek-R1: 5.6%, o3-mini high: 3.4%) - **GPT-OSS Medium (20B)**: +14.0% (Base: 14.0%, Base + RSA: 11.6%, DeepSeek-R1: 9.8%, o3-mini high: 7.2%) - **Qwen3 Instruct (4B)**: +15.1% (Base: 15.1%, Base + RSA: 12.8%, DeepSeek-R1: 10.6%, o3-mini high: 8.2%) - **Qwen3 Instruct (30B)**: +20.6% (Base: 20.6%, Base + RSA: 17.4%, DeepSeek-R1: 15.2%, o3-mini high: 12.8%) - **DeepSeek-R1**: -1% (Base: -1%, Base + RSA: -0.5%, DeepSeek-R1: -0.3%, o3-mini high: -0.1%) - **o3-mini (high)**: +20.6% (Base: 20.6%, Base + RSA: 17.4%, DeepSeek-R1: 15.2%, o3-mini high: 12.8%) **Trend**: - **Qwen3 Instruct (30B)** and **o3-mini (high)** lead, while **DeepSeek-R1** underperforms. --- ### **6. SuperGPQA** - **Models and Pass@1**: - **Qwen3 Instruct (4B)**: +5.5% (Base: 5.5%, Base + RSA: 4.2%, DeepSeek-R1: 3.0%, o3-mini high: 1.8%) - **Qwen3 Instruct (30B)**: +6.7% (Base: 6.7%, Base + RSA: 5.3%, DeepSeek-R1: 4.1%, o3-mini high: 2.9%) **Trend**: - **Qwen3 Instruct (30B)** outperforms the 4B version, with **o3-mini (high)** showing the highest Pass@1. --- ## Spatial Grounding and Color Verification - **Legend Colors**: - **Base (Green)** matches the green segments in all bars. - **Base + RSA (Orange)** matches the orange segments. - **DeepSeek-R1 (Blue)** matches the blue segments. - **o3-mini (high) (Dark Blue)** matches the dark blue segments. **Legend Position**: Bottom of the chart, aligned with the x-axis. --- ## Component Isolation - **Header**: Title "Model Performance Comparison Across Benchmarks". - **Main Chart**: Grouped bars by benchmark, with segmented bars for models and components. - **Footer**: Legend and axis labels. --- ## Data Table Reconstruction | Benchmark | Model | Base (%) | Base + RSA (%) | DeepSeek-R1 (%) | o3-mini (high) (%) | |-------------------------|------------------------|----------|----------------|------------------|--------------------| | AIME-25 | Nemotron Nano-v2 (9B) | 32.1 | 29.8 | 27.2 | 22.4 | | AIME-25 | Qwen3 Instruct (4B) | 27.2 | 25.1 | 22.4 | 19.4 | | AIME-25 | Qwen3 Instruct (30B) | 19.4 | 17.2 | 14.8 | 12.6 | | AIME-25 | GPT-OSS Medium (20B) | 22.4 | 20.6 | 18.9 | 16.3 | | AIME-25 | DeepSeek-R1 | -1 | -2.3 | -3.5 | -4.7 | | AIME-25 | o3-mini (high) | 22.4 | 20.6 | 18.9 | 16.3 | | Reasoning Gym Cognition + ARC | GPT-OSS Medium (20B) | 1.6 | 0.8 | 0.4 | -0.2 | | Reasoning Gym Cognition + ARC | Nemotron Nano-v2 (9B) | 5.6 | 4.2 | 3.0 | 1.8 | | Reasoning Gym Cognition + ARC | Qwen3 Instruct (4B) | 9.7 | 7.5 | 5.3 | 3.1 | | Reasoning Gym Cognition + ARC | Qwen3 Instruct (30B) | 8.2 | 6.4 | 4.7 | 2.9 | | Reasoning Gym Cognition + ARC | DeepSeek-R1 | -1 | -0.5 | -0.3 | -0.1 | | Reasoning Gym Cognition + ARC | o3-mini (high) | 8.2 | 6.4 | 4.7 | 2.9 | | LiveCodeBench-v6 | Qwen3 Instruct (4B) | 7.1 | 5.3 | 3.8 | 2.1 | | LiveCodeBench-v6 | Qwen3 Instruct (30B) | 7.1 | 5.3 | 3.8 | 2.1 | | LiveCodeBench-v6 | GPT-OSS Medium (20B) | 5.6 | 4.1 | 2.9 | 1.7 | | LiveCodeBench-v6 | o3-mini (high) | 5.6 | 4.1 | 2.9 | 1.7 | | HMMT-25 | Qwen3 Instruct (4B) | 20.4 | 18.2 | 15.6 | 12.8 | | HMMT-25 | Nemotron Nano-v2 (9B) | 25.9 | 23.7 | 21.4 | 19.2 | | HMMT-25 | Qwen3 Instruct (30B) | 25.1 | 22.9 | 20.6 | 18.3 | | HMMT-25 | GPT-OSS Medium (20B) | 12.6 | 10.8 | 9.4 | 7.2 | | HMMT-25 | DeepSeek-R1 | -1 | -0.5 | -0.3 | -0.1 | | HMMT-25 | o3-mini (high) | 35.0 | 32.8 | 30.4 | 28.1 | | Reasoning Gym Games | Nemotron Nano-v2 (9B) | 8.8 | 7.2 | 5.6 | 3.4 | | Reasoning Gym Games | GPT-OSS Medium (20B) | 14.0 | 11.6 | 9.8 | 7.2 | | Reasoning Gym Games | Qwen3 Instruct (4B) | 15.1 | 12.8 | 10.6 | 8.2 | | Reasoning Gym Games | Qwen3 Instruct (30B) | 20.6 | 17.4 | 15.2 | 12.8 | | Reasoning Gym Games | DeepSeek-R1 | -1 | -0.5 | -0.3 | -0.1 | | Reasoning Gym Games | o3-mini (high) | 20.6 | 17.4 | 15.2 | 12.8 | | SuperGPQA | Qwen3 Instruct (4B) | 5.5 | 4.2 | 3.0 | 1.8 | | SuperGPQA | Qwen3 Instruct (30B) | 6.7 | 5.3 | 4.1 | 2.9 | --- ## Notes - **Negative Values**: Indicate underperformance relative to the base model (e.g., DeepSeek-R1 in AIME-25). - **Component Contributions**: Each bar’s segments sum to the total Pass@1 for that model on the benchmark. - **Model Variants**: Qwen3 Instruct has 4B and 30B versions; GPT-OSS Medium is 20B; DeepSeek-R1 and o3-mini (high) are distinct configurations. This structured analysis ensures all textual and numerical data from the chart is captured, with trends and component relationships clearly articulated. </details> Figure 4: RSA significantly improves Pass@ $1$ across math, code, general reasoning, and knowledge recall tasks. We observe consistent gains across diverse model families, including standard instruction-tuned models and long CoT “thinking” models. Further details provided in § D.2. Table 1: We report Pass@1 scores for RSA and other test-time scaling baselines. RSA results obtained with aggregation size $K=4$ , population size $N=16$ , run for $T=10$ steps. Majority-voting and rejection-sampling are budget-matched with RSA. Results are averaged over 4 seeds for all tasks except SuperGPQA, where we use 1 seed. Further details in Appendix D. | Method $\downarrow$ Task $→$ Reference | Math reasoning AIME-25 43.91 | Code gen. HMMT-25 27.17 | General reasoning / planning LiveCodeBench-v6 49.63 | Knowledge recall RG Games 53.88 | RG Cognition+ARC 42.31 | SuperGPQA 41.85 | | --- | --- | --- | --- | --- | --- | --- | | Rejection sampling | 48.23 | 32.76 | 50.90 | 56.41 | 45.11 | 46.18 | | Self-refinement | 53.33 | 39.17 | 51.40 | 65.50 | 49.17 | 43.5 | | Majority voting | 68.33 | 35.00 | – | 65.15 | 46.07 | 48.2 | | Self-aggregation (RSA, $T=1$ ) | 56.51 $±$ 3.06 | 36.15 $±$ 1.26 | 51.94 $±$ 0.15 | 65.21 $±$ 0.98 | 48.63 $±$ 0.81 | 45.91 | | RSA | 73.18 $±$ 2.20 | 47.55 $±$ 1.00 | 56.72 $±$ 0.65 | 68.98 $±$ 1.10 | 51.96 $±$ 2.24 | 47.39 | 5 Experiments We first demonstrate the effectiveness of RSA as a test-time scaling strategy in § 5.1 and § 5.2 through comprehensive evaluations on math, code generation, general reasoning, and knowledge recall benchmarks. In § 5.3, we analyze how RSA’s three key parameters – the aggregation set size $K$ , the population size $N$ , and the number of sequential steps $T$ – contribute to its success. Finally, in § 5.4, we show that aggregation-aware RL training can further enhance RSA’s performance. Tasks. We evaluate RSA across four benchmark categories, providing further details in § D.1: - Math. We use AIME-25 and HMMT-25 from MathArena (Balunović et al., 2025), each containing $30$ challenging competition-level math problems. - General reasoning. We construct two datasets with $100$ problems each from Reasoning Gym (Stojanovski et al., 2025), using tasks from the games category, and cognition + ARC categories. - Code generation. We use LiveCodeBench-v6 (Jain et al., 2024) which contains $1055$ problems. - Knowledge-based reasoning. We use SuperGPQA (M-A-P Team et al., 2025), a graduate-level knowledge-based reasoning benchmark, to test effectiveness of RSA on tasks requiring factual recall. Given the large dataset size, we evaluate on $1000$ randomly chosen multiple-choice questions. 5.1 RSA outperforms other test-time scaling methods We benchmark RSA against sequential and parallel test-time scaling methods with Qwen3-4B-Instruct-2507 as the base model. To ensure consistency across tasks, we fix the population size to $N=16$ , the aggregation set size to $K=4$ , and the number of recursive updates to $T=10$ . Results are averaged over $4$ seeds, except for SuperGPQA where we report a single seed due to computational constraints. For fairness, we restrict comparisons to methods that require no additional training or external verifiers. Further experimental details are provided in § D.3. Sequential baselines. We consider $T$ -step self-refinement (Madaan et al., 2023), which corresponds to RSA with $K=1$ and $N=1$ , which we run for $T=10$ steps. Parallel baselines. We evaluate majority voting (Wang et al., 2023) and rejection sampling with self-verification (Weng et al., 2023), budget-matched with RSA by using $N× T$ generations. We also include single-step self-aggregation (Li et al., 2025), equivalent to RSA with $K=4$ and $T=1$ . Table 1 reports Pass@1 results across all benchmarks, showing that RSA consistently outperforms both sequential and parallel baselines. Against self-refinement, RSA achieves higher performance, demonstrating that aggregating multiple solutions provides clear advantages over refining a single one. Notably, RSA with $T=10$ outperforms its single-step variant ( $T=1$ ), highlighting the benefits of recursive aggregation. When compared to parallel methods, RSA achieves superior results on all tasks except SuperGPQA, where majority voting is particularly effective due to the multiple-choice answer format. We omit majority voting on LiveCodeBench-v6 since code solutions rarely coincide exactly, and refer to Appendix D for further details. 5.2 RSA yields consistent gains across different models We apply RSA to a diverse set of instruction-tuned models spanning a wide range of parameter counts, architectures, and reasoning abilities, including long chain-of-thought “thinking” models, sparse Mixture-of-Experts (MoE) architectures, and hybrid state-space models. Specifically, among instruction-tuned models, we consider Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 (Yang et al., 2025), while among long chain-of-thought thinking models, we include Qwen3-4B-Thinking-2507, gpt-oss-20b (medium) (OpenAI et al., 2025), and NVIDIA-Nemotron-Nano-9B-v2 (NVIDIA, 2025). Table 2 provides a detailed overview of the characteristics of the models considered in this work. Fig. 4 shows that RSA leads to substantial improvements on all tasks across a wide range of models. Remarkably, applying RSA to Qwen3-4B-Instruct-2507, a substantially weaker model, matches and in some cases outperforms strong reasoning models like DeepSeek-R1 and o3-mini (high) without RSA. Taken together, these results establish RSA as a strong and general test-time scaling strategy for a wide variety of tasks. <details> <summary>2509.26626v1/x5.png Details</summary>  ### Visual Description # Technical Document Extraction: RSA Model Performance Analysis ## Chart Structure Overview The image contains **six line charts** arranged in a 2x3 grid, each visualizing the performance of an RSA model across different datasets and aggregation sizes. Key elements include: - **X-axis**: "RSA Step" (1–10) - **Y-axis**: "Pass@1" metric (varying ranges per chart) - **Lines**: Color-coded by aggregation size (K=1, 2, 3, 4) - **Shaded Regions**: Confidence intervals for each line - **Dashed Line**: "Owen3-4B-Instruct" baseline --- ## Legend & Color Mapping - **Location**: Center-top of the grid - **Colors**: - `blue` = K=1 - `orange` = K=2 - `green` = K=3 - `red` = K=4 --- ## Chart-Specific Analysis ### 1. HMMT-25 - **Y-axis Range**: 0.25–0.50 - **Trends**: - All lines slope **upward** (K=4 > K=3 > K=2 > K=1). - "Owen3-4B-Instruct" (dashed black) remains **below all lines**. - **Key Data Points**: - At RSA Step 10: K=4 ≈ 0.49, K=1 ≈ 0.39. ### 2. AIME-25 - **Y-axis Range**: 0.4–0.7 - **Trends**: - All lines slope **upward** (K=4 > K=3 > K=2 > K=1). - "Owen3-4B-Instruct" (dashed black) remains **below all lines**. - **Key Data Points**: - At RSA Step 10: K=4 ≈ 0.68, K=1 ≈ 0.52. ### 3. LiveCodeBench-v6 - **Y-axis Range**: 0.5–0.6 - **Trends**: - All lines slope **upward** (K=4 > K=3 > K=2 > K=1). - "Owen3-4B-Instruct" (dashed black) remains **below all lines**. - **Key Data Points**: - At RSA Step 10: K=4 ≈ 0.59, K=1 ≈ 0.52. ### 4. Reasoning Gym Games - **Y-axis Range**: 0.55–0.7 - **Trends**: - All lines slope **upward** (K=4 > K=3 > K=2 > K=1). - "Owen3-4B-Instruct" (dashed black) remains **below all lines**. - **Key Data Points**: - At RSA Step 10: K=4 ≈ 0.69, K=1 ≈ 0.64. ### 5. Reasoning Gym Cognition + ARC - **Y-axis Range**: 0.425–0.525 - **Trends**: - All lines slope **upward** (K=4 > K=3 > K=2 > K=1). - "Owen3-4B-Instruct" (dashed black) remains **below all lines**. - **Key Data Points**: - At RSA Step 10: K=4 ≈ 0.51, K=1 ≈ 0.47. --- ## Embedded Text Box (Central) </details> Figure 5: Pass@ $1$ vs. RSA steps, for fixed population size $N=16$ , using Qwen3-4B-Instruct-2507. Error bands indicate standard deviation over 4 seeds. Larger $K$ generally improves performance. 5.3 Effect of scaling RSA hyperparameters We perform experiments using Qwen3-4B-Instruct-2507 to answer the following questions: - How does the performance of RSA vary with parallel and sequential scaling parameters? - What underlying mechanisms explain the performance gains of RSA? - How to select the parameters effectively under a compute budget? Monotonic improvement with sequential depth $T$ . Fig. 5 plots the Pass@1 scores over self-aggregation steps for different aggregation set sizes $K$ . Performance improves monotonically on nearly all tasks, with the only significant downward trend on Reasoning Gym Cognition + ARC after five steps. Overall, these results demonstrate that RSA scales effectively with increasing depth. Increasing aggregation size $K$ improves performance. Fig. 5 shows that increasing the aggregation size $K$ improves performance. The largest gain is observed when moving from $K=1$ to $K=2$ , highlighting that aggregating over multiple reasoning chains provides substantial improvement over single-trajectory refinement. We observe diminishing returns beyond $K=3$ on most tasks, possibly because the model cannot effectively attend to very long contexts. Effect of increasing population size $N$ . Next, we study the impact of total population size $N$ , i.e., the number of unique candidates available for aggregation at each step. In Fig. 6, we plot the final Pass@ $1$ scores for different tasks using a fixed aggregation size of $K=4$ and $T=10$ sequential steps, while varying $N∈\{4,8,16,32\}$ . We observe that increasing $N$ always initially improves performance, but scaling $N$ to very large values leads to a small performance drop on AIME-25 and Reasoning Gym Games. We investigate the role of population size further in the following section, where it emerges as the key parameter controlling the asymptotic performance of RSA. <details> <summary>2509.26626v1/x6.png Details</summary>  ### Visual Description # Technical Document: Line Chart Analysis ## Chart Overview The image is a line chart comparing the performance of four different systems across varying population sizes. The y-axis represents the metric "Pass@1" (ranging from 0.35 to 0.75), and the x-axis represents "Population size N" (values: 4, 8, 16, 32). Four data series are plotted with distinct colors and markers, each representing a system. --- ## Axes and Labels - **X-axis (Population size N)**: - Tick marks at 4, 8, 16, 32. - Label: "Population size N". - **Y-axis (Pass@1)**: - Range: 0.35 to 0.75. - Label: "Pass@1". --- ## Legend The legend is located at the bottom of the chart and maps colors to systems: 1. **Blue (●)**: LiveCodeBench-v6 2. **Orange (●)**: RG Games 3. **Pink (●)**: RG Cognition + ARC 4. **Green (●)**: HMMT-25 --- ## Data Series and Trends ### 1. **LiveCodeBench-v6 (Blue)** - **Trend**: Starts at ~0.55 (N=4), increases slightly to ~0.57 (N=8), then plateaus. - **Data Points**: - N=4: 0.55 - N=8: 0.57 - N=16: 0.57 - N=32: 0.57 ### 2. **RG Games (Orange)** - **Trend**: Starts at ~0.65 (N=4), peaks at ~0.72 (N=16), then slightly declines. - **Data Points**: - N=4: 0.65 - N=8: 0.70 - N=16: 0.72 - N=32: 0.70 ### 3. **RG Cognition + ARC (Pink)** - **Trend**: Starts at ~0.50 (N=4), increases to ~0.52 (N=16), then stabilizes. - **Data Points**: - N=4: 0.50 - N=8: 0.51 - N=16: 0.52 - N=32: 0.52 ### 4. **HMMT-25 (Green)** - **Trend**: Starts at ~0.35 (N=4), increases to ~0.48 (N=16), then plateaus. - **Data Points**: - N=4: 0.35 - N=8: 0.45 - N=16: 0.48 - N=32: 0.48 --- ## Confidence Intervals - Shaded regions around each line represent confidence intervals: - **RG Games (Orange)**: Widest intervals (e.g., ~0.65–0.75 at N=4, ~0.70–0.75 at N=16). - **LiveCodeBench-v6 (Blue)**: Narrower intervals (~0.55–0.60 at N=4, ~0.57–0.60 at N=16). - **RG Cognition + ARC (Pink)**: Moderate intervals (~0.50–0.55 at N=4, ~0.51–0.55 at N=16). - **HMMT-25 (Green)**: Narrowest intervals (~0.35–0.40 at N=4, ~0.45–0.50 at N=16). --- ## Spatial Grounding - **Legend Position**: Bottom of the chart. - **Color Consistency**: All lines match their legend labels (e.g., blue = LiveCodeBench-v6). --- ## Key Observations 1. **RG Games** achieves the highest Pass@1 scores, peaking at N=16. 2. **HMMT-25** shows the lowest performance but improves significantly with larger population sizes. 3. **LiveCodeBench-v6** and **RG Cognition + ARC** exhibit similar trends, with minimal variation across population sizes. --- ## Notes - No non-English text is present in the chart. - All data points and trends are visually extracted; no numerical tables are embedded. </details> Figure 6: Pass@ $1$ at $T=10$ over population size $N$ (fixed $K=4$ ). Pass@N as a predictor of asymptotic performance. The Pass@ $N$ score for a population of $N$ solutions is equal to $1$ if at least one final answer out of the $N$ is correct. The top row of Fig. 7 shows the average Pass@ $N$ score of the population across iterations of RSA for different values of $N$ . For the math tasks (AIME-25, HMMT-25), Pass@ $N$ remains relatively stable, whereas for LiveCodeBench-v6 it decreases by $6$ - $8\%$ . As expected, larger $N$ yields a higher baseline Pass@ $N$ score. We find that the gap between Pass@ $N$ and Pass@ $1$ is a useful predictor of the ‘aggregability’ of a set of solutions. The bottom row of Fig. 7 shows this gap over iterations. As the number of RSA iterations grows, Pass@ $1$ converges to Pass@ $N$ , which acts as an upper bound on the performance. The Pass@ $N$ $-$ Pass@ $1$ gap consistently drops faster for smaller $N$ with fixed aggregation set size $K$ . Intuitively, RSA preserves good reasoning patterns in the population, and high-quality reasoning chains can mix within the population in fewer aggregation iterations if the population size is small. Therefore, a larger population size $N$ enables better asymptotic performance, but requires either more sequential iterations $T$ or faster mixing via larger aggregation size $K$ . See Appendix C for a population diversity analysis over RSA steps, which further validates these findings. <details> <summary>2509.26626v1/x7.png Details</summary>  ### Visual Description # Technical Document Extraction: Multi-Chart Analysis ## Image Structure The image contains a 2x3 grid of charts, each with two subplots: 1. **Top Row**: Line charts showing `Pass@N` vs `RSA Step` 2. **Bottom Row**: Scatter plots showing `Pass@1` vs `Population size N` ### Legend Placement - All legends are located at the **bottom center** of each chart - Color coding: - Blue = Population size 4 - Orange = Population size 8 - Green = Population size 16 - Red = Population size 32 --- ## Chart Analysis ### 1. AIME-25 #### Top Chart: Pass@N vs RSA Step - **X-axis**: RSA Step (2, 4, 6, 8, 10) - **Y-axis**: Pass@N (0.6 to 0.75) - **Trends**: - Blue (N=4): Flat line at ~0.6 - Orange (N=8): Slight fluctuation between 0.65-0.7 - Green (N=16): Stable ~0.7 - Red (N=32): Slight decline from 0.75 to 0.7 - **Key Data Points**: - N=4: [2,0.6], [4,0.6], [6,0.6], [8,0.6], [10,0.6] - N=8: [2,0.65], [4,0.68], [6,0.67], [8,0.69], [10,0.68] - N=16: [2,0.72], [4,0.71], [6,0.71], [8,0.71], [10,0.71] - N=32: [2,0.75], [4,0.74], [6,0.73], [8,0.72], [10,0.71] #### Bottom Chart: Pass@1 vs Population Size N - **X-axis**: Population size N (4, 8, 16, 32) - **Y-axis**: Pass@1 (0 to 0.3) - **Trends**: - All lines show steep decline with increasing N - N=4: Starts at 0.3, ends at 0.05 - N=32: Starts at 0.25, ends at 0.01 - **Key Data Points**: - N=4: [4,0.3], [8,0.2], [16,0.1], [32,0.05] - N=8: [4,0.25], [8,0.18], [16,0.12], [32,0.07] - N=16: [4,0.22], [8,0.15], [16,0.09], [32,0.04] - N=32: [4,0.2], [8,0.13], [16,0.07], [32,0.02] --- ### 2. HMMT-25 #### Top Chart: Pass@N vs RSA Step - **X-axis**: RSA Step (2, 4, 6, 8, 10) - **Y-axis**: Pass@N (0.35 to 0.55) - **Trends**: - Blue (N=4): Flat line at ~0.35 - Orange (N=8): Slight fluctuation between 0.4-0.45 - Green (N=16): Stable ~0.48 - Red (N=32): Slight decline from 0.52 to 0.5 - **Key Data Points**: - N=4: [2,0.35], [4,0.35], [6,0.35], [8,0.35], [10,0.35] - N=8: [2,0.4], [4,0.42], [6,0.41], [8,0.43], [10,0.42] - N=16: [2,0.48], [4,0.49], [6,0.49], [8,0.49], [10,0.49] - N=32: [2,0.52], [4,0.51], [6,0.5], [8,0.51], [10,0.5] #### Bottom Chart: Pass@1 vs Population Size N - **X-axis**: Population size N (4, 8, 16, 32) - **Y-axis**: Pass@1 (0 to 0.2) - **Trends**: - All lines show steep decline with increasing N - N=4: Starts at 0.2, ends at 0.05 - N=32: Starts at 0.18, ends at 0.02 - **Key Data Points**: - N=4: [4,0.2], [8,0.15], [16,0.08], [32,0.05] - N=8: [4,0.18], [8,0.12], [16,0.06], [32,0.03] - N=16: [4,0.15], [8,0.09], [16,0.04], [32,0.02] - N=32: [4,0.13], [8,0.07], [16,0.03], [32,0.01] --- ### 3. LiveCodeBench-v6 #### Top Chart: Pass@N vs RSA Step - **X-axis**: RSA Step (2, 4, 6, 8, 10) - **Y-axis**: Pass@N (0.55 to 0.65) - **Trends**: - Blue (N=4): Flat line at ~0.55 - Orange (N=8): Slight fluctuation between 0.58-0.62 - Green (N=16): Stable ~0.62 - Red (N=32): Slight decline from 0.65 to 0.6 - **Key Data Points**: - N=4: [2,0.55], [4,0.55], [6,0.55], [8,0.55], [10,0.55] - N=8: [2,0.58], [4,0.6], [6,0.59], [8,0.61], [10,0.6] - N=16: [2,0.62], [4,0.63], [6,0.63], [8,0.63], [10,0.63] - N=32: [2,0.65], [4,0.64], [6,0.63], [8,0.62], [10,0.6] #### Bottom Chart: Pass@1 vs Population Size N - **X-axis**: Population size N (4, 8, 16, 32) - **Y-axis**: Pass@1 (0 to 0.2) - **Trends**: - All lines show steep decline with increasing N - N=4: Starts at 0.18, ends at 0.05 - N=32: Starts at 0.15, ends at 0.02 - **Key Data Points**: - N=4: [4,0.18], [8,0.13], [16,0.07], [32,0.05] - N=8: [4,0.16], [8,0.1], [16,0.05], [32,0.03] - N=16: [4,0.14], [8,0.08], [16,0.04], [32,0.02] - N=32: [4,0.12], [8,0.06], [16,0.03], [32,0.01] --- ## Cross-Validation Summary 1. All legend colors match line/scatter plot colors exactly 2. Y-axis ranges consistent across charts for same metric 3. X-axis labels identical across all charts 4. Trend patterns consistent with legend color coding 5. All data points align with described trends ## Language Note All text appears in English. No non-English content detected. </details> Figure 7: Pass@N (top row) and Pass@ $N$ $-$ Pass@ $1$ (bottom row) across RSA steps for different values of $N$ . Larger $N$ results in higher Pass@N score, but requires more steps to mix, delaying the convergence of Pass@1 to Pass@ $N$ . All results with fixed $K=4$ . Tuning RSA under compute budgets. Our results indicate that jointly increasing $N$ , $K$ , and $T$ improves RSA performance. In practice, the key question is how to scale them relative to one another given a limited compute budget. Based on the above analysis we note that: - Population size $N$ controls the asymptotic performance. - Larger aggregation set size $K$ for a fixed $N$ leads to faster mixing of high quality chains (for $K>1$ ). - Longer self-aggregation depth $T$ monotonically improves performance. When a higher number of sequential reasoning steps $T$ are feasible, it allows for a smaller $K$ provided $N$ is large. Conversely, when $T$ is limited due to time constraints and increasing $K$ is impractical (e.g., due to context length constraints), $N$ should also be reduced; a large population that fails to mix effectively is less useful than a smaller batch that evolves rapidly together. We expect these findings to generalize to other model families. 5.4 RSA improves with aggregation-aware RL <details> <summary>2509.26626v1/x8.png Details</summary>  ### Visual Description # Technical Document Extraction: Aggregation-aware RL Training Performance ## Main Title **Aggregation-aware RL training leads to substantial gains with RSA. Standard RL, on the other hand, hurts RSA performance.** --- ### Chart Structure - **Grid Layout**: 6 charts arranged in 2 rows × 3 columns. - **Axes**: - **X-axis**: "RSA Step" (values: 2, 4, 6, 8, 10) - **Y-axis**: "Pass@1" (ranging from ~0.3 to ~0.7 depending on dataset) - **Legend**: - **Position**: Center of the grid (spatial grounding: [x_center, y_center]) - **Labels**: - **Blue**: Base + RSA - **Green**: Standard RL + RSA - **Orange**: Aggregation-aware RL + RSA --- ### Chart Analysis by Dataset #### 1. HMMT-25 - **Title**: HMMT-25 - **Trends**: - **Base + RSA (Blue)**: Starts at ~0.3, increases steadily to ~0.48 by step 10. - **Standard RL + RSA (Green)**: Flat line at ~0.45. - **Aggregation-aware RL + RSA (Orange)**: Sharp upward trend from ~0.3 to ~0.55. - **Key Insight**: Aggregation-aware RL + RSA outperforms others by ~10% at step 10. #### 2. Reasoning Gym Games - **Title**: Reasoning Gym Games - **Trends**: - **Base + RSA (Blue)**: Starts at ~0.65, plateaus at ~0.68. - **Standard RL + RSA (Green)**: Flat line at ~0.65. - **Aggregation-aware RL + RSA (Orange)**: Starts at ~0.65, rises to ~0.72. - **Key Insight**: Aggregation-aware RL + RSA achieves ~7% higher performance than Standard RL + RSA. #### 3. AIME-25 - **Title**: AIME-25 - **Trends**: - **Base + RSA (Blue)**: Starts at ~0.5, increases to ~0.7. - **Standard RL + RSA (Green)**: Flat line at ~0.65. - **Aggregation-aware RL + RSA (Orange)**: Steeper rise from ~0.5 to ~0.7. - **Key Insight**: Aggregation-aware RL + RSA matches Base + RSA performance while outperforming Standard RL + RSA. #### 4. LiveCodeBench-v6 - **Title**: LiveCodeBench-v6 - **Trends**: - **Base + RSA (Blue)**: Starts at ~0.5, rises to ~0.62. - **Standard RL + RSA (Green)**: Flat line at ~0.58. - **Aggregation-aware RL + RSA (Orange)**: Steeper ascent from ~0.5 to ~0.65. - **Key Insight**: Aggregation-aware RL + RSA gains ~7% over Standard RL + RSA. #### 5. Reasoning Gym Cognition + ARC - **Title**: Reasoning Gym Cognition + ARC - **Trends**: - **Base + RSA (Blue)**: Starts at ~0.45, plateaus at ~0.52. - **Standard RL + RSA (Green)**: Flat line at ~0.5. - **Aggregation-aware RL + RSA (Orange)**: Sharp rise from ~0.45 to ~0.55. - **Key Insight**: Aggregation-aware RL + RSA achieves ~10% higher performance than Standard RL + RSA. --- ### Cross-Chart Observations 1. **Legend Consistency**: - All charts use the same color coding (blue/green/orange) as the central legend. - No discrepancies between line colors and legend labels. 2. **Performance Pattern**: - **Aggregation-aware RL + RSA** consistently outperforms other methods across all datasets. - **Standard RL + RSA** shows minimal or no improvement over baseline (Base + RSA) in most cases. 3. **RSA Step Impact**: - Performance improves with increasing RSA steps (steps 2–10) for all methods. - Aggregation-aware RL + RSA demonstrates the steepest learning curve. --- ### Conclusion Aggregation-aware RL training with RSA significantly enhances performance compared to Standard RL + RSA across diverse datasets. The orange line (Aggregation-aware RL + RSA) dominates in all charts, validating the main title's claim. </details> Figure 8: Pass@ $1$ across RSA steps for the base, standard RL, and aggregation-aware RL policies with Qwen3-4B-Instruct-2507. Standard RL training generally hurts performance when using RSA, whereas aggregation-aware training leads to marked improvement on most tasks. We next analyze the impact of the aggregation-aware RL training procedure described in § 4. Setup. We use Qwen3-4B-Instruct-2507 as the reference model. We construct a general reasoning dataset by combining 16,000 randomly sampled math problems from DeepScaleR (Luo et al., 2025), and 2048 problems each from six Reasoning Gym tasks where the reference model performs poorly (tower_of_hanoi, sokoban, knight_swap, rush_hour, arc_1d, and sentence_reordering). For each query in the dataset, we generate four candidate solutions using the reference model, which are then used to form aggregation prompts as shown in Fig. 3. We train an aggregation-aware model on this augmented dataset by jointly optimizing (4) for the standard prompts and (5) for the aggregation prompts. As a baseline, we also train a model on the original dataset with only standard prompts by optimizing (4). The model trained using standard RL is only trained to generate solutions directly and is not optimized to aggregate reasoning chains. Both models are trained for 300 steps using RLOO (Ahmadian et al., 2024). (Further details in § D.4.) For evaluation, we run RSA for $T=10$ steps with the reference, standard RL post-trained, and aggregation-aware RL post-trained models on AIME-25, HMMT-25, LiveCodeBench-v6, and the Reasoning Gym Games and Cognition + ARC test sets from § 5. We ensure no data contamination between the training and test sets. We fix the aggregation size $K=4$ and population size $N=16$ . Results. Fig. 8 shows that in four out of five cases, RSA with the standard RL fine-tuned model underperforms RSA with the reference model, validating our hypothesis that distribution shifts incurred due to test-time scaling can lead to performance degradation after RL. The aggregation-aware policy, on the other hand, always outperforms standard RL and significantly outperforms the reference in four out of five tasks, with AIME-25 being the only outlier. Interestingly, we see massive gains on LiveCodeBench, despite the fact that our training dataset completely lacks any coding problems, which might indicate that the aggregation skills exhibit strong out of domain transferability. Overall, these experiments clearly demonstrate the benefits of aggregation-aware RL. Considering the implementation simplicity and the resulting robustness gains to RSA (or even to single-step self-aggregation), we strongly encourage its adoption for post-training. 6 Conclusion and future work We introduce Recursive Self-Aggregation (RSA), a hybrid test-time scaling framework that treats reasoning as an evolutionary process. By recursively aggregating reasoning chains, RSA enables models to cross-reference and recombine information across multiple candidates, while still retaining the depth of sequential refinement. This allows RSA to generate solutions that consistently outperform single-trajectory refinement and purely parallel scaling strategies. We further show that RL fine-tuning the LLM to perform aggregation amplifies RSA’s benefits, yielding superior performance. Future work. In future work, RSA can be composed with other test-time scaling methods to further improve performance, for example, by using self-verification to filter low-quality candidates from the population, thus introducing an explicit fitness function to the evolutionary algorithm. Another promising idea is to use multi-step reinforcement learning to train the policy for the end-to-end RSA procedure, moving beyond the greedy single-step aggregation explored in this work. Acknowledgments The authors thank Emiliano Penaloza for helpful comments. The research was enabled in part by computational resources provided by the Digital Research Alliance of Canada (https://alliancecan.ca), Mila (https://mila.quebec), NVIDIA, and the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231 using NERSC award NERSC DDR-ERCAP0034652. YB acknowledges funding from National Sciences and Engineering Council of Canada (NSERC) and the Canadian Institute for Advanced Research (CIFAR). GL acknowledges support from NSERC Discovery Grant RGPIN-2018-04821, the Canada Research Chair in Neural Computations and Interfacing, and a Canada-CIFAR AI Chair. GB acknowledges funding from NSERC and CIFAR. NM acknowledges support from the CIFAR Learning in Machines and Brains program. VS was supported by a UNIQUE scholarship. MJ is supported by a FRQNT Doctoral Fellowship (https://doi.org/10.69777/366694). SM acknowledges funding from FRQNT Doctoral Fellowship (https://doi.org/10.69777/372208) Prepared by LLNL under Contract DE-AC52-07NA27344 and supported by the LLNL-LDRD Program under Project No. 24-ERD-058. This manuscript has been authored by Lawrence Livermore National Security, LLC under Contract No. DE-AC52-07NA27344 with the U.S. Department of Energy. The United States Government retains, and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. Ethics statement Improving test-time scaling performance of LLMs can enhance their usefulness across domains but also carries risks associated with misuse and unintended generalization. Careful evaluation and responsible deployment are essential as these methods are applied to more capable models. Reproducibility statement We provide all the details to reproduce our results in § 5 and appendix D. Code for our experiments is available at https://github.com/HyperPotatoNeo/RSA. LLM use LLMs were used to assist paper editing and to write the code for experiments. References - Agrawal et al. (2025) Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflective prompt evolution can outperform reinforcement learning, 2025. URL https://arxiv.org/abs/2507.19457. - Ahmadian et al. (2024) Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740. - Balunović et al. (2025) Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL https://matharena.ai/. - Besta et al. (2024) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 17682–17690, 2024. - Bi et al. (2025) Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. In Forty-second International Conference on Machine Learning, 2025. - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. - Gao et al. (2023a) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866. PMLR, 23–29 Jul 2023a. URL https://proceedings.mlr.press/v202/gao23h.html. - Gao et al. (2023b) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764–10799. PMLR, 2023b. - Gu & Dao (2024) Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2. - Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC. - Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. - Hemberg et al. (2024) Erik Hemberg, Stephen Moskal, and Una-May O’Reilly. Evolving code with a large language model, 2024. URL https://arxiv.org/abs/2401.07102. - Inoue et al. (2025) Yuichi Inoue, Kou Misaki, Yuki Imajuku, So Kuroki, Taishi Nakamura, and Takuya Akiba. Wider or deeper? scaling llm inference-time compute with adaptive branching tree search. arXiv preprint arXiv:2503.04412, 2025. - Jaech et al. (2024) Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. - Jain et al. (2024) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974. - Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. - Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. - Lambert et al. (2025) Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=i1uGbfHHpH. - Lee et al. (2025) Kuang-Huei Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. Evolving deeper llm thinking, 2025. URL https://arxiv.org/abs/2501.09891. - Li et al. (2024) Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. Benchmarking and improving generator-validator consistency of language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=phBS6YpTzC. - Li et al. (2025) Zichong Li, Xinyu Feng, Yuheng Cai, Zixuan Zhang, Tianyi Liu, Chen Liang, Weizhu Chen, Haoyu Wang, and Tuo Zhao. Llms can generate a better answer by aggregating their own responses. arXiv preprint arXiv:2503.04104, 2025. - Luo et al. (2025) Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2, 2025. Notion Blog. - M-A-P Team et al. (2025) M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tianshun Xing, Ming Xu, Zhenzhu Yang, Zekun Moore Wang, Junting Zhou, Yuelin Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yinghao Ma, Tianyang Pang, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu, Shanghaoran Quan, Yizhou Tan, Zili Wang, Chenqing Wang, Hao Wang, Yiya Wang, Yubo Wang, Jiajun Xu, Kexin Yang, Ruibin Yuan, Yuanhao Yue, Tianyang Zhan, Chun Zhang, Jinyang Zhang, Xiyue Zhang, Xingjian Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng, Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang, Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, and Ge Zhang. Supergpqa: Scaling llm evaluation across 285 graduate disciplines, 2025. URL https://arxiv.org/abs/2502.14739. - Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=S37hOerQLB. - Meyerson et al. (2024) Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Adam Gaier, Arash Moradi Karkaj, Amy K. Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. ACM Trans. Evol. Learn. Optim., 4(4):27:1–27:40, December 2024. URL https://doi.org/10.1145/3694791. - Muennighoff et al. (2025) Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393. - Naik et al. (2023) Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. Diversity of thought improves reasoning abilities of llms. arXiv preprint arXiv:2310.07088, 2023. - Novikov et al. (2025) Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. - NVIDIA (2025) NVIDIA. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025. URL https://arxiv.org/abs/2508.14444. - OpenAI et al. (2025) OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925. - Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155. - Romera-Paredes et al. (2024) Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, 2024. - Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. - Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. - Sheng et al. (2024) Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. - Snell et al. (2025) Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4FWAwZtd2n. - Stojanovski et al. (2025) Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025. URL https://arxiv.org/abs/2505.24760. - Surina et al. (2025) Anja Surina, Amin Mansouri, Lars Quaedvlieg, Amal Seddas, Maryna Viazovska, Emmanuel Abbe, and Caglar Gulcehre. Algorithm discovery with llms: Evolutionary search meets reinforcement learning. arXiv preprint arXiv:2504.05108, 2025. - Trung et al. (2024) Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7601–7614, 2024. - Wang et al. (2024) Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692, 2024. - Wang et al. (2025) Qibin Wang, Pu Zhao, Shaohan Huang, Fangkai Yang, Lu Wang, Furu Wei, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. Learning to refine: Self-refinement of parallel reasoning in llms, 2025. URL https://arxiv.org/abs/2509.00084. - Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw. - Warner et al. (2024) Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL https://arxiv.org/abs/2412.13663. - Weng et al. (2023) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. - Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. - Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2023. - Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=5Xc1ecxO1h. - Zhang et al. (2025a) Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, 2025a. - Zhang et al. (2025b) Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. A survey on test-time scaling in large language models: What, how, where, and how well?, 2025b. URL https://arxiv.org/abs/2503.24235. - Zhao et al. (2025) Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, and Ilia Kulikov. The majority is not always right: Rl training for solution aggregation. arXiv preprint arXiv:2509.06870, 2025. Appendix A Additional related work Chain-of-thought aggregation. Several recent papers have explored self-aggregation as a parallel scaling strategy. Li et al. (2025) study simple single-step aggregation, while Wang et al. (2025) enhance aggregation ability through supervised fine-tuning (SFT), which requires access to a stronger teacher LLM. Concurrent to our work, Zhao et al. (2025) trained RL policies for single-step aggregation. Our work conducts more extensive experiments across a broader suite of tasks with ablations, and further motivate aggregation-aware RL as a means to improve the performance of test-time recursive aggregation as an additional contribution. Naik et al. (2023) does not use self-aggregation, instead including an “approach” in the prompt to generate multiple diverse solutions, from which an answer is selected using majority voting. The algorithm could easily be modified to use self-aggregation as the combination strategy instead. None of these works explored the sequential scaling and evolutionary components introduced in our work. Wang et al. (2024) is closely related to our approach, and uses an ensemble of LLMs to generate proposals that are jointly aggregated by a stronger model in an iterative loop. In contrast, RSA uses a single LLM and mixes the population by aggregating random subsets at each step while maintaining a fixed population size greater than the aggregation size to maintain population diversity, which we identify as critical factor (See § 5.3 and C for further analysis). Evolutionary methods. Another line of work closely related to RSA is using LLMs within evolutionary algorithms. Yang et al. (2023) propose using LLMs as proposers and mutators within an evolutionary optimization loop. They assume access to an external fitness function to evaluate the solutions. Romera-Paredes et al. (2024) propose FunSearch which builds upon a similar idea using LLMs to modify and propose new python functions given a scoring function. Similar to our aggregation-aware RL approach, EvoTune (Surina et al., 2025) trains the LLM within an evolutionary process with RL to improve the LLM in the context of program synthesis. Other related works. Several hybrid scaling strategies build on Tree-of-Thoughts (ToT) (Yao et al., 2023). (Inoue et al., 2025), expands trees over coherent text units (“thoughts”) and applies adaptive branching to Monte Carlo Tree Search, with external or self-verification serving as the value function. Graph-of-Thoughts (GoT) (Besta et al., 2024) generalizes Tree-of-Thoughts by organizing reasoning units (“thoughts”) into a directed acyclic graph (DAG), allowing nodes to have multiple parents (through aggregation) in addition to branching. Forest-of-Thought (Bi et al., 2025) expands multiple ToTs in parallel, and combines their final solutions using majority voting; if no consensus is found, it uses an LLM to compare the reasoning processes and outcomes of the different trees to give a final answer. A key weakness of these approaches is their reliance on either external verification of outcomes or value functions for scoring partial reasoning chains, the latter being a notoriously difficult problem. They also typically require careful prompting to ensure that the generated chains consist of meaningful atomic “thoughts”. To date, we are not aware of applications of these methods to long CoT reasoning models. Appendix B RSA algorithm Input: LLM $p_{\theta}$ with fixed parameters, problem ${\mathbf{x}}$ , population size $N$ , subset size $K$ , steps $T$ . Output: Final population ${\mathcal{P}}_{T}=\{\tau^{(T)}_{1},...,\tau^{(T)}_{N}\}$ . 1 // Initialization 2 $\{\tau_{i}^{(1)}\}_{i=1}^{N}\sim p_{\theta}(\,·\mid{\mathbf{x}})$ 3 ${\mathcal{P}}_{1}←\{\tau^{(1)}_{1},...,\tau^{(1)}_{N}\}$ // Recursively for $t=1,...,T-1$ 4 for $t← 1$ to $T-1$ do // Subsampling 5 for $i← 1$ to $N$ do 6 Sample $\{m^{(t)}_{i,1},...,m^{(t)}_{i,K}\}\sim\mathrm{Uniform}\left(\{1,...,N\}\right)$ indices without replacement 7 Form aggregation set $S_{i}^{(t)}←\{\tau^{(t)}_{m^{(t)}_{i,1}},...,\tau^{(t)}_{m^{(t)}_{i,K}}\}$ 8 ${\mathcal{S}}_{t}←\{S_{1}^{(t)},...,S_{N}^{(t)}\}$ // Self-aggregation 9 for $i← 1$ to $N$ do 10 $\tau_{i}^{(t+1)}\sim p_{\theta}(\,·\mid S_{i}^{(t)},{\mathbf{x}})$ 11 ${\mathcal{P}}_{t+1}←\{\tau^{(t+1)}_{1},...,\tau^{(t+1)}_{N}\}$ 12 // Termination return ${\mathcal{P}}_{T}$ and optionally sample $\hat{\tau}\sim\mathrm{Uniform}({\mathcal{P}}_{T})$ . Algorithm 1 Recursive Self-Aggregation (RSA) Appendix C Population diversity analysis Setup. In this experiment, we study how the diversity of the population evolves across RSA steps, using the AIME-25 dataset as an illustrative example. To quantify this, we require a metric to measure semantic diversity within a batch of CoTs. We embed the reasoning chains to generate sequence embeddings with ModernBERT (Warner et al., 2024), a strong Transformer encoder, and use the average cosine distance between embeddings in the population as a simple diversity metric. While this is an imperfect metric, it can still reveal interesting trends when plotted over time. Effect of varying $K$ . The left of Fig. 9 plots the average population diversity across RSA steps with a fixed population size $N=16$ and varying the aggregation size $K=2,3,4$ . For all $K$ , diversity rises sharply after the first aggregation step ( $t=2$ ) and then steadily decays. After 10 steps, larger $K$ yields lower diversity. This aligns with our intuition and previously observed results – larger aggregation sizes promote faster mixing of high-quality samples, leading to quicker convergence toward high-reward solutions (Fig. 5). Conversely, smaller $K$ slows mixing, explaining the weaker performance observed in Fig. 5. Effect of varying $N$ . The right panel of Fig. 9 shows average population diversity across RSA steps with fixed aggregation size $K=4$ and varying population sizes $N=4,8,16$ . After 10 steps, diversity is lowest for $N=4$ and highest for $N=16$ , though the relative differences are smaller than in the previous experiment varying $K$ . Taken together with the earlier result that larger $K$ accelerates mixing and improves performance for fixed $N$ , these findings support our hypothesis from § 5.3 – scaling up $N$ should be accompanied by increasing $K$ or $T$ , otherwise the batch will fail to mix in time and the average sample quality will remain poor after $T$ steps. <details> <summary>2509.26626v1/x9.png Details</summary>  ### Visual Description # Technical Document Extraction: AIME-25 Diversity Analysis ## Header - **Title**: "AIME-25, fixed population size N=16" - **Language**: English (no other languages detected) ## Main Chart ### Axes - **X-axis**: "RSA Step" (integer values 1–10) - **Y-axis**: "Diversity" (continuous scale from 0.005 to 0.025) ### Data Series #### K=2 (Orange Squares) - **Trend**: - Starts at 0.006 (Step 1) - Peaks at 0.016 (Step 2) - Declines steadily to 0.005 (Step 10) - **Key Points**: - Step 1: 0.006 - Step 2: 0.016 - Step 3: 0.010 - Step 4: 0.008 - Step 5: 0.006 - Step 6: 0.006 - Step 7: 0.006 - Step 8: 0.006 - Step 9: 0.006 - Step 10: 0.005 #### K=3 (Green Triangles) - **Trend**: - Starts at 0.006 (Step 1) - Peaks at 0.015 (Step 2) - Declines steadily to 0.004 (Step 10) - **Key Points**: - Step 1: 0.006 - Step 2: 0.015 - Step 3: 0.009 - Step 4: 0.007 - Step 5: 0.005 - Step 6: 0.005 - Step 7: 0.005 - Step 8: 0.004 - Step 9: 0.004 - Step 10: 0.004 #### K=4 (Red Diamonds) - **Trend**: - Starts at 0.006 (Step 1) - Peaks at 0.020 (Step 2) - Declines steadily to 0.003 (Step 10) - **Key Points**: - Step 1: 0.006 - Step 2: 0.020 - Step 3: 0.008 - Step 4: 0.007 - Step 5: 0.005 - Step 6: 0.005 - Step 7: 0.004 - Step 8: 0.004 - Step 9: 0.003 - Step 10: 0.003 ### Legend - **Position**: Top-right corner - **Components**: - Orange square: K=2 - Green triangle: K=3 - Red diamond: K=4 ## Spatial Grounding - Legend confirmed in top-right quadrant. - All data series align with legend colors/markers: - K=2 (orange squares) matches orange line. - K=3 (green triangles) matches green line. - K=4 (red diamonds) matches red line. ## Trend Verification - All series peak at RSA Step 2, then decline. - K=4 maintains highest diversity throughout. - K=2 and K=3 converge to similar values by Step 10. ## Component Isolation 1. **Header**: Title and population size. 2. **Main Chart**: Axes, data series, and trends. 3. **Legend**: Color-marker mappings. ## Conclusion The chart illustrates diversity trends for K=2, K=3, and K=4 under fixed population size N=16. All series peak at RSA Step 2, with K=4 showing the highest diversity. Post-peak, diversity declines monotonically for all K values, with K=4 remaining dominant. </details> <details> <summary>2509.26626v1/x10.png Details</summary>  ### Visual Description # Technical Document Extraction: AIME-25 Diversity Analysis ## Chart Overview **Title**: AIME-25, fixed aggregation size K=4 **Type**: Line chart with markers **Purpose**: Visualizes diversity metrics across RSA steps for different population sizes (N=4, N=8, N=16). --- ## Axis Labels and Scales - **X-axis**: - Label: "RSA Step" - Range: 1 to 10 (integer increments) - Ticks: 1, 2, 3, ..., 10 - **Y-axis**: - Label: "Diversity" - Range: 0.005 to 0.025 - Increment: 0.005 --- ## Legend - **Location**: Upper-right corner - **Entries**: 1. **Blue circles**: N=4 2. **Orange squares**: N=8 3. **Green triangles**: N=16 --- ## Data Series Analysis ### N=4 (Blue Circles) - **Trend**: - Starts at ~0.006 (step 1). - Peaks at **0.012** (step 2). - Declines steadily to ~0.002 (step 10). - **Key Points**: - Step 1: 0.006 - Step 2: 0.012 - Step 3: 0.006 - Step 4: 0.004 - Step 5: 0.005 - Step 6: 0.004 - Step 7: 0.003 - Step 8: 0.003 - Step 9: 0.003 - Step 10: 0.002 ### N=8 (Orange Squares) - **Trend**: - Starts at ~0.006 (step 1). - Peaks sharply at **0.025** (step 2). - Drops rapidly to ~0.003 (step 10). - **Key Points**: - Step 1: 0.006 - Step 2: 0.025 - Step 3: 0.007 - Step 4: 0.006 - Step 5: 0.004 - Step 6: 0.004 - Step 7: 0.003 - Step 8: 0.003 - Step 9: 0.003 - Step 10: 0.003 ### N=16 (Green Triangles) - **Trend**: - Starts at ~0.006 (step 1). - Peaks at **0.020** (step 2). - Declines gradually to ~0.003 (step 10). - **Key Points**: - Step 1: 0.006 - Step 2: 0.020 - Step 3: 0.007 - Step 4: 0.006 - Step 5: 0.005 - Step 6: 0.004 - Step 7: 0.004 - Step 8: 0.003 - Step 9: 0.003 - Step 10: 0.003 --- ## Cross-Series Observations 1. **Peak at Step 2**: - All series peak at RSA step 2, with N=8 showing the highest diversity (0.025). 2. **Post-Peak Divergence**: - N=8 declines most sharply after step 2. - N=16 maintains higher diversity than N=4 after step 2. 3. **Convergence**: - By step 10, all series converge to ~0.003 diversity. --- ## Spatial Grounding - **Legend Position**: Upper-right corner (coordinates: [x=0.85, y=0.85] relative to chart bounds). - **Color-Symbol Consistency**: - Blue circles (N=4) match all blue data points. - Orange squares (N=8) match all orange data points. - Green triangles (N=16) match all green data points. --- ## Final Notes - **Language**: All text is in English. - **Data Completeness**: All axis labels, legends, and data points are explicitly labeled. - **No Missing Information**: No unreadable text or omitted labels. </details> Figure 9: Left: Diversity over RSA steps with fixed population size $N=16$ and varying aggregation batch $K$ on AIME-25. Larger $K$ accelerates mixing, shown through faster drop in population diversity. Right: Diversity over RSA steps with fixed $K=4$ and varying $N$ on AIME-25. Increasing $N$ enhances the diversity of reasoning chains, and hence the Pass@ $N$ score, which determines asymptotic performance. However, very large $N$ relative to $K$ can slow mixing and hinder performance. Appendix D Experiment details In this section we provide all experiment details necessary to reproduce our results. We provide code here. D.1 Task details - Math. We evaluate on the complete AIME-25 and HMMT-25 datasets, each consisting of 30 problems. These tasks use a binary reward: 1.0 if the predicted answer is symbolically equivalent to the ground truth and 0.0 otherwise, evaluated using Math-Verify. - General reasoning. Reasoning Gym (Stojanovski et al., 2025) consists of a broad suite of tasks divided into different categories, and difficulty levels. For our evaluations, we construct two datasets – one from the “Games” category (17 tasks), requiring general reasoning and planning, and the other by combining the “Cognition” and “ARC” categories (7 + 2 tasks), requiring pattern recognition. Each of the datasets consists of 100 randomly generated problems, equally split between the tasks in the categories. We selected these categories as we found them to be the most difficult for our base models, particularly Qwen3-4B-Instruct-2507 which we use for our ablations. We evaluate on the “easy” version of the problems, since the “hard” version is significantly more challenging, with even frontier reasoning models obtaining 0.0 reward on most tasks. The reward function is task-dependent, and can be found alongside task descriptions in the Reasoning Gym repository. - Code generation. We evaluate code generation using the complete LiveCodeBench-v6 dataset (Jain et al., 2024) consisting of 1055 problems. The task uses a binary reward of 1.0 if the generated Python code passes all provided test cases upon execution, else 0.0. - Knowledge based reasoning. We use SuperGPQA (M-A-P Team et al., 2025) as our knowledge-based reasoning benchmark. Although RSA is designed to enhance deep reasoning, we include this dataset for completeness and still observe substantial gains. SuperGPQA consists of multiple-choice questions and assigns a binary reward of 1.0 when the selected option matches the ground truth. D.2 Models used We use Qwen3-4B-Instruct-2507 as the core model for all experiments and ablations. This choice was motivated by its small parameter count, which makes inference and RL fine-tuning tractable, while still offering strong base reasoning ability. For all models, we fix the response length to values that avoid frequent truncation and keep this constant across tasks. All model details, including their characteristics and response lengths, are tabulated in Table 2. D.3 Baseline details - Rejection sampling. We prompt the model to self-verify $N=160$ candidate solutions. We then compute the mean score over the positively sampled solutions – equivalent in expectation and lower variance than sampling one of these at random. - Self-refinement. For $T=10$ steps, generated solutions are fed back into the model, which is prompted to detect errors and refine its reasoning chain. - Majority voting. We extract the final answers from all $N=160$ reasoning chains and group equivalent ones. The majority group is then selected and compared with the ground truth. - Self-aggregation. Equivalent to RSA with a single step of aggregation, we first generate a batch of $K=4$ solutions and aggregate them with the model to produce the final answer. We stick to $K=4$ for self-aggregation since the context lengths cannot scale beyond this point without model performance degradation. As a result, this baseline is not “budget-matched” with RSA similar to the other parallel baselines above, but we note that the ability to grow the effective batch size well above the context constraints of the model is one of the major advantages of recursive aggregation over single-step aggregation. D.4 RL training setup We use verl (Sheng et al., 2024) as our framework to train the RL policies described in § 5.4. The aggregation-aware dataset is a 50-50 split between standard and aggregation prompts. To generate the aggregation prompts, we use our standard inference procedure § D.5 with the Qwen3-4B-Instruct-2507 reference policy to generate $K=4$ candidate reasoning chains per query. All RL training parameters are shared between the standard and aggregation-aware RL training runs. We use RLOO (Ahmadian et al., 2024) as the training algorithm with the following hyperparameters; learning rate $=1e-6$ , KL coefficient $\beta=0.0$ , batch size $=256$ , training steps $=300$ , response length $=8192$ , and max prompt length $=33792$ to fit the aggregation prompts. D.5 LLM inference settings For consistency and fairness, we share the same inference settings across all experiments. We use vLLM (Kwon et al., 2023) for both RSA and all baselines. We keep sampling parameters consistent across experiments. We set the sampling temperature = $1.0$ , top_p = $1.0$ , and min_p = $1.0$ , all of which are the default settings. Table 2: List of models used in our experiments. We include Mixture-of-Experts (MoE) (Shazeer et al., 2017) and state-space model (SSM) architectures (Gu et al., 2022; Gu & Dao, 2024). Our evaluation spans both non-thinking and thinking models, with appropriately larger response lengths allocated to the latter. | Qwen3-4B-Instruct-2507 | ✗ | ✗ | ✗ | 8192 | | --- | --- | --- | --- | --- | | Qwen3-30B-A3B-Instruct-2507 | ✓ | ✗ | ✗ | 8192 | | Qwen3-4B-Thinking-2507 | ✗ | ✗ | ✓ | 32768 | | gpt-oss-20b (medium) | ✓ | ✗ | ✓ | 16384 | | NVIDIA-Nemotron-Nano-9B-v2 | ✗ | ✓ | ✓ | 16384 | Appendix E Qualitative example We present a qualitative example below, where we provide Qwen3-4B-Instruct-2507 with four candidate solutions for the problem: “Compute the sum of the positive divisors (including $1$ ) of $9!$ that have units digit $1$ .” We highlight reasoning steps in the aggregated solution that are lifted from individual candidates, including parts that are newly added by the model and were not present in any candidate. We also provide the relevant parts from the candidate solutions that appear in the aggregated solution. - Step 1: While all candidate solutions begin by calculating $9!$ , some of them compute the products pairwise. The language in the aggregated solution most closely mirrors the first candidate. - Step 2: Different candidates compute prime factors in slightly different ways. The only solution that mentions Legendre’s formula explicitly is the second candidate. - Step 3: All solutions identified that any multiple of $5$ cannot have a units digit of $1$ . However, only the fourth candidate correctly identified that even numbers also cannot have a units digit of $1$ , which is then used in the aggregated solution. This significantly shortens the search space for valid divisors. - Step 4: This step had the most diversity among the candidates. The first candidate exhaustively listed all possible combinations and pooled them into six different cases. While the aggregated solution considers the same divisors as candidate # 4 (because of correctly identifying it can’t be an even number), it arranges them into a table, which was not done in any individual candidate. This shows that the aggregated solution can add new information not present in any candidates. Aggregated Solution We are asked to compute the sum of the positive divisors of $9!$ that have a units digit of 1. Step 1: Compute $9!$ $\displaystyle 9!=9× 8× 7× 6× 5× 4× 3× 2× 1=362880$ Step 2: Prime Factorization of $9!$ Using Legendre’s formula, the prime factorization is: $\displaystyle\text{Exponent of 2: }\left\lfloor\frac{9}{2}\right\rfloor+\left\lfloor\frac{9}{4}\right\rfloor+\left\lfloor\frac{9}{8}\right\rfloor=4+2+1=7$ $\displaystyle\text{Exponent of 3: }\left\lfloor\frac{9}{3}\right\rfloor+\left\lfloor\frac{9}{9}\right\rfloor=3+1=4$ $\displaystyle\text{Exponent of 5: }\left\lfloor\frac{9}{5}\right\rfloor=1\qquad\text{Exponent of 7: }\left\lfloor\frac{9}{7}\right\rfloor=1$ Thus, $9!=2^{7}× 3^{4}× 5^{1}× 7^{1}$ Step 3: Identify Divisors with Units Digit 1 We are to find all positive divisors $d$ of $9!$ such that $d\bmod 10=1$ . Note: • Any divisor divisible by 5 must end in 0 or 5. Thus, no divisor divisible by 5 can have units digit 1. • Any divisor divisible by 2 is even $→$ ends in an even digit (0, 2, 4, 6, 8). So no even divisor can end in 1. Therefore, any divisor ending in 1 must be: • Odd $→$ not divisible by 2 or 5 • Not divisible by 5 • So must be a product of powers of 3 and 7 only That is, the divisor must be of the form: $\displaystyle d=3^{b}· 7^{c}$ where: $\displaystyle 0≤ b≤ 4\quad(\text{since }3^{4}\mid 9!),\qquad c∈\{0,1\}\quad(\text{since }7^{1}\mid 9!).$ Step 4: List All Such Divisors with Units Digit 1 b We compute all $d=3^{b}· 7^{c}$ for $b=0,...,4$ , $c∈\{0,1\}$ , and check which end in 1. $b$ $3^{b}$ $c$ $d$ Units Digit 0 $1$ 0 $1$ 1 ✓ 0 $1$ 1 $7$ 7 ✗ 1 $3$ 0 $3$ 3 ✗ 1 $3$ 1 $21$ 1 ✓ 2 $9$ 0 $9$ 9 ✗ 2 $9$ 1 $63$ 3 ✗ 3 $27$ 0 $27$ 7 ✗ 3 $27$ 1 $189$ 9 ✗ 4 $81$ 0 $81$ 1 ✓ 4 $81$ 1 $567$ 7 ✗ Only the following values end in 1: $1,\quad 21,\quad 81.$ All are divisors of $9!$ : $1\text{: trivial divisor},\qquad 21=3× 7\text{: both powers present},\qquad 81=3^{4}\text{: }3^{4}\mid 9!.$ Step 5: Compute the Sum $1+21+81=\boxed{103}$ Final Answer $\boxed{103}$ Candidate #1 We are asked to compute the sum of the positive divisors of $\mathbf{9!}$ that have units digit 1. Step 1: Compute $\mathbf{9!}$ $9!=9× 8× 7× 6× 5× 4× 3× 2× 1$ We can compute it step by step: • $9!=362880$ $\vdots$ Candidate #2 We are asked to compute the sum of the positive divisors of $\mathbf{9!}$ that end in the digit 1 (i.e., have units digit 1). $\vdots$ Step 2: Prime factorization of $\mathbf{9!}$ We find the prime factorization of $9!$ using Legendre’s formula: For a prime $p$ , exponent in $n!$ is: $\sum_{k=1}^{∞}\left\lfloor\frac{n}{p^{k}}\right\rfloor$ Primes $≤ 9$ : $2$ , $3$ , $5$ , $7$ Exponent of 2: $\left\lfloor\frac{9}{2}\right\rfloor+\left\lfloor\frac{9}{4}\right\rfloor+\left\lfloor\frac{9}{8}\right\rfloor=4+2+1=7$ Exponent of 3: $\left\lfloor\frac{9}{3}\right\rfloor+\left\lfloor\frac{9}{9}\right\rfloor=3+1=4$ Exponent of 5: $\left\lfloor\frac{9}{5}\right\rfloor=1$ Exponent of 7: $\left\lfloor\frac{9}{7}\right\rfloor=1$ So, $9!=2^{7}× 3^{4}× 5^{1}× 7^{1}$ $\vdots$ Candidate #4 We are asked to compute the sum of the positive divisors of $\mathbf{9!}$ that have units digit 1. $\vdots$ Step 3: List all divisors of $9!$ that end in 1 Rather than list all divisors (which would be $(7+1)(4+1)(1+1)(1+1)=8× 5× 2× 2=160$ ), we need only the ones ending in 1. We can generate the divisors in a smart way, or use modular arithmetic to filter. But since the total number is $160$ , and many divisors end in $0,2,4,5,6,8$ , we can hope that only a few end in 1. Let’s consider the last digit of a divisor. We can iterate through all divisors and check the last digit, but we can also use modular filtering: We are to find all $d\mid 9!$ such that $d\equiv 1±od{10}$ Note: If $d\mid 9!$ , and $d\equiv 1±od{10}$ , then $d$ is odd (since it ends in 1), and $d\not\equiv 0±od{5}$ , because any multiple of 5 ends in 0 or 5, and only 5 ends in 5, but 1 is not 5. So d cannot be divisible by 5. That’s a key observation! Key Observation: • Any divisor of $9!$ ending in 1 must not be divisible by 5. • Because: If divisible by 5 → ends in 0 or 5 → cant́ end in 1. So, we are only interested in divisors of $9!$ that are not divisible by 5 and end in 1. But even among those not divisible by 5, most end in 2, 4, 6, 8 — only some end in 1. So actual candidate divisors: divisors of $\frac{9!}{5}=\frac{362880}{5}=72576$ Since we are excluding all divisors divisible by 5. So, we now consider only divisors of $2^{7}× 3^{4}$ (since we remove the 5 factor), and among them, those ending in 1. So the possible divisors (that are not divisible by 5) are all numbers of the form: $d=2^{a}× 3^{b}\quad\text{where }0≤ a≤ 7,\\ 0≤ b≤ 4$ We want those such that $d\equiv 1±od{10}$ Let’s go through all combinations of $(a,b)$ , compute $d=2^{a}× 3^{b}$ , and check if $d\mod 10=1$ $\vdots$ Wait — important point: • If $a≥ 1$ , then $d$ is even → last digit is even → cannot be 1. • So only when $a=0$ can a divisor $d$ be odd, hence end in 1. Thus, all divisors ending in 1 must have $a=0$ . $\vdots$ What about 7? Ah! We forgot about 7! We initially focused on $2^{a}· 3^{b}$ , but we removed 5, but we haven’t included the factor of 7! Wait — this is a major mistake. We said “divisors not divisible by 5” — thatś correct. But divisors can include **7** — since $7^{1}$ divides $9!$ $\vdots$ Appendix F RSA prompts We use a simple prompt template with minor task-appropriate changes, without tuning beyond some initial adjustments. Although greater gains may be achievable with careful prompt engineering, we avoided this to prevent skewing results, and instead report performance that can be reasonably expected from a straightforward implementation of RSA. In fact, we look forward to future work that applies automated prompt optimization, such as GEPA (Agrawal et al., 2025), to design prompts tailored to the end-to-end RSA procedure, offering a cheaper alternative to the RL-based approach discussed at the end of § 6. The prompt generation functions for math (AIME-25, HMMT-25), Reasoning Gym, and SuperGPQA tasks are given below. The prompt illustrated in Fig. 3 is the essentially the same prompt without any task-specific formatting. The prompts used for LiveCodeBench are similar, but contain some additional instructions to condition on the provided starter code. RSA prompt generation code ⬇ 1 def aggregate_prompt (query: str, candidate_answers: List [str], task: str) -> str: 2 # Reasoning Gym 3 if task == ’rg’: 4 problem_kind = ’problem’ 5 format_hint = ’<answer>...</answer>’ 6 # SuperGPQA 7 elif task == ’supergpqa’: 8 problem_kind = ’multiple-choice problem’ 9 format_hint = ’\\boxed{}. Only include the correct option letter in \\boxed{}; for example \\boxed{A}’ 10 # Math (AIME-25, HMMT-25) 11 else: 12 problem_kind = ’math problem’ 13 format_hint = ’\\boxed{}’ 14 15 parts = [] 16 # If K=1, we are doing single-trajectory refinement 17 if len (candidate_answers) == 1: 18 parts. append ( 19 f "You are given a {problem_kind} and a candidate solution. " 20 "The candidate may be incomplete or contain errors. " 21 "Refine this trajectory and produce an improved, higher-quality solution. " 22 "If it is entirely wrong, attempt a new strategy. " 23 f "End with the final result in {format_hint}.\n" 24 ) 25 # If K>1, we are doing multi-trajectory aggregation 26 else: 27 parts. append ( 28 f "You are given a {problem_kind} and several candidate solutions. " 29 "Some candidates may be incorrect or contain errors. " 30 "Aggregate the useful ideas and produce a single, high-quality solution. " 31 "Reason carefully; if candidates disagree, choose the correct path. " 32 "If all are incorrect, then attempt a different strategy." 33 f "End with the final result in {format_hint}.\n" 34 ) 35 36 parts. append ("Problem:\n") 37 # Original query is appended here. 38 # It also contains the formatting instructions. 39 parts. append (query. strip () + "\n") 40 41 # If K=1, we are doing single-trajectory refinement 42 if len (candidate_answers) == 1: 43 parts. append ("Candidate solution (may contain mistakes):\n") 44 ans_str = (candidate_answers [0] or ""). strip () 45 parts. append (f "---- Candidate ----\n{ans_str}\n") 46 parts. append ( 47 f "Now refine the candidate into an improved solution. " 48 "Provide clear reasoning and end with the final answer in {format_hint}." 49 ) 50 # If K>1, we are doing multi-trajectory aggregation 51 else: 52 parts. append ("Candidate solutions (may contain mistakes):\n") 53 for i, ans in enumerate (candidate_answers, 1): 54 ans_str = (ans or ""). strip () 55 parts. append (f "---- Solution {i} ----\n{ans_str}\n") 56 parts. append ( 57 f "Now write a single improved solution. Provide clear reasoning and end with the final answer in {format_hint}." 58 ) 59 60 return "\n". join (parts) 61 62 def build_prompt (query: str, candidate_answers: Optional [List [str]], task: str): 63 # At t=1, candidate_answers is None and we return base query 64 # At t>1, candidate_answers is a list of strings and we return an aggregated prompt 65 if candidate_answers is not None: 66 prompt = aggregate_prompt (query, candidate_answers, task) 67 else: 68 prompt = query 69 return prompt