2505.11140

# Follow the Path: Reasoning over Knowledge Graph Paths to Improve LLM Factuality Abstract We introduce fs1, a simple yet effective method that improves the factuality of reasoning traces by sourcing them from large reasoning models (e.g., DeepSeek-R1) and grounding them by conditioning on knowledge graph (KG) paths. We fine-tune eight instruction-tuned Large Language Models (LLMs) on 3.9K factually grounded reasoning traces and rigorously evaluate them on six complex open-domain question-answering (QA) benchmarks encompassing 23.9K questions. Our results demonstrate that our fs1 -tuned model (32B parameters) consistently outperforms instruction-tuned counterparts with parallel sampling by 6-14 absolute points (pass@ $16$ ). Our detailed analysis shows that fs1 considerably improves model performance over more complex questions (requiring 3 or more hops on KG paths) and numerical answer types compared to the baselines. Furthermore, in single-pass inference, we notice that smaller LLMs show the most improvements. While prior works demonstrate the effectiveness of reasoning traces primarily in the STEM domains, our work shows strong evidence that anchoring reasoning to factual KG paths is a critical step in transforming LLMs for reliable knowledge-intensive tasks. 1 Introduction Factual consistency of LLM-generated output is a requirement for critical real-world applications. LLM reasoning in the form of “thinking” has shown promising improvements in model performance on complex downstream tasks, such as mathematical reasoning and puzzle-like questions using additional compute resources during inference (e.g., test-time scaling; Wu et al., 2024; Muennighoff et al., 2025; Zhang et al., 2025). However, it remains an open question whether these reasoning techniques improve factuality, particularly for complex multi-hop QA (mQA). This task tests a model’s ability to answer a question by synthesizing information from multiple pieces of evidence, often spread across different resources and requiring reasoning steps. We hypothesize that reasoning models should perform better than non-reasoning LLMs on the mQA task. To test this hypothesis, we source reasoning traces from state-of-the-art reasoning models and fine-tune several non-reasoning LLMs to attempt to induce reasoning capabilities. However, we have no guarantee that these reasoning traces from the large reasoning models are factually correct. In order to have a formal factual grounding in these traces, we condition the models on retrieved knowledge graph (KG) paths relevant to the questions. This is possible as KGs encode facts as directed, labeled graphs over entities and relations, which offers a verifiable foundation to inform each step of the reasoning process. We call our approach fs1 (factual simple test-time scaling; Muennighoff et al., 2025). We fine-tune eight different LLMs sizes on the original reasoning traces (rt; 3.4K samples) or on our KG-enhanced traces (fs1; 3.9K samples). We evaluate the fine-tuned models on six QA test sets spanning 23.9K questions, finding that fine-tuning on this amount of data can improve accuracy by 6-14 absolute points (pass@ $16$ ) for a 32B parameter model across the benchmarks. A snapshot of our method is in Figure 1. This setup enables us to address our research question (RQ): To what extent does grounding the reasoning processes of LLMs in KG paths enhance their factual accuracy for mQA? To address this question, our contributions are as follows: <details> <summary>x1.png Details</summary>  ### Visual Description ## Comparison Diagram: Model Performance on "Las Meninas" Question ### Overview The image presents a comparison of four different models' performance on the question: "The visual artist that created the art series of Las Meninas, where did they live?". Each model's approach and answer are displayed, along with an indication of whether the answer was correct (green checkmark) or incorrect (red 'X'). ### Components/Axes * **Title:** "The visual artist that created the art series of Las Meninas, where did they live?" * **Models (from left to right):** * Instruction-tuned * Chain-of-Thought * Thinking (rt) * Thinking with KGs (fs1) * **Visual Elements:** * Robot emoji representing the model. * Speech bubble above each robot indicating the model's name/type. * Text box below each robot showing the model's reasoning and answer. * Red 'X' or green checkmark indicating incorrect or correct answer, respectively. * For "Thinking with KGs (fs1)", there are icons representing "Question", a knowledge graph, and "Wikidata". ### Detailed Analysis or ### Content Details **1. Instruction-tuned:** * **Model Type:** Instruction-tuned * **Reasoning/Answer:** "The answer is Paris." * **Result:** Incorrect (Red 'X' in the top-right) **2. Chain-of-Thought:** * **Model Type:** Chain-of-Thought * **Reasoning/Answer:** "Let me think step-by-step... My answer is Vienna." * **Result:** Incorrect (Red 'X' in the top-right) **3. Thinking (rt):** * **Model Type:** Thinking (rt) * **Reasoning/Answer:** * "\<think> Okay, so I need to... \</think>" * "The answer should be Barcelona." * **Result:** Incorrect (Red 'X' in the top-right) **4. Thinking with KGs (fs1):** * **Model Type:** Thinking with KGs (fs1) * **Reasoning/Answer:** * "\<Question> + [Knowledge Graph Icon] + [Wikidata Icon]" * "\<think> Okay, let me tackle this question... \</think>" * "The visual artist who created the art series [...] The answer is Madrid." * **Result:** Correct (Green checkmark in the top-right) ### Key Observations * Three out of the four models provided incorrect answers. * The "Thinking with KGs (fs1)" model, which incorporates knowledge graphs and Wikidata, was the only one to provide the correct answer (Madrid). * The models that failed provided the following answers: Paris, Vienna, and Barcelona. ### Interpretation The diagram illustrates the importance of incorporating external knowledge sources (knowledge graphs and Wikidata) when answering questions that require specific factual information. The "Thinking with KGs (fs1)" model's success suggests that leveraging structured knowledge bases can significantly improve the accuracy of language models in question-answering tasks. The failure of the other models, which rely on instruction tuning or chain-of-thought reasoning alone, highlights the limitations of these approaches when dealing with questions that require accessing and reasoning about external knowledge. </details> Figure 1: Snapshot of Method. We show a snapshot of the experiments executed in this study. There are four settings on how a question can be answered; (1) direct answer from an instruction-tuned model, (2) step-by-step reasoning via Chain-of-Thought, (3) original “thinking”, and (4) knowledge-graph enhanced “thinking”. We show an example of how (4) looks like in Figure 4. - We demonstrate that with test-time scaling (parallel sampling), our fs1 -tuned Qwen2.5-32B model improves factual accuracy by 6-14 absolute points (at pass@ $16$ ). - We conduct an analysis over the question and answer types (e.g., question difficulty, answer type, and domains) to investigate where fs1 -tuned models provide improvements. We find that fs1 -tuned models perform better on more difficult questions, requiring 3 hops or more. - We examine performance of eight fs1 -tuned models (360M-32B parameters) in a pass@ $1$ setting against baselines. We find that smaller LLMs have the largest increase in performance, whereas larger models see less profound improvements. - We release 3.4K raw reasoning traces and 3.9K KG-enhanced reasoning traces both sourced from QwQ-32B and Deepseek-R1. All code, datasets, and models are publicly available under an MIT license: https://github.com/jjzha/fs1. 2 Reasoning Data rt: Distilling Reasoning Traces. To obtain reasoning traces, we use ComplexWebQuestions (CWQ; Talmor & Berant, 2018), a dataset designed for complex mQA. The CWQ dataset is created by automatically generating complex SPARQL queries based on Freebase (Bollacker et al., 2008). These queries are then automatically transformed into natural language questions, which are further refined by human paraphrasing. We take the CWQ dev. set, which consists of 3,519 questions, to curate the reasoning traces. We query both QwQ-32B (Qwen Team, 2025) and Deepseek-R1 (671B; DeepSeek-AI, 2025). By querying the model directly with a question, e.g., “ What art movement was Pablo Picasso part of? ”, we retrieve the reasoning traces surrounded by “think” tokens (<think>...</think>) and force the model to give the final answer to the question in \boxed{} format. We extract around 3.4K correct-only traces (final answer is correct), which we call rt. We show full examples in Figure 8 and Figure 9 (Appendix C). fs1: Enhancing Reasoning Traces with Knowledge Graph Paths. We attempt to steer the reasoning traces with KG paths to remove the inaccuracies in the traces. Since the CWQ dataset consists of entities from Freebase, we align them to their corresponding Wikidata entities. For each question in the dev. set of the CWQ dataset, relevant KG paths are extracted from Wikidata using random walks using SPARQL queries as shown in Appendix E. Each mQA pair in the dataset may contain multiple valid KG paths, which are linearized graphs that retain the structural information of the KG. The paths are generated by extracting the relevant entities from the question and the gold answer. These diverse KG paths that can lead to the same answer reflect the possible diversity of the reasoning traces. Therefore, including linearized graphs improves the interpretability and the explainability of the reasoning traces. The prompt to obtain the improved reasoning traces is shown in Figure 4, by prompting QwQ-32B and Deepseek-R1 again. Full examples are in Figure 10 and Figure 11 (Appendix C). Table 1: Training Data Statistics. Statistics of reasoning traces of QwQ-32B and Deepseek-R1 on CWQ based on the Qwen2.5-32B tokenizer. The original reasoning traces (rt) are from simply querying the question to the reasoning models, whereas fs1 indicates the statistics when queried with the knowledge graphs. We calculate the performance of the models’ final answer via LLM-as-a-Judge. We show that fs1 has higher performance in terms of accuracy compared to rt. | | | QwQ-32B | R1-685B | Total | | | | | --- | --- | --- | --- | --- | --- | --- | --- | | rt | fs1 | rt | fs1 | rt | fs1 | | | | Exact Match | | 0.46 | $\uparrow$ 0.63 | 0.56 | $\uparrow$ 0.72 | 0.51 | $\uparrow$ 0.67 | | Sem. Match (all-MiniLM-L6-v2) | | 0.50 | $\uparrow$ 0.58 | 0.55 | $\uparrow$ 0.63 | 0.52 | $\uparrow$ 0.60 | | LLM-as-a-Judge (gpt-4o-mini) | | 0.44 | $\uparrow$ 0.61 | 0.54 | $\uparrow$ 0.70 | 0.49 | $\uparrow$ 0.65 | | Samples with only correct answers | | | | | | | | | Number of Samples | | 1,533 | 1,972 | 1,901 | 1,914 | 3,434 | 3,886 | | Avg. Reasoning Length (subwords) | | 937 | 897 | 1,043 | 637 | 990 | 767 | | Avg. Answer Length (subwords) | | 40 | 93 | 64 | 116 | 52 | 104 | <details> <summary>x2.png Details</summary>  ### Visual Description ## Violin Plot: Token Count Distribution by Data Source and Model ### Overview The image presents two violin plots comparing the distribution of token counts (number of subwords) for two language models, QwQ-32B and DeepSeek-R1, across two data sources, 'rt' and 'fs1'. Each violin plot shows the distribution's shape, median, first quartile (Q1), and third quartile (Q3). ### Components/Axes * **Title (Left Plot):** QwQ-32B * **Title (Right Plot):** DeepSeek-R1 * **Y-axis:** Token Count (# Subwords), ranging from 0 to 8000 with gridlines at intervals of 2000. * **X-axis:** Data Source, with two categories: 'rt' and 'fs1'. * **Violin Plot Colors:** Blue for 'rt' data source, Orange for 'fs1' data source. * **Quartile Markers:** Dashed lines indicate the first quartile (Q1) and third quartile (Q3). * **Median Marker:** Text labels indicate the median value. ### Detailed Analysis **QwQ-32B (Left Plot):** * **rt (Blue):** * The violin plot is centered around lower token counts, with a long tail extending to higher counts. * Q1: 392 * Median: 552 * Q3: 1017 * **fs1 (Orange):** * The violin plot is also centered around lower token counts, with a long tail extending to higher counts. * Q1: 386 * Median: 553 * Q3: 1039 **DeepSeek-R1 (Right Plot):** * **rt (Blue):** * The violin plot is centered around lower token counts, with a long tail extending to higher counts. * Q1: 431 * Median: 635 * Q3: 1274 * **fs1 (Orange):** * The violin plot is centered around lower token counts, with a shorter tail compared to the 'rt' data. * Q1: 359 * Median: 496 * Q3: 792 ### Key Observations * For QwQ-32B, the median token counts are nearly identical for both 'rt' and 'fs1' data sources (552 vs 553). * For DeepSeek-R1, the median token count is higher for the 'rt' data source (635) compared to 'fs1' (496). * The Q3 values are higher than the median values for all data source and model combinations, indicating a right-skewed distribution. * DeepSeek-R1 has a higher Q3 value for 'rt' (1274) compared to QwQ-32B (1017), suggesting a wider range of higher token counts in the 'rt' data for DeepSeek-R1. ### Interpretation The violin plots provide a visual representation of the distribution of token counts for the two language models across the two data sources. The data suggests that: * QwQ-32B exhibits similar token count distributions for both 'rt' and 'fs1' data sources, as indicated by the nearly identical median values. * DeepSeek-R1 shows a noticeable difference in token count distributions between the two data sources, with 'rt' having a higher median and Q3 compared to 'fs1'. This could indicate that DeepSeek-R1 processes or tokenizes the 'rt' data differently, resulting in a higher number of subwords. * The right-skewed distributions observed in all violin plots suggest that while most data points have lower token counts, there are instances with significantly higher token counts, contributing to the long tails. * The differences in token count distributions between the models and data sources could be attributed to variations in the models' architectures, training data, or tokenization strategies. Further investigation would be needed to determine the specific factors contributing to these differences. </details> Figure 2: Distribution of Reasoning Traces. We show the distribution of the reasoning length among the queried models. In the left plot, we show rt and in the right plot, we show fs1. We show that particularly for fs1 and Deepseek-R1, the reasoning length is shorter in terms of subwords. Data Statistics. In Table 1 and Figure 2, we compare the reasoning trace accuracy and statistics of rt and fs1. We evaluate reasoning traces using three methods: (1) Exact Match, checking if the \boxed{} answer exactly matches or is a sub-phrase of any gold answer; (2) Semantic Match, accepting answers with a cosine similarity score $>$ 0.5; and (3) LLM-as-a-Judge, verifying entity alignment using gpt-4o-mini-2024-07-18. Results show that fs1 achieves higher accuracy, indicating that it contains more factual answers. Traces from rt are longer (up to 1K subwords), fs1 traces are typically shorter (around 800 subwords). The median length in subwords is similar for QwQ-32B (552 for rt and 553 for fs1), while there is a difference for Deepseek-R1 (635 median for rt and 496 for fs1). Spot-checking reveals that fs1 yields a more definitive answer. 3 Methodology 3.1 Training and Inference We fine-tune six Qwen2.5-Instruct models (0.5B to 32B) on rt and fs1, using only reasoning traces with correct final answers. During inference, we evaluate the model on the original questions to test its performance. Following Muennighoff et al. (2025), we train for 5 epochs with a sequence length of 8,192, a batch size of 16, a learning rate of $1× 10^{-5}$ (cosine schedule, 5% warmup), and a weight decay of $1× 10^{-4}$ . The models are optimized with a standard supervised fine-tuning (SFT) loss, which minimizes the negative log-likelihood (implemented as the cross-entropy function) of target tokens in an autoregressive manner. Let $y_{t}^{*}$ be the correct token and $p_{\theta}(y_{t}^{*}\mid x,y_{<t})$ be the model’s probability of predicting it. The model optimizes the function: $$ \mathcal{L}_{\text{SFT}}(\theta)=-\frac{1}{T}\sum_{t=1}^{T}\log p_{\theta}\left(y_{t}^{*}\mid x,y_{<t}\right). \tag{1} $$ For inference, we use a temperature ( $T$ ) of $0.7$ and top_p of $0.8$ for original instruct models. Otherwise, we use $T=0.6$ and top_p of $0.95$ . Further details on hardware and costs are in Appendix B. fs1 Prompt Example ⬇ When did the sports team owned by Leslie Alexander win the NBA championship? While answering the question, make use of the following linearised graph as an inspiration in your reasoning, not as the only answer: 1994 NBA Finals, winner, Houston Rockets Houston Rockets, owned by, Leslie Alexander 1995 NBA Finals, winner, Houston Rockets Houston Rockets, owned by, Leslie Alexander. Put your final answer within \ boxed {}. – (For illustration) Gold Answer: [”1994 and 1995”] Figure 3: fs1 Prompt Example. We depict how we prompt both Deepseek-R1 and QwQ-32B to obtain better reasoning traces with KG paths. LLM-as-a-Judge (Llama-3.3-70B) ⬇ gold answer: [” Joule per gram per kelvin ”, ” Joule per kilogram per kelvin ”] predicted answer: ” J /(kg $ \ cdot$ K)” Is the gold answer entity or value contained in the predicted answer? Respond only with 0 (no) or 1 (yes). # Llama -3.3-70 B - Instruct outputs ”1” Figure 4: Prompt for LLM-as-a-Judge. We show the LLM-as-a-Judge prompt for evaluating whether the predicted and gold answer refer to the same real-world entity, where regular exact string matching will not capture the alignment between the gold and predicted answer in this example (i.e., the measurement unit). Table 2: Test Benchmark. Overview of the mQA test sets used in our evaluation. | CWQ (Talmor & Berant, 2018) | apache-2.0 | 3.5K | Multi-hop QA from WebQuestionsSP with compositional SPARQL queries for Freebase paraphrased by crowd workers. | | --- | --- | --- | --- | | ExaQT (Jia et al., 2021) | cc-by-4.0 | 3.2K | Temporal-QA benchmark combining eight KB-QA datasets, focusing on time-specific queries. | | GrailQA (Gu et al., 2021) | apache-2.0 | 6.8K | Freebase QA dataset with annotated answers and logical forms (SPARQL/S-expressions) across 86 domains. | | SimpleQA (Wei et al., 2024a) | MIT | 4.3K | Fact-seeking questions with verified answers, designed to measure and challenge the factual accuracy of language models. | | Mintaka (Sen et al., 2022) | cc-by-4.0 | 4.0K | Multilingual QA (9 languages), entity-linked pairs across diverse domains (English test split). | | WebQSP (Yih et al., 2016) | apache-2.0 | 2.0K | Enhanced WebQuestions with Freebase QA annotated with SPARQL ( $\sim$ 82% coverage). | | Total | | 23.9K | | <details> <summary>x3.png Details</summary>  ### Visual Description ## Heatmap: Dataset Comparison Metrics ### Overview The image presents three heatmaps comparing different datasets (CWQ_train, CWQ_test, ExaQT, GrailQA, SimpleQA, Mintaka, WebQSP) based on three metrics: Cosine Similarity Count (> 0.90), Exact Match Count, and Average Cosine Similarity. Each heatmap is a lower triangular matrix, where the rows and columns represent the datasets being compared. The color intensity of each cell represents the value of the metric for the corresponding dataset pair. ### Components/Axes * **Titles:** * Left: Cosine Similarity Count (> 0.90) * Center: Exact Match Count * Right: Average Cosine Similarity * **Rows (Left Side):** CWQ_train, CWQ_test, ExaQT, GrailQA, SimpleQA, Mintaka, WebQSP * **Columns (Bottom Side):** CWQ_train, CWQ_test, ExaQT, GrailQA, SimpleQA, Mintaka, WebQSP * **Color Scale:** The color scale is not explicitly shown, but it appears to range from dark green (low values) to yellow (high values). ### Detailed Analysis #### Cosine Similarity Count (> 0.90) This heatmap shows the count of pairs with a cosine similarity greater than 0.90. * **CWQ_train vs. CWQ_test:** 109 * **CWQ_train vs. ExaQT:** 51 * **CWQ_test vs. ExaQT:** 80 * **CWQ_train vs. GrailQA:** 1 * **CWQ_train vs. SimpleQA:** 0 * **CWQ_test vs. GrailQA:** 0 * **CWQ_test vs. SimpleQA:** 1 * **ExaQT vs. GrailQA:** 0 * **ExaQT vs. SimpleQA:** 0 * **GrailQA vs. SimpleQA:** 1 * **CWQ_train vs. Mintaka:** 0 * **CWQ_test vs. Mintaka:** 1 * **ExaQT vs. Mintaka:** 4 * **GrailQA vs. Mintaka:** 0 * **SimpleQA vs. Mintaka:** 0 * **CWQ_train vs. WebQSP:** 12 * **CWQ_test vs. WebQSP:** 26 * **ExaQT vs. WebQSP:** 83 * **GrailQA vs. WebQSP:** 1 * **SimpleQA vs. WebQSP:** 0 * **Mintaka vs. WebQSP:** 0 * **WebQSP vs. WebQSP:** 15 #### Exact Match Count This heatmap shows the count of exact matches between the datasets. * **CWQ_train vs. CWQ_test:** 0 * **CWQ_train vs. ExaQT:** 0 * **CWQ_test vs. ExaQT:** 0 * **CWQ_train vs. GrailQA:** 0 * **CWQ_test vs. GrailQA:** 0 * **ExaQT vs. GrailQA:** 0 * **CWQ_train vs. SimpleQA:** 0 * **CWQ_test vs. SimpleQA:** 0 * **ExaQT vs. SimpleQA:** 0 * **GrailQA vs. SimpleQA:** 0 * **CWQ_train vs. Mintaka:** 0 * **CWQ_test vs. Mintaka:** 0 * **ExaQT vs. Mintaka:** 0 * **GrailQA vs. Mintaka:** 0 * **SimpleQA vs. Mintaka:** 0 * **CWQ_train vs. WebQSP:** 0 * **CWQ_test vs. WebQSP:** 1 * **ExaQT vs. WebQSP:** 0 * **GrailQA vs. WebQSP:** 0 * **SimpleQA vs. WebQSP:** 0 * **Mintaka vs. WebQSP:** 0 * **WebQSP vs. WebQSP:** 0 #### Average Cosine Similarity This heatmap shows the average cosine similarity between the datasets. * **CWQ_train vs. CWQ_test:** 0.15 * **CWQ_train vs. ExaQT:** 0.11 * **CWQ_test vs. ExaQT:** 0.12 * **CWQ_train vs. GrailQA:** 0.08 * **CWQ_test vs. GrailQA:** 0.08 * **ExaQT vs. GrailQA:** 0.05 * **CWQ_train vs. SimpleQA:** 0.11 * **CWQ_test vs. SimpleQA:** 0.12 * **ExaQT vs. SimpleQA:** 0.13 * **GrailQA vs. SimpleQA:** 0.07 * **CWQ_train vs. Mintaka:** 0.11 * **CWQ_test vs. Mintaka:** 0.12 * **ExaQT vs. Mintaka:** 0.12 * **GrailQA vs. Mintaka:** 0.07 * **SimpleQA vs. Mintaka:** 0.11 * **CWQ_train vs. WebQSP:** 0.12 * **CWQ_test vs. WebQSP:** 0.13 * **ExaQT vs. WebQSP:** 0.11 * **GrailQA vs. WebQSP:** 0.06 * **SimpleQA vs. WebQSP:** 0.11 * **Mintaka vs. WebQSP:** 0.11 * **WebQSP vs. WebQSP:** 0.11 ### Key Observations * **Cosine Similarity Count:** ExaQT and WebQSP have a high cosine similarity count (83), indicating many similar pairs. CWQ_test also shows a high similarity with ExaQT (80). CWQ_train and CWQ_test also have a high similarity (109). * **Exact Match Count:** Exact matches are rare, with only CWQ_test and WebQSP having a single exact match. * **Average Cosine Similarity:** The average cosine similarity values are relatively low across all dataset pairs, ranging from 0.05 to 0.15. ### Interpretation The heatmaps provide insights into the similarity and overlap between different datasets. The high cosine similarity counts between CWQ_train, CWQ_test, ExaQT, and WebQSP suggest that these datasets contain many similar questions or examples. The low exact match counts indicate that while the questions may be semantically similar, they are rarely identical. The average cosine similarity values suggest that, on average, the datasets are not highly similar, but there are pockets of high similarity as indicated by the cosine similarity counts. The data suggests that ExaQT and WebQSP share a significant number of semantically similar questions. </details> Figure 5: Data Overlap. We show data overlap between the train set and benchmark. On the left, one can observe the count of similar questions when the cosine similarity $>$ 0.90 (measured with paraphrase-MiniLM-L6-v2; Reimers & Gurevych, 2019). In the middle, we measure exact match counts. On the right, we show the average pairwise cosine similarity across the full test sets. 3.2 Benchmarks and Evaluation We show the test datasets, licenses, size and a short description in Table 2. We have four baselines, namely Qwen2.5-72B-Instruct (Qwen Team, 2024), QwQ-32B, Deepseek-R1, and o3-mini (OpenAI, 2025). To evaluate our models, we select a suite of six mQA benchmarks with a total of 23.9K questions. We have four setups for benchmarking the models: (1) All models including baselines are evaluated zero-shot (i.e., only querying the question); (2) the models are queried using zero-shot chain-of-thought prompting (Kojima et al., 2022; Wei et al., 2022), where we simply append the prompt “ Put your final answer within \boxed{}. Think step-by-step. ”; (3) we benchmark the models fine-tuned on rt; (4) we benchmark the models fine-tuned on fs1. In Figure 12 (Section D.1), we show an example of each dataset in the test benchmark. Possible Data Leakage. In Figure 5, we show the overlap of the questions in the training set of ComplexWebQuestions (CWQ_train) versus all the other benchmarks used in our study (all questions lower-cased). On the left, we count the times that the cosine similarity between questions exceeds 0.90. We can see that there is the most overlap between CWQ_train and CWQ_test (109 questions), and the second most is between WebQSP and ExaQT (83 questions). In the middle, we show that there is almost to none exact string match between the questions. On the right, we show the average pairwise cosine similarity across the benchmarks is lower or equal to 0.15. Evaluation Metric. Similar to previous studies, e.g., Ma et al. (2025), we report pass@ $k$ , which reflects the probability that at least one out of $k$ randomly selected completions (drawn from a total of $n$ completions per problem) is correct. As such, it serves as an upper-bound on practical performance, which would require a subsequent selection mechanism. Formally, pass@ $k$ is given by: $\mathbb{E}_{\text{problems}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$ , where $n$ is the number of generated completions per problem and $c$ is the count of correct completions (Chen et al., 2021). For our benchmarks, we evaluate $k=\{1,2,4,8,16\}$ . In practice, pass@32 is typically reported for formal theorem-proving tasks, while pass@1 (reducing to standard top-1 accuracy) is standard for math and coding tasks as mentioned by Ma et al. (2025). In this work for factual mQA, we report until $k=16$ . LLM-as-a-Judge. To decide whether an answer is correct or not (1 or 0), our main evaluation approach is using LLM-as-a-judge with Llama-3.3-70B-Instruct We compare both gpt-4o-mini-2024-07-18 and Llama-3.3-70B-Instruct on a large subsample of our outputs and saw there there is almost no difference in predictions. Additionally, Llama-3.3-70B is rated higher in LM Arena than gpt-4o-mini (at time of writing 79 ${}^{\text{th}}$ vs. 83 ${}^{\text{rd}}$ respectively). to determine whether a predicted answer obtained from the \boxed{} output is referring to the same real-world entity as the gold answer. An example of this is shown in Figure 4. When the model does not generate a \boxed{} output, we take the last 10 subwords as predicted answer, which LLM-as-a-judge can infer what the predicted real-world entity is when there is not exact string matching. This same approach is used in Table 1. Compared to exact string matching and semantic similarity evaluation methods, LLM-as-a-Judge rates the quality of output similarly compared to the other methods. <details> <summary>x4.png Details</summary>  ### Visual Description ## Line Charts: Pass@k Performance on Question Answering Datasets ### Overview The image presents a series of line charts comparing the performance of four different methods (inst, cot, rt, fs1) on six question answering datasets (ComplexWebQuestions, ExaQT, GrailQA, Mintaka, SimpleQA, WebQSP). The charts display the "pass@k" metric (percentage of questions answered correctly within the top k predictions) as a function of k. ### Components/Axes * **Titles (Top Row, Left to Right):** ComplexWebQuestions, ExaQT, GrailQA * **Titles (Bottom Row, Left to Right):** Mintaka, SimpleQA, WebQSP * **Y-axis Label (All Charts):** pass@k (%) * **X-axis Label (All Charts):** k * **X-axis Scale (All Charts):** 1, 2, 4, 8, 16 * **Y-axis Scale (Varies by Chart):** * ComplexWebQuestions: 40 to 60 * ExaQT: 35 to 55 * GrailQA: 35 to 55 * Mintaka: 70 to 85 * SimpleQA: 10 to 25 * WebQSP: 60 to 75 * **Legend (Bottom Center):** * `inst`: Cyan line with circle markers * `cot`: Green dash-dot line with diamond markers * `rt`: Purple dotted line with triangle markers * `fs1`: Pink dashed line with square markers ### Detailed Analysis **ComplexWebQuestions** * `inst` (Cyan): Starts at approximately 43%, increases to about 53% at k=16. * `cot` (Green): Starts at approximately 44%, increases to about 54% at k=16. * `rt` (Purple): Starts at approximately 47%, increases to about 58% at k=16. * `fs1` (Pink): Starts at approximately 48%, increases to about 62% at k=16. **ExaQT** * `inst` (Cyan): Starts at approximately 37%, increases to about 48% at k=16. * `cot` (Green): Starts at approximately 37%, increases to about 53% at k=16. * `rt` (Purple): Starts at approximately 41%, increases to about 54% at k=16. * `fs1` (Pink): Starts at approximately 42%, increases to about 57% at k=16. **GrailQA** * `inst` (Cyan): Starts at approximately 35%, increases to about 47% at k=16. * `cot` (Green): Starts at approximately 35%, increases to about 52% at k=16. * `rt` (Purple): Starts at approximately 40%, increases to about 53% at k=16. * `fs1` (Pink): Starts at approximately 41%, increases to about 55% at k=16. **Mintaka** * `inst` (Cyan): Starts at approximately 69%, increases to about 78% at k=16. * `cot` (Green): Starts at approximately 70%, increases to about 83% at k=16. * `rt` (Purple): Starts at approximately 75%, increases to about 84% at k=16. * `fs1` (Pink): Starts at approximately 68%, increases to about 86% at k=16. **SimpleQA** * `inst` (Cyan): Starts at approximately 9%, increases to about 18% at k=16. * `cot` (Green): Starts at approximately 9%, increases to about 22% at k=16. * `rt` (Purple): Starts at approximately 12%, increases to about 23% at k=16. * `fs1` (Pink): Starts at approximately 9%, increases to about 24% at k=16. **WebQSP** * `inst` (Cyan): Starts at approximately 61%, increases to about 70% at k=16. * `cot` (Green): Starts at approximately 61%, increases to about 74% at k=16. * `rt` (Purple): Starts at approximately 66%, increases to about 73% at k=16. * `fs1` (Pink): Starts at approximately 60%, increases to about 78% at k=16. ### Key Observations * **General Trend:** All methods show an increase in `pass@k` as `k` increases across all datasets. The rate of increase diminishes as k increases. * **Relative Performance:** The `fs1` method generally achieves the highest `pass@k` values across most datasets, followed by `rt` and `cot`. The `inst` method typically has the lowest `pass@k` values. * **Dataset Difficulty:** The `pass@k` values vary significantly across datasets, indicating varying levels of difficulty. SimpleQA has the lowest overall performance, while Mintaka has the highest. * **Performance Saturation:** The performance gain diminishes as k increases, suggesting that after a certain point, increasing k provides diminishing returns. ### Interpretation The data suggests that the `fs1` method is generally the most effective among the four methods evaluated for question answering across these datasets. The performance differences between the methods highlight the impact of different approaches on question answering accuracy. The varying performance across datasets indicates that the difficulty of question answering is highly dependent on the specific dataset characteristics. The diminishing returns observed as k increases suggest that optimizing the ranking of the top few predictions is crucial for maximizing performance. </details> Figure 6: Upper-bound Test-Time Scaling for Factual Reasoning. We show with Qwen2.5-32B that parallel scaling is beneficial for complex mQA, measured by pass@ $k$ , especially when fine-tuned on fs1, instead of conducting single-pass inference. 4 Results and Discussion 4.1 Results with Test-time Scaling Parallel scaling can achieve lower latency by enabling multiple (identical) models to run simultaneously locally (via e.g., multiple GPUs or batching techniques) or via API based methods to generate multiple answers. Formally, parallel sampling entails an aggregation technique that combines $N$ independent solutions into a single final prediction, commonly known as a best-of- $N$ approach (Chollet, 2019; Irvine et al., 2023; Brown et al., 2024a; Li et al., 2022). Formally, given a set of $N$ predictions $P=\{p_{1},...,p_{N}\}$ , the best-of- $N$ method selects a prediction $p∈ P$ as the final output. In this work, we present results using pass@ $k$ (see Section 3.2), extending the number of sampled $k$ (until $k$ = 16). In Figure 6, we show parallel scaling results by performing 16 inference runs with Qwen2.5-32B -Instruct, CoT, rt, fs1 on each test dataset. For parallel sampling, we limit ourselves to Qwen2.5-32B as running 16 inferences for 8 models for all 4 settings would require 12.2M model inferences for the test benchmarks, which is computationally prohibitive. As $k$ increases, pass@ $k$ (indicating whether at least one generation is correct) rises steadily across all benchmarks. Parallel sampling boosts the chance of producing a correct answer, especially when fine-tuned on fs1. For example, on CWQ, we see a performance increase of 16 absolute points at $k$ = 16 and on SimpleQA around 6 absolute points at the same $k$ compared to their original instruction-tuned counterpart. <details> <summary>x5.png Details</summary>  ### Visual Description ## Bar Chart: Relative Improvement (RI) by Hops ### Overview The image is a bar chart comparing the relative improvement (RI) of three different methods (cot, rt, fs1) across varying numbers of hops (1, 2, 3, 3+). The y-axis represents the RI in percentage, specifically "RI (%); pass@16", and the x-axis represents the number of hops. ### Components/Axes * **Title:** Relative Improvement (RI) by Hops * **X-axis:** Hops (categorical): 1, 2, 3, 3+ * **Y-axis:** RI (%); pass@16 (numerical): Scale from 0 to 60, with gridlines at intervals of 20. * **Legend:** Located in the top-right corner. * cot (light blue) * rt (teal, with diagonal stripes) * fs1 (dark teal, with diagonal stripes) ### Detailed Analysis Here's a breakdown of the data for each hop count and method: * **Hop 1:** * cot: Approximately 38% * rt: Approximately 36% * fs1: Approximately 33% * **Hop 2:** * cot: Approximately 38% * rt: Approximately 45% * fs1: Approximately 35% * **Hop 3:** * cot: Approximately 34% * rt: Approximately 54% * fs1: Approximately 60% * **Hop 3+:** * cot: Approximately 34% * rt: Approximately 24% * fs1: Approximately 52% ### Key Observations * The 'fs1' method shows the most significant improvement as the number of hops increases from 1 to 3. * The 'rt' method also shows improvement from hop 1 to hop 3, but decreases at hop 3+. * The 'cot' method remains relatively stable across all hop counts, with a slight decrease at hop 3 and 3+. * For hop count 3, 'fs1' has the highest relative improvement, reaching approximately 60%. ### Interpretation The chart suggests that the 'fs1' method benefits the most from an increased number of hops, indicating it might be more effective in scenarios requiring multiple steps or iterations. The 'rt' method shows a similar trend up to 3 hops, but its performance drops off when considering 3+ hops. The 'cot' method appears to be less sensitive to the number of hops, maintaining a consistent level of relative improvement. The "pass@16" likely refers to a metric or threshold used to evaluate the performance of these methods, indicating the percentage of times the method passes this threshold. The data implies that 'fs1' is the most effective method when multiple hops are involved, while 'cot' provides a more consistent performance regardless of the number of hops. The drop in 'rt' performance at 3+ hops could indicate a limitation or instability in that method when applied over a larger number of steps. </details> (a) Performance by number of hops required to answer the question measured in pass@ $16$ . <details> <summary>x6.png Details</summary>  ### Visual Description ## Bar Chart: Relative Improvement (RI) by Answer Type ### Overview The image is a bar chart comparing the relative improvement (RI) of three different methods (cot, rt, fs1) across five answer types: date, place, person, other, and number. The y-axis represents the relative improvement in percentage (%). ### Components/Axes * **Title:** Relative Improvement (RI) by Answer Type * **Y-axis:** RI (%), with tick marks at 0, 10, 20, 30, 40, and 50. * **X-axis:** Answer Type, with categories: date, place, person, other, number. * **Legend:** Located in the top-right corner. * cot (yellow) * rt (teal with diagonal lines) * fs1 (dark blue with diagonal lines) ### Detailed Analysis Here's a breakdown of the relative improvement for each answer type and method: * **Date:** * cot: Approximately 48% * rt: Approximately 42% * fs1: Approximately 55% * **Place:** * cot: Approximately 33% * rt: Approximately 21% * fs1: Approximately 19% * **Person:** * cot: Approximately 24% * rt: Approximately 26% * fs1: Approximately 13% * **Other:** * cot: Approximately 19% * rt: Approximately 30% * fs1: Approximately 31% * **Number:** * cot: Approximately 17% * rt: Approximately 28% * fs1: Approximately 30% ### Key Observations * fs1 shows the highest relative improvement for the "date" answer type. * cot generally has a higher relative improvement for "place" and "person" answer types compared to "other" and "number". * For "other" and "number" answer types, rt and fs1 show similar relative improvements. * The relative improvement varies significantly depending on the answer type and method used. ### Interpretation The bar chart illustrates the performance of different methods (cot, rt, fs1) in improving the accuracy of answers for various question types. The data suggests that the effectiveness of each method is highly dependent on the type of answer being sought. For example, fs1 appears to be particularly effective for "date" questions, while cot shows better performance for "place" and "person" questions. The chart highlights the importance of selecting the appropriate method based on the specific type of question being addressed. The "other" and "number" categories show a more consistent performance between rt and fs1, suggesting these methods might be more generally applicable or that these question types are less sensitive to the specific method used. </details> (b) Performance by answer type (i.e., what type of entity the answer is) in pass@ $16$ . <details> <summary>x7.png Details</summary>  ### Visual Description ## Chart Type: Bar Chart ### Overview The image is a bar chart comparing the relative improvement (RI) by domain for three different methods: 'cot', 'rt', and 'fs1'. The y-axis represents the RI (%) at pass@16, and the x-axis represents different domains such as art, sports, other, geography, tv shows, video games, politics, music, sci & tech, and history. ### Components/Axes * **Title:** Relative Improvement (RI) by Domain * **X-axis:** Domains (art, sports, other, geography, tv shows, video games, politics, music, sci & tech, history) * **Y-axis:** RI (%); pass@16, with values ranging from 0 to 80 in increments of 20. * **Legend (Top-Right):** * cot (light purple) * rt (dark purple with black horizontal lines) * fs1 (light red with black diagonal lines) ### Detailed Analysis Here's a breakdown of the approximate values for each domain and method: * **Art:** * cot: ~63% * rt: ~73% * fs1: ~59% * **Sports:** * cot: ~50% * rt: ~38% * fs1: ~49% * **Other:** * cot: ~42% * rt: ~35% * fs1: ~49% * **Geography:** * cot: ~36% * rt: ~20% * fs1: ~48% * **TV Shows:** * cot: ~34% * rt: ~35% * fs1: ~53% * **Video Games:** * cot: ~62% * rt: ~64% * fs1: ~84% * **Politics:** * cot: ~32% * rt: ~29% * fs1: ~42% * **Music:** * cot: ~28% * rt: ~22% * fs1: ~17% * **Sci & Tech:** * cot: ~18% * rt: ~13% * fs1: ~13% * **History:** * cot: ~3% * rt: ~20% * fs1: ~20% ### Key Observations * The 'fs1' method shows the highest relative improvement in the "video games" domain, significantly outperforming 'cot' and 'rt'. * The 'cot' method shows the highest relative improvement in the "art" domain. * The 'rt' method shows the highest relative improvement in the "art" domain. * All three methods show the lowest relative improvement in the "history" domain. * 'fs1' generally shows a higher relative improvement than 'cot' and 'rt' across most domains. ### Interpretation The bar chart illustrates the relative performance of three different methods ('cot', 'rt', and 'fs1') across various domains. The 'fs1' method appears to be particularly effective in the "video games" domain, suggesting it may be well-suited for tasks within that area. Conversely, all methods struggle with the "history" domain, indicating a potential area for further research and improvement. The differences in performance across domains highlight the importance of considering the specific application when selecting a method. </details> (c) Performance measured per domain in pass@ $16$ . Figure 7: Relative Improvements across Different Axes. We show the relative performance improvement (%) at pass@ $16$ of different Qwen-32B (i.e., CoT, rt and fs1) against the original instruct model. In (a), we show the performance of the models by the number of hops required to answer the question. In (b), we show the performance of the models by answer type. In (c), we show the performance by domain of the question. Absolute numbers are in Figure 13 (Appendix F). 4.2 What type of samples do models seem to fail on? In Figure 7, we investigate what kind of questions the model (Qwen2.5-32B) seems to fail on. We take metadata information from SimpleQA (Wei et al., 2024a), which indicates the question difficulty in number of hops required to answer the question (Figure 7(a)), type of answer (Figure 7(b)) and the domain of the question (Figure 7(c)). For question difficulty, we source the number of hops for each question in SimpleQA from Lavrinovics et al. (2025). We count the number of relations (P) from Wikidata, which would indicate the number of hops required to go from entity A to B. When a question does not contain any relations, we assume it takes more than 3 hops to answer the question. In Figure 7(a), we observe that fs1 has lower relative improvements on easier questions (e.g., 1 or 2 hops required), but outperforms the other models when the question gets more complex (3 or more hops required). This indicates that inducing KG paths helps answering complex questions. In Figure 7(b), we show that fs1 has the most relative improvement on numerical answers, such as numbers, dates and also miscellaneous answer types. Last, in Figure 7(c), fs1 performs best on questions related to video games, geography, politics, music, and miscellaneous questions. Additionally, rt performs best on art and history-related questions. Last, CoT performs best on questions related to sports. 4.3 Single pass results across scale In Table 3, we show results in terms of accuracy via LLM-as-a-Judge at pass@ $1$ (i.e., one inference run per question) on all test datasets. For the baselines, we observe that o3-mini is the dominant model, achieving the highest score on five out of six datasets, such as its 0.774 accuracy on Mintaka and 0.680 on WebQSP. The only exception is SimpleQA, where R1-70B performs best with a score of 0.188. These are followed by Qwen2.5-72B-Instruct and QwQ-32B in overall performance. Observing the Qwen2.5 results, the benefits of fine-tuning on rt and fs1 are most pronounced at the sub-billion parameter scale. For instance, fine-tuning the 0.5B model on fs1 yields substantial relative gains across all tasks, peaking at a +74.6% on WebQSP. However, as model size increases, the performance differences become more nuanced. For the 1.5B model, the same fs1 fine-tuning leads to performance degradation on four out of six datasets, such as ExaQT (-4.7%) and WebQSP (-1.1%). While larger models like the 32B still benefit from fine-tuning (e.g., rt and fs1 are often the best performers in their group), the relative gains are smaller than those seen at the 0.5B scale. Our results also show that fine-tuning improvements do not uniformly generalize across different model families at the sub-billion parameter scale. A comparison between the fine-tuned Qwen2.5 and SmolLM2 models reveals a significant performance divergence. Specifically, fine-tuning on fs1 provided consistent enhancements for the Qwen2.5-0.5B model, improving its CWQ score from 0.135 to 0.209. In contrast, the same fine-tuning on SmolLM2-360M yielded mixed results; while it improved performance on most tasks, it caused a notable degradation of -15.9% on GrailQA. This variance diminishes with scale, as models at the 1.5B/1.7B parameter scale exhibit more convergent behavior. For example, fine-tuning with rt on GrailQA provides a nearly identical small boost to both Qwen2.5-1.5B (+1.9%) and SmolLM2-1.7B (+1.8%). Overall, We hypothesize this scale-dependent effect may occur because larger models (e.g., 32B) possess stronger parametric knowledge, making them less reliant on the explicit guidance from KG paths. Table 3: Single Pass (pass@1) Results on mQA Benchmarks. We show accuracy and relative performance gains on our benchmarks for several baselines, Qwen2.5, and SmolLM2 models. For each size, we show the original instruction-tuned model followed by versions fine-tuned with chain-of-thought, rt, and fs1. Parentheses indicate the relative improvement over the instruction-tuned counterpart. The benefits of fine-tuning are most pronounced for smaller models. | Model Large Language Model Baselines Qwen2.5-72B | CWQ 0.481 | ExaQT 0.440 | GrailQA 0.361 | SimpleQA 0.117 | Mintaka 0.736 | WebQSP 0.653 | | --- | --- | --- | --- | --- | --- | --- | | QwQ-32B | 0.479 | 0.390 | 0.358 | 0.097 | 0.708 | 0.612 | | R1-70B | 0.501 | 0.476 | 0.340 | 0.188 | 0.755 | 0.549 | | o3-mini | 0.558 | 0.497 | 0.438 | 0.138 | 0.774 | 0.680 | | Small Language Models (0.36B-1.7B) | | | | | | | | SmolLM2-360M | 0.148 | 0.088 | 0.164 | 0.024 | 0.175 | 0.235 | | + cot | 0.151 (+2.0 %) | 0.101 (+14.8 %) | 0.169 (+3.0 %) | 0.025 (+4.2 %) | 0.188 (+7.4 %) | 0.230 (-2.1 %) | | + rt | 0.192 (+29.7 %) | 0.111 (+26.1 %) | 0.156 (-4.9 %) | 0.029 (+20.8 %) | 0.202 (+15.4 %) | 0.293 (+24.7 %) | | + fs1 | 0.179 (+20.9 %) | 0.093 (+5.7 %) | 0.138 (-15.9 %) | 0.027 (+12.5 %) | 0.197 (+12.6 %) | 0.264 (+12.3 %) | | Qwen2.5-0.5B | 0.135 | 0.058 | 0.127 | 0.023 | 0.131 | 0.173 | | + cot | 0.161 (+19.3 %) | 0.104 (+79.3 %) | 0.141 (+11.0 %) | 0.031 (+34.8 %) | 0.214 (+63.4 %) | 0.234 (+35.3 %) | | + rt | 0.190 (+40.7 %) | 0.089 (+53.4 %) | 0.155 (+22.0 %) | 0.022 (-4.3 %) | 0.178 (+35.9 %) | 0.286 (+65.3 %) | | + fs1 | 0.209 (+54.8 %) | 0.101 (+74.1 %) | 0.166 (+30.7 %) | 0.035 (+52.2 %) | 0.202 (+54.2 %) | 0.302 (+74.6 %) | | Qwen2.5-1.5B | 0.234 | 0.170 | 0.208 | 0.031 | 0.316 | 0.360 | | + cot | 0.252 (+7.7 %) | 0.179 (+5.3 %) | 0.216 (+3.8 %) | 0.041 (+32.3 %) | 0.318 (+0.6 %) | 0.391 (+8.6 %) | | + rt | 0.255 (+9.0 %) | 0.173 (+1.8 %) | 0.212 (+1.9 %) | 0.038 (+22.6 %) | 0.294 (-7.0 %) | 0.360 (+0.0 %) | | + fs1 | 0.263 (+12.4 %) | 0.162 (-4.7 %) | 0.204 (-1.9 %) | 0.035 (+12.9 %) | 0.301 (-4.7 %) | 0.356 (-1.1 %) | | SmolLM2-1.7B | 0.248 | 0.176 | 0.219 | 0.032 | 0.293 | 0.408 | | + cot | 0.285 (+14.9 %) | 0.177 (+0.6 %) | 0.209 (-4.6 %) | 0.032 (+0.0 %) | 0.295 (+0.7 %) | 0.409 (+0.2 %) | | + rt | 0.306 (+23.4 %) | 0.184 (+4.5 %) | 0.223 (+1.8 %) | 0.038 (+18.7 %) | 0.366 (+24.9 %) | 0.454 (+11.3 %) | | + fs1 | 0.305 (+23.0 %) | 0.179 (+1.7 %) | 0.218 (-0.5 %) | 0.036 (+12.5 %) | 0.341 (+16.4 %) | 0.426 (+4.4 %) | | Large Language Models (3B-32B) | | | | | | | | Qwen2.5-3B | 0.317 | 0.214 | 0.252 | 0.044 | 0.396 | 0.466 | | + cot | 0.302 (-4.7 %) | 0.222 (+3.7 %) | 0.248 (-1.6 %) | 0.048 (+9.1 %) | 0.431 (+8.8 %) | 0.477 (+2.4 %) | | + rt | 0.363 (+14.5 %) | 0.235 (+9.8 %) | 0.279 (+10.7 %) | 0.053 (+20.5 %) | 0.495 (+25.0 %) | 0.483 (+3.6 %) | | + fs1 | 0.330 (+4.1 %) | 0.205 (-4.2 %) | 0.253 (+0.4 %) | 0.045 (+2.3 %) | 0.444 (+12.1 %) | 0.406 (-12.9 %) | | Qwen2.5-7B | 0.376 | 0.281 | 0.299 | 0.070 | 0.548 | 0.580 | | + cot | 0.383 (+1.9 %) | 0.292 (+3.9 %) | 0.295 (-1.3 %) | 0.062 (-11.4 %) | 0.580 (+5.8 %) | 0.565 (-2.6 %) | | + rt | 0.401 (+6.6 %) | 0.296 (+5.3 %) | 0.300 (+0.3 %) | 0.067 (-4.3 %) | 0.576 (+5.1 %) | 0.517 (-10.9 %) | | + fs1 | 0.408 (+8.5 %) | 0.272 (-3.2 %) | 0.303 (+1.3 %) | 0.053 (-24.3 %) | 0.551 (+0.5 %) | 0.492 (-15.2 %) | | Qwen2.5-14B | 0.392 | 0.336 | 0.318 | 0.068 | 0.624 | 0.599 | | + cot | 0.422 (+7.7 %) | 0.356 (+6.0 %) | 0.322 (+1.3 %) | 0.080 (+17.6 %) | 0.664 (+6.4 %) | 0.592 (-1.2 %) | | + rt | 0.451 (+15.1 %) | 0.352 (+4.8 %) | 0.331 (+4.1 %) | 0.082 (+20.6 %) | 0.678 (+8.7 %) | 0.562 (-6.2 %) | | + fs1 | 0.454 (+15.8 %) | 0.339 (+0.9 %) | 0.328 (+3.1 %) | 0.079 (+16.2 %) | 0.654 (+4.8 %) | 0.558 (-6.8 %) | | Qwen2.5-32B | 0.428 | 0.362 | 0.334 | 0.087 | 0.674 | 0.621 | | + cot | 0.435 (+1.6 %) | 0.366 (+1.1 %) | 0.332 (-0.6 %) | 0.099 (+13.8 %) | 0.696 (+3.3 %) | 0.614 (-1.1 %) | | + rt | 0.471 (+10.0 %) | 0.366 (+1.1 %) | 0.342 (+2.4 %) | 0.094 (+8.0 %) | 0.680 (+0.9 %) | 0.563 (-9.3 %) | | + fs1 | 0.477 (+11.4 %) | 0.361 (-0.3 %) | 0.344 (+3.0 %) | 0.078 (-10.3 %) | 0.682 (+1.2 %) | 0.576 (-7.2 %) | 5 Related Work Different methods that involve long chain-of-thought processes (Kojima et al., 2022; Wei et al., 2022) involving reflection, backtracking, thinking (e.g., DeepSeek-AI, 2025; Muennighoff et al., 2025), self-consistency (e.g., Wang et al., 2023), and additional computation at inference time, such as test-time scaling (Wu et al., 2024; Muennighoff et al., 2025; Zhang et al., 2025), have shown promising improvements in LLM performance on complex reasoning tasks. Our work intersects with efforts in factuality, knowledge graph grounding, and test-time scaling. Graph-enhanced In-context Learning. Enhancing the factual consistency of LLMs using KGs has been explored in different directions, including semantic parsing methods that convert natural language questions into formal KG queries (Lan & Jiang, 2020; Ye et al., 2022). Retrieval-augmented methods (KG-RAG) (Li et al., 2023; Sanmartin, 2024; Jiang et al., 2023) aim to reduce LLMs’ reliance on latent knowledge by incorporating explicit, structured information from a KG; reasoning on graphs (RoG) models (Luo et al., 2024) generate relation paths grounded by KGs as faithful paths for the model to follow. Recent works like Tan et al. (2025) also use KG paths to guide reasoning, our fs1 approach focuses on distilling and fine-tuning general instruction models on these factually-grounded traces, and we provide a broad empirical study on its effect across model scales. Long Form Factuality. Factuality in NLP involves multiple challenges (Augenstein et al., 2024), and while prior efforts have established reasoning datasets like SAFE (Wei et al., 2024b) and SimpleQA (Wei et al., 2024a), they often lack explicit grounding in structured knowledge subgraphs. In contrast, Tian et al. (2024) directly address factual accuracy by fine-tuning models on automatically generated preference rankings that prioritize factual consistency. Test-Time Scaling as a Performance Upper-Bound. Our evaluation using pass@ $k$ is situated within the broader context of test-time scaling, which seeks to improve performance by dedicating more compute at inference. This field encompasses parallel scaling (e.g., Best-of-N), where multiple candidate solutions are generated to increase the probability of finding a correct one (Chollet, 2019; Irvine et al., 2023; Brown et al., 2024a; Li et al., 2022), and sequential scaling, where a single solution is iteratively refined through techniques like chain-of-thought prompting and revision (Wei et al., 2022; Nye et al., 2021; Madaan et al., 2023; Lee et al., 2025; Hou et al., 2025; Huang et al., 2023; Min et al., 2024; Muennighoff et al., 2025; Wang et al., 2024b; Li et al., 2025; Jurayj et al., 2025). While practical applications of parallel scaling depend on a selection mechanism (e.g., majority voting or reward-model-based scoring) to choose the final answer (Wang et al., 2023; Christiano et al., 2017; Lightman et al., 2024; Wang et al., 2024a; Wu et al., 2024; Beeching et al., 2025; Pan et al., 2024; Hassid et al., 2024; Stroebl et al., 2024), the performance of any such method is fundamentally limited by the quality of the underlying generations, often facing diminishing returns (Brown et al., 2024b; Snell et al., 2024; Wu et al., 2024; Levi, 2024). Our work, therefore, focuses on improving the quality of each individual reasoning trace through fine-tuning, thereby directly boosting the upper-bound potential that is measured by pass@ $k$ . Domain-specific Test-Time Scaling. Test-time scaling also spans specific domains like coding and medicine. Z1-7B optimizes coding tasks through constrained reasoning windows, reducing overthinking while maintaining accuracy (Yu et al., 2025). In medicine, extended reasoning boosts smaller models’ clinical QA performance significantly (Huang et al., 2025), complemented by structured datasets like MedReason, which enhance factual reasoning via knowledge-graph-guided paths (Wu et al., 2025), similar to our work. 6 Conclusion In this work, we have investigated whether grounding reasoning traces on knowledge graph paths and training models on them yield tangible gains in factual accuracy on complex open-domain QA tasks. After distilling over 3K original and knowledge-graph-enhanced reasoning traces from models such as QwQ-32B and Deepseek-R1, we fine-tuned 8 LLMs on rt and fs1 and evaluated them across 6 diverse benchmarks. In short, with parallel sampling, we consistently improve 6-14 absolute points in accuracy over their instruction-tuned counterpart. Particularly, using SimpleQA, we highlight that CoT and rt perform better on simpler questions (1 or 2 hops required), whereas our fs1 -tuned model performs better on more complex questions, requiring 3 hops or more. Lastly, we examined the performance of eight fs1 -tuned models across different parameter scales, finding that smaller models (below the 1.7B parameter range) show the largest increase in performance, while larger models see less profound improvements in a pass@1 setting. By releasing all code, models, and reasoning traces, we provide a rich resource for future work on process-level verification and the development of factuality-aware reward models. In turn, we hope this work facilitates more factual large language models, making them more useful for real-world usage. Limitations. Our approach assumes that conditioning on KG paths improves the accuracy of reasoning traces, though it does not guarantee perfect intermediate processes. Additionally, accurately evaluating entity answers poses challenges; we attempted to mitigate this limitation using LLM-based judgments, but these methods have their own inherent limitations. For evaluation, we note that pass@ $k$ is an upper-bound performance measure. A practical implementation would require an additional selection mechanism, such as majority voting or a verifier model, to choose the final answer. Last, some of the the test datasets used might be on the older side and English only, where we do not have control on whether the data has been included in any type of LLM pre- or post-training. Future Work. Several future research directions emerge from both the KG and test-time scaling perspectives. One promising avenue is leveraging these reasoning traces to develop process reward models, which are designed for complex reasoning and decision-making tasks where evaluating intermediate (factual reasoning) steps is critical to achieving the desired outcomes. This in fact is a crucial step towards more factual LLMs. This can be done together with KGs, one possible example could be (Amayuelas et al., 2025), where they attempt to ground every generation with a knowledge graph entry. This can possibly be done during the generation of long reasoning. Ethics Statement The primary ethical motivation for this research is to enhance the factuality and reliability of LLMs. By addressing the probability of these models to generate incorrect information, our work aims to contribute positively to the development of more trustworthy AI systems. We do not foresee any direct negative ethical implications arising from this research. Instead, our goal is to provide a methodology that mitigates existing risks associated with misinformation, thereby promoting a safer and more beneficial application of language technologies. Reproducibility Statement We are committed to full reproducibility. All associated artifacts, such as source code, datasets, and pretrained model weights, will be made publicly available via GitHub and the Huggingface Hub upon publication. Further details regarding the computational environment, including hardware specifications, software dependencies, and hyperparameters used for fine-tuning and inference, are documented in Section 3 and Appendix B. We acknowledge that minor variations in numerical results may arise from discrepancies in hardware or software versions; however, we have provided sufficient detail to allow for a faithful replication of our experimental setup. Acknowledgments MZ and JB are supported by the research grant (VIL57392) from VILLUM FONDEN. We would like to thank the AAU-NLP group for helpful discussions and feedback on an earlier version of this article. We acknowledge the Danish e-Infrastructure Cooperation for awarding this project access (No. 465001263; DeiC-AAU-N5-2024078 - H2-2024-18) to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through DeiC, Denmark. References - Amayuelas et al. (2025) Alfonso Amayuelas, Joy Sain, Simerjot Kaur, and Charese Smiley. Grounding llm reasoning with knowledge graphs, 2025. URL https://arxiv.org/abs/2502.13247. - Augenstein et al. (2024) Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, et al. Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence, 6(8):852–863, 2024. - Beeching et al. (2025) Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2025. URL https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute. - Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pp. 1247–1250, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581026. doi: 10.1145/1376616.1376746. URL https://doi.org/10.1145/1376616.1376746. - Brown et al. (2024a) Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. ArXiv preprint, abs/2407.21787, 2024a. URL https://arxiv.org/abs/2407.21787. - Brown et al. (2024b) Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. ArXiv preprint, abs/2407.21787, 2024b. URL https://arxiv.org/abs/2407.21787. - Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. - Chollet (2019) François Chollet. On the measure of intelligence, 2019. - Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299–4307, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html. - DeepSeek-AI (2025) DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv preprint, abs/2501.12948, 2025. URL https://arxiv.org/abs/2501.12948. - Gu et al. (2021) Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond I.I.D.: three levels of generalization for question answering on knowledge bases. In Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (eds.), WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pp. 3477–3488. ACM / IW3C2, 2021. doi: 10.1145/3442381.3449992. URL https://doi.org/10.1145/3442381.3449992. - Hassid et al. (2024) Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The larger the better? improved llm code-generation via budget reallocation. ArXiv preprint, abs/2404.00725, 2024. URL https://arxiv.org/abs/2404.00725. - Hou et al. (2025) Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling. ArXiv preprint, abs/2501.11651, 2025. URL https://arxiv.org/abs/2501.11651. - Huang et al. (2023) Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051–1068, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.67. URL https://aclanthology.org/2023.emnlp-main.67. - Huang et al. (2025) Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the potential of test-time scaling for medical reasoning with large language models, 2025. URL https://arxiv.org/abs/2504.00869. - Irvine et al. (2023) Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp. Rewarding chatbots for real-world engagement with millions of users. ArXiv preprint, abs/2303.06135, 2023. URL https://arxiv.org/abs/2303.06135. - Jia et al. (2021) Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. Complex temporal question answering on knowledge graphs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM ’21, pp. 792–802, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384469. doi: 10.1145/3459637.3482416. URL https://doi.org/10.1145/3459637.3482416. - Jiang et al. (2023) Jinhao Jiang, Kun Zhou, Xin Zhao, Yaliang Li, and Ji-Rong Wen. ReasoningLM: Enabling structural subgraph reasoning in pre-trained language models for question answering over knowledge graph. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3721–3735, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.228. URL https://aclanthology.org/2023.emnlp-main.228. - Jurayj et al. (2025) William Jurayj, Jeffrey Cheng, and Benjamin Van Durme. Is that your final answer? test-time scaling improves selective question answering. ArXiv preprint, abs/2502.13962, 2025. URL https://arxiv.org/abs/2502.13962. - Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html. - Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. - Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. ArXiv preprint, abs/1910.09700, 2019. URL https://arxiv.org/abs/1910.09700. - Lan & Jiang (2020) Yunshi Lan and Jing Jiang. Query graph generation for answering multi-hop complex questions from knowledge bases. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 969–974, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.91. URL https://aclanthology.org/2020.acl-main.91. - Lavrinovics et al. (2025) Ernests Lavrinovics, Russa Biswas, Katja Hose, and Johannes Bjerva. Multihal: Multilingual dataset for knowledge-graph grounded evaluation of llm hallucinations, 2025. URL https://arxiv.org/abs/2505.14101. - Lee et al. (2025) Kuang-Huei Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. Evolving deeper llm thinking. ArXiv preprint, abs/2501.09891, 2025. URL https://arxiv.org/abs/2501.09891. - Levi (2024) Noam Levi. A simple model of inference scaling laws. ArXiv preprint, abs/2410.16377, 2024. URL https://arxiv.org/abs/2410.16377. - Li et al. (2025) Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Llms can easily learn to reason from demonstrations: structure, not content, is what matters! ArXiv preprint, abs/2502.07374, 2025. URL https://arxiv.org/abs/2502.07374. - Li et al. (2023) Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. Graph reasoning for question answering with triplet retrieval. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 3366–3375, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.208. URL https://aclanthology.org/2023.findings-acl.208. - Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Gabriel Synnaeve, David Imrie, Peter Ying, Peter Battaglia, and Oriol Vinyals. Competition-level code generation with alphacode. ArXiv preprint, abs/2203.07814, 2022. URL https://arxiv.org/abs/2203.07814. - Lightman et al. (2024) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=v8L0pN6EOi. - Luo et al. (2024) Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=ZGNWW7xZ6Q. - Ma et al. (2025) Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking, 2025. URL https://arxiv.org/abs/2504.09858. - Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html. - Min et al. (2024) Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Feng Ji, Qi Zhang, and Fuli Luo. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. ArXiv preprint, abs/2412.09413, 2024. URL https://arxiv.org/abs/2412.09413. - Muennighoff et al. (2025) Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. ArXiv preprint, abs/2501.19393, 2025. URL https://arxiv.org/abs/2501.19393. - Nye et al. (2021) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Wojciech Zaremba, Illya Sutskever, Charles Sutton, Martin Wattenberg, Michael Terry, Jeff Dean, Douglas Eck, Bastien Potier, and Ethan Dyer. Show your work: Scratchpads for intermediate computation with language models. ArXiv preprint, abs/2112.00114, 2021. URL https://arxiv.org/abs/2112.00114. - OpenAI (2025) OpenAI. Openai o3-mini: Pushing the frontier of cost-effective reasoning., 2025. URL https://openai.com/index/openai-o3-mini/. - Pan et al. (2024) Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. ArXiv preprint, abs/2412.21139, 2024. URL https://arxiv.org/abs/2412.21139. - Qwen Team (2024) Qwen Team. Qwen2.5: A party of foundation models, 2024. URL https://qwenlm.github.io/blog/qwen2.5/. - Qwen Team (2025) Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, 2025. URL https://qwenlm.github.io/blog/qwq-32b/. - Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410/. - Sanmartin (2024) Diego Sanmartin. Kg-rag: Bridging the gap between knowledge and creativity. ArXiv preprint, abs/2405.12035, 2024. URL https://arxiv.org/abs/2405.12035. - Sen et al. (2022) Priyanka Sen, Alham Fikri Aji, and Amir Saffari. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 1604–1619, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclanthology.org/2022.coling-1.138/. - Snell et al. (2024) Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. ArXiv preprint, abs/2408.03314, 2024. URL https://arxiv.org/abs/2408.03314. - Stroebl et al. (2024) Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of llm resampling with imperfect verifiers. ArXiv preprint, abs/2411.17501, 2024. URL https://arxiv.org/abs/2411.17501. - Talmor & Berant (2018) Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 641–651, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1059. URL https://aclanthology.org/N18-1059/. - Tan et al. (2025) Xingyu Tan, Xiaoyang Wang, Qing Liu, Xiwei Xu, Xin Yuan, and Wenjie Zhang. Paths-over-graph: Knowledge graph empowered large language model reasoning. In Proceedings of the ACM on Web Conference 2025, pp. 3505–3522, 2025. - Tian et al. (2024) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=WPZ2yPag4K. - Wang et al. (2024a) Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426–9439, 2024a. - Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=1PL1NIMMrw. - Wang et al. (2024b) Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024b. URL http://papers.nips.cc/paper_files/paper/2024/hash/a399456a191ca36c7c78dff367887f0a-Abstract-Conference.html. - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. - Wei et al. (2024a) Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. ArXiv preprint, abs/2411.04368, 2024a. URL https://arxiv.org/abs/2411.04368. - Wei et al. (2024b) Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. Long-form factuality in large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024b. URL http://papers.nips.cc/paper_files/paper/2024/hash/937ae0e83eb08d2cb8627fe1def8c751-Abstract-Conference.html. - Wu et al. (2025) Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, Yihan Cao, Hui Ren, Xiang Li, Xiaoxiao Li, and Yuyin Zhou. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs, 2025. URL https://arxiv.org/abs/2504.00993. - Wu et al. (2024) Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. ArXiv preprint, abs/2408.00724, 2024. URL https://arxiv.org/abs/2408.00724. - Ye et al. (2022) Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. RNG-KBQA: Generation augmented iterative ranking for knowledge base question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6032–6043, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.417. URL https://aclanthology.org/2022.acl-long.417. - Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 201–206, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-2033. URL https://aclanthology.org/P16-2033/. - Yu et al. (2025) Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Z1: Efficient test-time scaling with code, 2025. URL https://arxiv.org/abs/2504.00810. - Zhang et al. (2025) Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models, 2025. URL https://arxiv.org/abs/2503.24235. Appendix A Large Language Model Use We made use of LLMs to polish our writing and plotting our figures. Appendix B Training and Inference For running Deepseek-R1, o3-mini, and some LLM-as-a-Judge experiments with gpt-4o-mini, we use API-based solutions via OpenAI https://platform.openai.com/ or TogetherAI https://api.together.ai/. The costs of running inference on all data took around 250 USD. For fine-tuning and running inference of the local models, we make use of a large HPC cluster with hardware configurations comprising multiple nodes (depending on model size; e.g., 32B models require 4 nodes for training and 1 node for inference), each with node contains eight AMD MI250x GPU modules alongside a single 64-core AMD EPYC “Trento” CPU. The library we use for inference is vllm (Kwon et al., 2023). For all the experiments it resulted in around 6,500 GPU hours spent. B.1 Environmental Impact We acknowledge that conducting a large-scale analysis using LLMs comes with an environmental impact. Experiments were conducted using private infrastructure in [Redacted] running on green energy. A cumulative of 6,500 GPU hours of computation was performed on AMD MI250x GPU modules, which has a TDP of 500 Watts. The experiments were ran from February to September 2025. During this time, the average carbon efficiency in [Redacted] was 0.085 $kg/kWh$ . According to https://app.electricitymaps.com/map. This means we released about 276 $kg$ of $CO_{2}$ equivalent. Estimations were conducted using the Machine Learning Impact calculator Find the tool here: https://mlco2.github.io/impact. presented in (Lacoste et al., 2019). Appendix C Example Reasoning Traces We show four full examples in Figure 8 and Figure 9 for rt and Figure 10 and Figure 11 for fs1. For visualization purposes, we split up the specific parts in special tags: <question>Q</question>, <think>T</think>, <gold>G</gold> to indicate what it refers to. Deepseek-R1 Example (rt) ⬇ < question > In Family Guy, Seth Macfarlane plays the parent of Chris Griffin. What is his character ’ s name? </ question > < think > Okay, so I need to figure out the name of the character Seth MacFarlane voices who is the parent of Chris Griffin in Family Guy. Let me start by recalling what I know about the show.\ n \ nFirst, Family Guy is an animated sitcom created by Seth MacFarlane. He ’ s known for voicing several characters in the show. The main family is the Griffins. The children in the family are Chris, Meg, and Stewie, and there ’ s also the talking dog Brian. The parents would be the father and mother of these kids.\ n \ nI remember that the father is Peter Griffin, who is the bumbling, overweight patriarch of the family. Seth MacFarlane does voice Peter Griffin, along with other characters like Stewie and Brian. The mother is Lois Griffin, but I think she ’ s voiced by Alex Borstein, not Seth MacFarlane. So, since the question is about the parent of Chris Griffin that Seth MacFarlane plays, it must be Peter Griffin.\ n \ nWait, let me double - check. Chris is the son, so his parents are Peter and Lois. Since Seth MacFarlane voices Peter, that ’ s the answer. But just to be thorough, are there any other parents in the show that Seth voices? For example, maybe a grandparent? But Chris ’ s grandparents would be Peter ’ s parents or Lois ’ s parents. I think Peter ’ s father is named Francis Griffin, but I don ’ t recall if Seth voices him. Lois ’ s parents are Carter and Babs Pewterschmidt, and they are voiced by other actors. So no, the main parent characters voiced by Seth would be Peter. \ n \ nAlso, Stewie is the baby, and Brian is the dog, both voiced by Seth, but they aren ’ t parents. So yes, the answer should be Peter Griffin. </ think > The character voiced by Seth MacFarlane who is the parent of Chris Griffin in * Family Guy * is \\ boxed {Peter Griffin}. < gold > [” Peter Griffin ”, ” Peter Lowenbrou Griffin ”] </ gold > Figure 8: Example Instance (rt). A successful reasoning trace for a simple multi-hop question by Deepseek-R1. QwQ-32B Example (rt) ⬇ < question > What Eastern Europe main religion worshipers also attend St. Mary ’ s Cathedral, in Batticaloa, Sri Lanka? </ question > < think > Okay, so the user is asking about the main religion worshippers in Eastern Europe who attend St. Mary ’ s Cathedral in Batticaloa, Sri Lanka. Hmm, first I need to break this down. Let me start by recalling that Batticaloa is a city in Sri Lanka, which is in South Asia. The cathedral mentioned is a Christian church, specifically a Catholic cathedral since it ’ s named St. Mary ’ s.\ n \ nNow, the question is about Eastern Europeans who worship there. Wait, Eastern Europe ’ s main religions are predominantly Christian, like Eastern Orthodox or Roman Catholic in some countries. But Sri Lanka ’ s main religions are Buddhism, Hinduism, Islam, and then there ’ s a Christian minority. So why would Eastern Europeans be attending a cathedral there? Maybe there ’ s a community of Eastern Europeans living or working in Batticaloa? Or perhaps it ’ s a question about the cathedral ’ s affiliation?\ n \ nAlternatively, maybe there ’ s confusion in the question. St. Mary ’ s Cathedral in Batticaloa is likely part of the Roman Catholic Church. Eastern Europeans who are Catholic, like from countries such as Poland, Lithuania, or parts of Croatia, might attend. But Eastern Orthodox Christians would typically go to their own churches. Since the cathedral is Catholic, the main worshippers there would be local Catholics, but the question specifies Eastern Europeans. \ n \ nWait, perhaps the user is thinking of the term \” Eastern Europe \” in a different context. Maybe they ’ re referring to the Eastern Catholic Churches, which are in communion with Rome but have Eastern rites. For example, Ukrainian Catholics or Ruthenian Catholics. But would they be in Sri Lanka? Maybe there ’ s a small expat community or missionary group. Alternatively, maybe the question is mixing up terms, and the answer is Roman Catholic, as that ’ s the cathedral ’ s denomination, and Eastern Europeans of that faith would attend. \ n \ nAlternatively, could there be a confusion with Batticaloa ’ s demographics? The area has a significant Tamil population, but Tamils are mostly Hindu or Christian. The cathedral serves the local Catholic community, which includes Sri Lankan Tamils and others. But the question specifically asks about Eastern Europeans. Maybe the answer is Roman Catholics from Eastern Europe, but I need to confirm if there ’ s a specific group. \ n \ nAlternatively, perhaps the question is a trick, and the main religion in Eastern Europe (like Orthodox) isn ’ t the one at the cathedral, so the answer is Roman Catholics. Since the cathedral is Catholic, Eastern European Catholics would be the ones attending. So the main religion worshippers there from Eastern Europe would be Roman Catholics. \ n \ nI think that ’ s the most logical answer here. The cathedral is Catholic, so Eastern Europeans who are Roman Catholic would attend. The main religions in Eastern Europe include Orthodox and Catholic, so the Catholic ones from there would be the answer. </ think > \\ boxed {Roman Catholics}. < gold > [” Catholicism ”, ” Roman Catholic convert ”, ” Roman Catholic ”] </ gold > Figure 9: Example Instance (rt). A successful reasoning trace for a simple multi-hop question by QwQ-32B. QwQ-32B Example (fs1) ⬇ < question > What geographic region is the country that used the Ottoman Lira used? While answering the question, make use of the following linearised graph as an inspiration in your reasoning, not as the only answer: Konstantin Mikhailovich Bazili, place of birth, Istanbul Istanbul, work location, Near East. Put your final answer within \ boxed {}. </ question > < think > Okay, let ’ s tackle this question step by step. The user is asking about the geographic region of the country that used the Ottoman Lira. First, I need to recall what the Ottoman Lira was. From what I remember, the lira was the currency used in the Ottoman Empire. So the country in question here is the Ottoman Empire. But since the Ottoman Empire is a historical entity and not a current country, the region would refer to where the empire was located. Now, the Ottoman Empire was a large and extensive empire that existed from the 14 th century until after World War I. Its heartland was in Anatolia (modern - day Turkey) and Southeast Europe. The capital was Constantinople, which is now known as Istanbul, Turkey. The empire also controlled much of the Balkans, parts of North Africa, and the Middle East. … (cutoff) … Wait, but the question is about the geographic region of the country that used the Ottoman Lira. The country is the Ottoman Empire. The geographic region that it is in is the Near East. So combining the clues provided (Istanbul being part of the Near East), the answer would be the Near East. Alternatively, if they want the specific country now, like Turkey, but since the Ottoman Empire was not Turkey, but Turkey is the successor state. The question is a bit ambiguous. However, given the clues pointing towards Near East as the work location linked to Istanbul, I think the intended answer is the Near East region. So the final answer would be Near East. </ think > The Ottoman Lira was the currency of the Ottoman Empire, which was centered in the geographic region known historically as the Near East. This region included the empire ’ s heartland in Anatolia (modern - day Turkey), Istanbul (then Constantinople), parts of Southeast Europe, and the Middle East. The provided clues about Istanbul, the former capital, and its association with the Near East further reinforce this connection. \\ boxed {Near East}. < gold > [” Near East ”] </ gold > Figure 10: Example Instance (fs1). A successful reasoning trace for a question by QwQ-32B. Deepseek-R1 Example (fs1) ⬇ < question > What part of the world map is the Government of the Republic of Seychelles located in? While answering the question, make use of the following linearised graph as an inspiration in your reasoning, not as the only answer: Seychellois Creole, country, Seychelles Seychelles, continent, Africa Seychellois Creole, indigenous to, Seychelles Seychelles, continent, Africa. Put your final answer within \ boxed {}. </ question > < think > Okay, so I need to figure out where the Government of the Republic of Seychelles is located on the world map. Let me start by recalling what I know about Seychelles. I think Seychelles is a country made up of islands. But where exactly? The user provided a linearised graph that mentions Seychelles is in the continent of Africa. Hmm, that ’ s interesting because when I think of Africa, I usually picture the mainland, but there are also island countries associated with Africa. Let me verify that. Seychelles is an archipelago in the Indian Ocean, right? So it ’ s a group of islands northeast of Madagascar. Since Madagascar itself is off the southeastern coast of Africa, Seychelles being northeast of there would place it in the Indian Ocean, near the African continent. The graph also mentions Seychellois Creole being indigenous to Seychelles, which makes sense as the local language. So, putting this together, the Government of the Republic of Seychelles is located in the country of Seychelles, which is part of the African continent. Even though it ’ s an island nation, geographically and politically, it ’ s considered part of Africa. Therefore, the answer should be that the government is located in Seychelles, which is in Africa. </ think > The Government of the Republic of Seychelles is located in the country of Seychelles. According to the provided graph, Seychelles is part of the continent of Africa. Although geographically situated in the Indian Ocean northeast of Madagascar, it is politically and regionally classified as part of Africa. \\ boxed {Africa} < gold > [ ” Africa ”, ” Afrique ” ] </ gold > Figure 11: Example Instance (fs1). A successful reasoning trace for a question by Deepseek-R1. Appendix D Test Benchmark D.1 Examples Test Benchmark In Figure 12, we show for each dataset in the test benchmark an example of how the question and answers look like. Benchmark Test Examples ⬇ < cwq > Who was the president in 2012 of the country where Unidad de Valor Constante is the chosen currency? Answers: [ ” Rafael Correa ”, ” Rafael Correa Delgado ”, ” Rafael Vicente Correa Delgado ” ] </ cwq > < exaqt > what nba team won the back to back championship in 1989 and 1990? Answer: [ ” Detroit Pistons ” ] </ exaqt > < grailqa > lonnie wheeler contributed to a book edition published by what publisher? Answer: [ ” Simon & Schuster ” ] </ grailqa > < simpleqa > Who received the IEEE Frank Rosenblatt Award in 2010? Answer: [ ” Michio Sugeno ” ] </ simpleqa > < mintaka > How many books are in Goosebumps? Answer: [ ”235” ] </ mintaka > < webqsp > where did diego velazquez die? Answer: [ ” Madrid ” ] </ webqsp > Figure 12: Text Examples. For each dataset in the benchmark, we show an example. Appendix E SPARQL queries The query format to retrieve the Wikidata entities for Freebase entities is given by SELECT ?wikientity WHERE {?wikientity wdt:P646 $FREEBASE_ENTITY} The general structure of the SPARQL queries for 2-hop paths between the source and target entities are given by SELECT ?p1 ?p1Label ?o1 ?o1Label ?p2 ?p2Label WHERE {wd:$SOURCE_ENTITY ?p1 ?o1. ?o1 ?p2 wd:$TARGET_ENTITY} SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],mul,en". } SELECT ?o1 ?o1Label ?p1 ?p1Label ?p2 ?p2Label WHERE { ?p1 ?o1. ?o1 ?p2 wd:$TARGET_ENTITY} SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],mul,en". } Appendix F Absolute Performance Results for Analysis We show the absolute performance numbers from Figure 7 at pass@ $16$ in Figure 13. <details> <summary>x8.png Details</summary>  ### Visual Description ## Bar Chart: Absolute Performance by Hops ### Overview The image is a bar chart comparing the absolute performance of four different methods ("inst", "cot", "rt", and "fs1") across varying numbers of hops (1, 2, 3, and 3+). The y-axis represents "pass@16", a performance metric, ranging from 0.0 to 0.3. ### Components/Axes * **Title:** Absolute Performance by Hops * **Y-axis:** pass@16, ranging from 0.0 to 0.3 in increments of 0.1. * **X-axis:** Hops, with categories 1, 2, 3, and 3+. * **Legend:** Located in the top-left corner. * inst: Light gray bar * cot: Light blue bar * rt: Teal bar with diagonal stripes * fs1: Dark teal bar with diagonal stripes ### Detailed Analysis Here's a breakdown of the performance for each method across different hop counts: * **inst (Light Gray):** * Hop 1: Approximately 0.18 * Hop 2: Approximately 0.22 * Hop 3: Approximately 0.20 * Hop 3+: Approximately 0.18 * Trend: Increases from hop 1 to hop 2, then decreases slightly at hop 3, and returns to the original value at hop 3+. * **cot (Light Blue):** * Hop 1: Approximately 0.20 * Hop 2: Approximately 0.26 * Hop 3: Approximately 0.24 * Hop 3+: Approximately 0.22 * Trend: Increases from hop 1 to hop 2, then decreases at hop 3 and hop 3+. * **rt (Teal with Stripes):** * Hop 1: Approximately 0.23 * Hop 2: Approximately 0.31 * Hop 3: Approximately 0.30 * Hop 3+: Approximately 0.25 * Trend: Increases from hop 1 to hop 2, remains high at hop 3, then decreases at hop 3+. * **fs1 (Dark Teal with Stripes):** * Hop 1: Approximately 0.24 * Hop 2: Approximately 0.29 * Hop 3: Approximately 0.32 * Hop 3+: Approximately 0.26 * Trend: Increases from hop 1 to hop 3, then decreases at hop 3+. ### Key Observations * "fs1" generally shows the highest performance across all hop counts, peaking at hop 3. * "inst" consistently shows the lowest performance across all hop counts. * All methods show a performance increase from hop 1 to hop 2. * Performance tends to decrease slightly at hop 3+ compared to hop 2 or 3 for all methods. ### Interpretation The chart illustrates the performance of different methods in relation to the number of hops. The "fs1" method appears to be the most effective overall, while "inst" is the least. The performance of all methods seems to be affected by the number of hops, with a general trend of increasing performance up to a certain number of hops (2 or 3), followed by a slight decrease at "3+". This suggests that there might be an optimal number of hops for these methods, beyond which performance starts to diminish. The "pass@16" metric likely represents a measure of accuracy or success rate, and the chart provides a comparative analysis of how different approaches fare under varying conditions. </details> (a) Performance by number of hops required to answer the question measured in pass@ $16$ . <details> <summary>x9.png Details</summary>  ### Visual Description ## Bar Chart: Absolute Performance by Answer Type ### Overview The image is a bar chart comparing the absolute performance (pass@16) of four different methods ("inst", "cot", "rt", and "fs1") across five answer types ("date", "number", "other", "person", and "place"). The y-axis represents the "pass@16" score, ranging from 0.0 to 0.3. ### Components/Axes * **Title:** Absolute Performance by Answer Type * **X-axis:** Answer Type (categories: date, number, other, person, place) * **Y-axis:** pass@16 (scale: 0.0, 0.1, 0.2, 0.3) * **Legend:** Located in the top-right corner. * inst: Light gray bar * cot: Yellow bar * rt: Teal bar with diagonal stripes * fs1: Dark blue bar with diagonal stripes ### Detailed Analysis Here's a breakdown of the performance for each answer type and method: * **Date:** * inst: ~0.14 * cot: ~0.20 * rt: ~0.20 * fs1: ~0.21 * **Number:** * inst: ~0.23 * cot: ~0.26 * rt: ~0.29 * fs1: ~0.29 * **Other:** * inst: ~0.23 * cot: ~0.28 * rt: ~0.31 * fs1: ~0.31 * **Person:** * inst: ~0.12 * cot: ~0.14 * rt: ~0.14 * fs1: ~0.13 * **Place:** * inst: ~0.23 * cot: ~0.24 * rt: ~0.20 * fs1: ~0.20 ### Key Observations * The "other" answer type generally shows the highest performance across all methods. * The "person" answer type consistently shows the lowest performance across all methods. * "fs1" and "rt" methods tend to perform similarly and often outperform "inst" and "cot". * The performance difference between methods is most pronounced for the "number" and "other" answer types. ### Interpretation The chart illustrates the varying performance of different methods in answering questions based on the type of answer expected. The "other" category's high performance could indicate that the models are better at handling less specific or more varied types of information. Conversely, the lower performance on "person" questions might suggest difficulties in identifying and extracting information about individuals. The consistent outperformance of "fs1" and "rt" suggests that these methods are more effective overall in this context. The data suggests that the choice of method can significantly impact performance depending on the type of question being asked. </details> (b) Performance by answer type (i.e., what type of entity the answer is) in pass@ $16$ . <details> <summary>x10.png Details</summary>  ### Visual Description ## Bar Chart: Absolute Performance by Domain ### Overview The image is a bar chart comparing the absolute performance of four different methods ("inst", "cot", "rt", and "fs1") across ten different domains (art, geography, history, music, other, politics, sci & tech, sports, tv shows, and video games). The y-axis represents "pass@16", a performance metric, ranging from 0.0 to 0.3. ### Components/Axes * **Title:** Absolute Performance by Domain * **X-axis:** Domain (art, geography, history, music, other, politics, sci & tech, sports, tv shows, video games) * **Y-axis:** pass@16 (values: 0.0, 0.1, 0.2, 0.3) * **Legend:** Located in the top-right corner. * inst: Light gray bar * cot: Solid blue bar * rt: Dark purple bar with diagonal lines * fs1: Light red bar with diagonal lines ### Detailed Analysis Here's a breakdown of the performance of each method across the different domains: * **Art:** * inst: ~0.15 * cot: ~0.25 * rt: ~0.24 * fs1: ~0.26 * **Geography:** * inst: ~0.17 * cot: ~0.27 * rt: ~0.23 * fs1: ~0.24 * **History:** * inst: ~0.24 * cot: ~0.25 * rt: ~0.27 * fs1: ~0.29 * **Music:** * inst: ~0.21 * cot: ~0.23 * rt: ~0.23 * fs1: ~0.22 * **Other:** * inst: ~0.15 * cot: ~0.21 * rt: ~0.21 * fs1: ~0.22 * **Politics:** * inst: ~0.19 * cot: ~0.27 * rt: ~0.26 * fs1: ~0.28 * **Sci & Tech:** * inst: ~0.23 * cot: ~0.28 * rt: ~0.28 * fs1: ~0.28 * **Sports:** * inst: ~0.16 * cot: ~0.24 * rt: ~0.25 * fs1: ~0.24 * **TV Shows:** * inst: ~0.11 * cot: ~0.16 * rt: ~0.17 * fs1: ~0.16 * **Video Games:** * inst: ~0.14 * cot: ~0.16 * rt: ~0.16 * fs1: ~0.15 ### Key Observations * The "inst" method consistently underperforms compared to the other three methods across all domains. * "cot", "rt", and "fs1" show relatively similar performance, with "fs1" often having a slight edge. * The "history" and "sci & tech" domains generally have higher performance across all methods compared to "tv shows" and "video games". * The performance difference between the best and worst methods is most pronounced in the "tv shows" domain. ### Interpretation The chart illustrates the absolute performance of different methods across various domains. The "inst" method appears to be less effective than "cot", "rt", and "fs1" in all domains tested. The "history" and "sci & tech" domains seem to be easier or more suitable for these methods compared to "tv shows" and "video games". The consistent performance of "cot", "rt", and "fs1" suggests that these methods are more robust or adaptable across different domains. The data suggests that the choice of method can significantly impact performance, particularly in certain domains. </details> (c) Performance measured per domain in pass@ $16$ . Figure 13: Absolute Performance across Different Axes. We show the absolute performance at pass@ $16$ of different versions of Qwen-32B (i.e., instruct, CoT, rt and fs1). In (a), we show the performance of the models by answer type. In (b), we show the performance of the models by the number of hops required to answer the question. In (c), we show the performance by domain of the question.