2602.10480

# Neuro-Symbolic Synergy for Interactive World Modeling **Authors**: Hongyu Zhao, Siyu Zhou, Haolin Yang, Zengyi Qin, Tianyi Zhou ## Abstract Large language models (LLMs) exhibit strong general-purpose reasoning capabilities, yet they frequently hallucinate when used as world models (WMs), where strict compliance with deterministic transition rules—particularly in corner cases—is essential. In contrast, Symbolic WMs provide logical consistency but lack semantic expressivity. To bridge this gap, we propose Neuro-Symbolic Synergy (NeSyS), a framework that integrates the probabilistic semantic priors of LLMs with executable symbolic rules to achieve both expressivity and robustness. NeSyS alternates training between the two models using trajectories inadequately explained by the other. Unlike rule-based prompting, the symbolic WM directly constrains the LLM by modifying its output probability distribution. The neural WM is fine-tuned only on trajectories not covered by symbolic rules, reducing training data by 50% without loss of accuracy. Extensive experiments on three distinct interactive environments, i.e., ScienceWorld, Webshop, and Plancraft, demonstrate NeSyS ’s consistent advantages over baselines in both WM prediction accuracy and data efficiency. Our models and code are available at https://github.com/tianyi-lab/NeSyS. World modeling, Neurosymbolic, Language Model *[Error downloading image: 2602.10480v3/images/cover.png]* Figure 1: An example of the world modeling task. Given the current belief state and the agent’s next action, the world model needs to predict the next state. Both a neural world model (LLM) and a symbolic world model fail to answer the question by themselves. We propose to combine the two world models by viewing the symbolic scores as an energy term that modifies the probability distribution of the neural world model. ## 1 Introduction Large language models (LLMs) have recently emerged as strong general-purpose world models (WMs) for sequential decision-making problems (Yang et al., 2024c). With broad world knowledge and powerful general reasoning capabilities, they perform competitively across a wide range of tasks. However, LLMs often struggle to produce accurate predictions in domains with structured observations and well-defined dynamics, such as games and web environments (Levy et al., 2025; Chae et al., 2024). In particular, despite their strong performance in stochastic settings, they frequently fail in deterministic scenarios that require strict adherence to transition rules and constraints (Duan et al., 2024). Although extensive domain-specific fine-tuning can partially mitigate these issues, the inherently probabilistic nature of LLMs makes strict adherence to hard constraints difficult to guarantee. Moreover, their reliance on statistical learning limits their ability to reliably capture rare corner cases and long-tail behaviors. To address the shortcomings of WMs that rely solely on neural networks, a parallel line of work focuses on learning explicit symbolic WMs (Agravante et al., 2023; Liang et al., 2024). By enforcing hard rules and deterministic transition structures, such models naturally avoid constraint violations and long-horizon inconsistencies. However, their expressiveness is fundamentally limited: they struggle to generalize to real-world interactive environments whose dynamics are complex, high-dimensional, stochastic, or too underspecified to admit a tractable rule-based formulation. This limitation becomes particularly severe in natural-language-driven environments, where predicting the exact next state is often infeasible without leveraging the rich semantic priors embedded in large language models. These limitations suggest that neither purely neural WMs nor purely symbolic WMs are sufficient on their own, motivating the need for neuro-symbolic synergy (NeSyS) in world modeling. Figure 1 illustrates this motivation with an example in which NeSyS correctly answers a question that both approaches fail to resolve on their own. A variety of neuro-symbolic approaches have been explored. For instance, WALL-E (Zhou et al., 2024, 2025) injects symbolic rules into the decision process by prepending them to the LLM’s input context, thereby constraining next-state prediction implicitly. While effective when paired with large, instruction-tuned models, this strategy does not reliably transfer to smaller or more heavily fine-tuned models. In practice, such models frequently ignore or override injected instructions, limiting the robustness and scalability of context-based rule enforcement. In contrast, we propose to directly modify the probability distribution of the LLM with symbolic rules implemented by Python functions. This design yields two key advantages. First, the LLM is not required to follow additional instructions, eliminating any reliance on instruction-following quality or prompt sensitivity. Second, rules can be evaluated efficiently and locally (without additional inference on LLMs) by inspecting the probabilities of a small set of candidate answers annotated with ground-truth correctness, enabling rapid feedback and making rule learning substantially more sample-efficient than injecting rules into LLM context. Furthermore, we observe that extensive fine-tuning of the neural WM is often unnecessary, as many failure cases can be resolved through simple and generalizable rules. Based on this insight, we introduce rule-guided data selection, which filters out training examples whose dynamics are already captured by learned rules, thereby reducing training redundancy. Empirically, it halves the training data while maintaining comparable performance, highlighting the practical efficiency of explicit neuro-symbolic synergy. We examine the world modeling capabilities by a suite of benchmarks spanning three diverse and widely-studied environments: ScienceWorld (Wang et al., 2022), Webshop (Yao et al., 2022) and Plancraft (Dagan et al., 2024). They capture challenges in common-sense physical reasoning, real-world web interaction, and structured game dynamics, respectively. Across all three, we demonstrate that NeSyS consistently outperforms baselines on different choices of backbone LLMs, highlighting both its effectiveness and robustness. In summary, this work makes the following contributions: - We introduce Neuro-Symbolic Synergy (NeSyS), a framework that integrates LLM-based world models with symbolic rules by directly enforcing hard constraints through probability-level modification of LLM outputs. - We introduce a complementary training paradigm for neural and symbolic world models, where each model is trained only on data regimes not handled by the other, substantially reducing finetuning data requirements without sacrificing performance. - We empirically demonstrate that NeSyS achieves consistent performance improvements across diverse interactive environments and model scales. ## 2 Related Works Neuro-symbolic world models. To mitigate inherent limitations of pure neural models—such as hallucination—neuro-symbolic world models have emerged as a promising paradigm (Cano et al., 2025; Sehgal et al., 2023; Balloch et al., 2023). In complex, text-rich environments, existing methods often employ semantic parsers (Agravante et al., 2023) or VLMs (Liang et al., 2024) to translate environment dynamics into symbolic rules. To better synergize neural and symbolic components, recent works have explored enhancing LLM prompts with symbolic constraints: De Giorgis et al. (2025) utilize knowledge graphs to prompt the model with structured knowledge, while WALL-E (Zhou et al., 2024, 2025) prepends symbolic rules to the LLM’s context. Nevertheless, such prompt engineering approaches rely heavily on the model’s instruction-following capabilities. To address this reliance, our work proposes to directly modify the LLM’s output probability distribution using executable Python rules. Training data selection. Data selection is critical for optimizing the performance of fine-tuned LLMs while minimizing computational costs (Zhou et al., 2023). While recent literature has explored rule-based methods to identify high-quality training subsets (Li et al., 2024), our approach diverges in its objective. Instead of filtering for general data quality, we aim to maximize the complementary strengths of the neural model and the symbolic components by selecting instances that are difficult for symbolic rules to solve. Constrained decoding and logit manipulation. Constrained decoding is a well-established technique for forcing LLM outputs to adhere to specific formats (Willard & Louf, 2023; Beurer-Kellner et al., 2024). While traditional constrained decoding focuses on syntactic correctness (e.g., generating valid JSON), recent works have begun exploring semantic and logical constraints (Wu et al., 2024; Ma & Hu, 2025). NeSyS extends this line of research by using symbolic evaluations as an energy-based shifting factor in the probability landscape. ## 3 Problem Setting We study sequential decision-making problems in a partially observable Markov decision process (POMDP). At step $t$ , the agent needs to do action $a_{t}$ based on the current belief state $b_{t}$ . It will then receive the next state $s_{t+1}$ and a reward $r_{t}$ from the environment. We do not distinguish observation and state here. See Appendix F for a rigorous definition. A world model is defined as a predictive model that estimates the next state $s_{t+1}$ and reward $r_{t}$ conditioned on the current belief state $b_{t}$ and action $a_{t}$ . Following prior LLM-based agent work, we approximate the belief state by a textual context constructed from the task description $g$ and a truncated history of recent observations, actions, and rewards that fit within the model’s context window. ## 4 Method Our framework consists of two world models (WM): Neural WM and Symbolic WM. Neural WM is in the form of an LLM, while Symbolic WM is a weighted set of Python rules. ### 4.1 Framework Overview *[Error downloading image: 2602.10480v3/images/inference.png]* Figure 2: Overview of NeSyS. It consists of two world models: Neural WM and Symbolic WM. They are implemented as an LLM $\theta$ with likelihood function $P_{\theta}$ and a weighted rule set $\mathcal{F}$ , respectively. Neural WM generates $K$ candidates of the next state and reward pairs if there is no provided choices. The likelihood $p_{i}$ for each candidate is computed. Symbolic WM aggregates the score $e_{ij}$ produced by each rule $f_{j}$ . We then modify the likelihood $p_{i}$ with the score by Symbolic WM, and choose the candidate with the largest modified likelihood $\tilde{p}_{i}$ . For simplicity, the conditional $b_{t}$ and $a_{t}$ are omitted from the parameters of $P_{\theta}$ and $f_{j}$ . As mentioned in Section 1, the main idea of our strategy is to directly modify the probability distribution of Neural WM, i.e., LLM, with the symbolic rules learned by Symbolic WM. Given current belief state $b_{t}$ and action $a_{t}$ , we aim to select the optimal next-step state and reward. The whole inference pipeline is illustrated in Figure 2. On the left side, Neural WM computes the likelihood ${p_{i}}$ for candidate next state $s_{t+1}^{i}$ and reward $r_{t}^{i}$ . The $K$ candidates are either provided or generated by Neural WM itself. On the right side, Symbolic WM is implemented as a weighted set of executable Python functions $\mathcal{F}=\{(f_{j},w_{j})\}_{j=1}^{m}$ , where $f_{j}\!:\!(b_{t},a_{t},s_{t+1},r_{t})\mapsto[-1,1]$ . Each rule produces a score $e_{ij}\in[-1,1]$ indicating how likely the candidate $(s_{t+1}^{i},r_{t}^{i})$ is correct under this specific rule. Rules are typically inactive and output zero, but become active only when specific conditions are met (e.g., a particular action is taken or a relevant keyword appears in the belief state). We aggregate these logical judgments into a scalar shifting factor $E_{i}$ using learned rule weights $w_{j}$ : $$ E_{i}=\sum_{j=1}^{m}w_{j}e_{ij} $$ We combine these signals by treating the symbolic score as an energy term that modifies the neural likelihood $p_{i}$ . The modified likelihood $\tilde{p}_{i}$ is computed as $$ \tilde{p}_{i}=p_{i}\exp(\gamma E_{i}), $$ where the scaling factor $\gamma$ is a global hyperparameter. In our experiments, it’s set as either $1$ (no scaling) or the largest gap of log likelihood between candidates. We then select the candidate with the largest $\tilde{p}_{i}$ . This formulation allows Symbolic WM to enforce hard constraints via a negative $E_{i}$ or boost logical consistencies via a positive $E_{i}$ , effectively reshaping the probability landscape of the LLM. ### 4.2 Training Pipeline *[Error downloading image: 2602.10480v3/images/training_pipeline.png]* Figure 3: Training pipeline of NeSyS. It consists of two phases. In Phase 1 (Initialization), we initialize Neural WM with a pretrained LLM. We evaluate it on the development set to separate common sense from task-specific knowledge, generating rules for the latter to initialize Symbolic WM. In Phase 2 (Reciprocal Refinement), we use Symbolic WM to perform rule-guided data selection on the training set by filtering out simple cases. The remaining “hard” data are used to fine-tune Neural WM. Symbolic WM is then refined by addressing long-tailed cases where the updated Neural WM still fails. Legend is in the lower left corner. Weight optimization for Symbolic WM is omitted for clarity. As illustrated in Figure 3, our training pipeline consists of two phases: Initialization and Reciprocal Refinement. The pipeline iteratively filters out training / development samples that can be covered by the current framework, and improve the two WMs based on the errors of each other. This design ensures that the two WMs evolve complementarily rather than redundantly. #### 4.2.1 Phase 1: Initialization. Neural WM. We initialize Neural WM using a pretrained LLM. By evaluating this model on the development set, we can separate questions that can be solved with general common sense knowledge (correct samples) from those that require task-specific knowledge (incorrect samples). Symbolic WM. We aim to capture task-specific deterministic dynamics that Neural WM misses via explicit rules. The rules are induced via an automated debugging loop: 1. Error clustering: We identify mistakes made by the initialized Neural WM on a development set. These are clustered by syntactic and semantic similarity (details in Appendix C) to identify systematic failure modes. 1. Rule generation: For each cluster, we prompt gpt-5-mini to write a Python function that corrects the specific error while generalizing to similar cases. 1. Verification & weighting: A rule is added to $\mathcal{F}$ only if the accuracy of NeSyS on the development set is improved with the update $\mathcal{F}$ . We perform up to 3 reflection iterations with gpt-5-mini for failed rules. We initialize the scalar weights uniformly as $w_{j}=1$ . #### 4.2.2 Phase 2: Reciprocal Refinement. Neural WM. A key insight of our work is that fine-tuning a model on behaviors already captured by explicit rules is computationally wasteful and potentially redundant. To address this, we introduce rule-guided data selection. We evaluate the training trajectories using our initialized rule set $\mathcal{F}$ . For each step, we compute $k$ to be the number of rules in $\mathcal{F}$ that reports a non-zero score on it. The steps with smaller $k$ is harder to predict by the Symbolic WM alone and vice versa. We first collect all the steps with $k=0$ and sample from the remaining steps with probability proportional to $\frac{1}{k}$ so that harder steps are sampled more frequently. Neural WM is then fine-tuned only on the sampled examples. This focuses the Neural WM’s capacity on capturing complex, intuitive dynamics that are difficult to formalize as code. In practice, we find this strategy can discard about half of training data while maintaining or improving performance. Symbolic WM. Following refinement of Neural WM, rules that were useful for a weaker Neural WM might now negatively impact the performance. We first apply a cleaning pass: any rule that has a negative impact on the performance of Neural WM is discarded. We then add more rules by repeating the clustering and synthesis process described in Phase 1 to cover any new residual errors. Finally, in order to enhance the performance of NeSyS, we learn scalar weights $w_{j}$ via coordinate descent on the development set. Table 1: World modeling accuracy on ScienceWorld. Bold numbers highlight the best performance for each backbone model. | World Model Baseline Models Llama3.1-8B | Matter 36.1 | Chem. 29.4 | Class. 68.8 | Biology 42.7 | Forces 38.9 | Meas. 52.5 | Elec. 33.9 | Avg. 44.4 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-8B | 38.9 | 31.4 | 65.0 | 43.3 | 66.7 | 45.8 | 40.7 | 46.5 | | Qwen3-14B | 45.8 | 43.1 | 68.8 | 46.5 | 64.9 | 47.5 | 49.2 | 51.2 | | Phi-4-mini | 38.9 | 39.2 | 63.8 | 50.3 | 50.0 | 40.7 | 33.9 | 46.7 | | GPT-oss-20B | 34.7 | 49.0 | 48.8 | 47.7 | 40.6 | 33.9 | 25.4 | 41.2 | | GPT-5-mini | 45.8 | 51.0 | 72.5 | 52.2 | 72.2 | 50.8 | 50.8 | 55.4 | | Llama3.2-1B | | | | | | | | | | SFT (100% data) | 37.5 | 64.7 | 81.3 | 68.8 | 80.6 | 54.2 | 62.7 | 64.4 | | Neural WM (Phase 1) | 27.8 | 17.7 | 42.5 | 39.5 | 44.4 | 27.1 | 15.3 | 32.3 | | Symbolic WM (Phase 1) | 33.3 | 39.2 | 58.8 | 56.7 | 41.7 | 30.5 | 47.5 | 46.9 | | NeSyS (Phase 1) | 43.1 | 27.5 | 61.3 | 58.6 | 50.0 | 42.4 | 37.3 | 48.8 | | Neural WM (Phase 2) | 37.5 | 66.7 | 71.3 | 70.1 | 77.8 | 47.5 | 55.9 | 61.7 | | Symbolic WM (Phase 2) | 45.8 | 41.2 | 78.8 | 56.7 | 41.7 | 45.8 | 55.9 | 54.7 | | NeSyS (ours, 45% data The percentage of filtered data depends on rule coverage, which varies by benchmark.) | 47.2 | 64.7 | 88.8 | 73.3 | 91.7 | 50.9 | 59.3 | 68.3 | | Qwen3-4B | | | | | | | | | | SFT (100% data) | 48.6 | 66.7 | 91.3 | 68.8 | 100.0 | 59.3 | 50.8 | 68.3 | | Neural WM (Phase 1) | 38.9 | 35.3 | 63.8 | 43.3 | 63.9 | 42.4 | 47.5 | 46.9 | | Symbolic WM (Phase 1) | 38.9 | 47.1 | 52.5 | 61.2 | 41.7 | 33.9 | 54.2 | 50.0 | | NeSyS (Phase 1) | 52.8 | 43.1 | 68.8 | 62.4 | 63.9 | 55.9 | 54.2 | 58.6 | | Neural WM (Phase 2) | 50.0 | 58.8 | 91.3 | 66.9 | 94.4 | 59.3 | 66.1 | 68.5 | | Symbolic WM (Phase 2) | 43.1 | 58.8 | 68.8 | 62.4 | 75.0 | 44.1 | 55.9 | 58.4 | | NeSyS (ours, 45% data) | 50.0 | 60.8 | 88.8 | 75.8 | 94.4 | 59.3 | 66.1 | 71.0 | ## 5 Experiments ### 5.1 Experiment Setup Environments. We evaluate our methods on three diverse environments: ScienceWorld (Wang et al., 2022), Webshop (Yao et al., 2022) and Plancraft (Dagan et al., 2024), which respectively capture challenges in common-sense physical reasoning, real-world web interaction, and structured game dynamics. Based on these environments, we create multiple-choice questions for both developing and testing. Details can be found in Appendix E. Models and baselines. We evaluate the performance of Llama3.2-1B instruct version (Grattafiori et al., 2024) and Qwen3-4B (Yang et al., 2024a, b) as the backbone of the neural world model. The rules for the symbolic part are generated by gpt-5-mini (Singh et al., 2025). We compare against multiple strong open-source models along with proprietary models, including Llama3.1-8B instruct version, Qwen3-8B/14B, Phi-4-mini (Abouelenin et al., 2025), GPT-oss-20B (Agarwal et al., 2025) and GPT-5-mini. We also consider the natural baseline that fine-tunes the backbone LLM on the full training data. Additionally, we provide a detailed ablation analysis covering performance of each single world model and NeSyS across both Phase 1 and Phase 2. For fair comparison, we also learn a weight for NeSyS in Phase 1. ### 5.2 ScienceWorld To access the common-sense reasoning abilities of world models, we evaluate our methods on ScienceWorld, where the models need to navigate through a series of rooms and perform tasks aligned with a standardized elementary science curriculum. The tasks are divided into 7 different topics, including matter, chemistry, classification, biology, forces, measurement and electricity. We report accuracy on each topic along with the average accuracy in Table 1. Table 2: World modeling accuracy on Webshop. Bold numbers highlight the best performance for each backbone model. | World Model Baseline Models Llama3.1-8B | Search 0.0 | Layout 2.9 | Detail 10.2 | Decision 50.0 | Page Change 2.7 | Other 29.1 | Avg. 28.9 | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-8B | 0.0 | 0.0 | 14.3 | 39.5 | 2.7 | 32.6 | 25.1 | | Qwen3-14B | 0.0 | 2.9 | 12.2 | 46.2 | 2.7 | 31.9 | 27.9 | | Phi-4-mini | 0.0 | 0.0 | 4.1 | 50.0 | 2.7 | 11.3 | 24.9 | | GPT-oss-20B | 0.0 | 0.0 | 16.3 | 50.0 | 5.4 | 0.7 | 24.0 | | GPT-5-mini | 89.4 | 94.1 | 81.6 | 66.9 | 94.6 | 96.5 | 81.4 | | Llama3.2-1B | | | | | | | | | SFT (100% data) | 0.0 | 58.8 | 20.4 | 50.0 | 10.7 | 100.0 | 47.5 | | Neural WM (Phase 1) | 0.0 | 0.0 | 0.0 | 50.0 | 0.9 | 0.7 | 22.2 | | Symbolic WM (Phase 1) | 100.0 | 23.5 | 81.6 | 83.8 | 75.0 | 96.5 | 83.4 | | NeSyS (Phase 1) | 98.5 | 2.9 | 73.5 | 83.8 | 71.4 | 95.0 | 80.9 | | Neural WM (Phase 2) | 0.0 | 52.9 | 19.2 | 50.0 | 12.4 | 100.0 | 45.9 | | Symbolic WM (Phase 2) | 100.0 | 88.2 | 98.0 | 83.8 | 100.0 | 96.5 | 91.5 | | NeSyS (ours, 60% data) | 100.0 | 91.2 | 95.9 | 83.8 | 100.0 | 100.0 | 92.2 | | Qwen3-4B | | | | | | | | | SFT (100% data) | 0.0 | 64.7 | 18.4 | 52.5 | 11.6 | 92.2 | 47.3 | | Neural WM (Phase 1) | 0.0 | 0.0 | 16.3 | 50.0 | 3.6 | 27.0 | 28.9 | | Symbolic WM (Phase 1) | 100.0 | 23.5 | 73.5 | 66.6 | 33.0 | 97.2 | 68.9 | | NeSyS (Phase 1) | 97.0 | 2.9 | 65.3 | 83.8 | 29.5 | 99.3 | 74.4 | | Neural WM (Phase 2) | 0.0 | 64.7 | 19.2 | 51.0 | 13.5 | 93.1 | 46.1 | | Symbolic WM (Phase 2) | 100.0 | 100.0 | 100.0 | 83.8 | 95.5 | 92.9 | 90.8 | | NeSyS (ours, 60% data) | 100.0 | 100.0 | 95.9 | 83.8 | 100.0 | 100.0 | 92.6 | Our framework improves the performance of full fine-tuning with fewer training samples. For the Llama3.2-1B-instruct backbone, NeSyS achieves an average accuracy of 68.3% using only 45% of the training data, surpassing the standard SFT baseline (64.4%) which utilizes the full dataset. Similarly, for the larger Qwen3-4B model, NeSyS (71.0%) also outperforms the SFT baseline (68.3%). This validates our hypothesis that a significant portion of interactive trajectory data is redundant once the underlying deterministic rules are extracted. In other words, a large part of training data can be replaced by symbolic rules. Each WM benefits from the reciprocal refinement phase. Comparing the results from Phase 1 to Phase 2 validates the effectiveness of our iterative training pipeline. For both backbone models, the performance of each single WM significantly improves during the reciprocal refinement phase. While the improvement of Neural WM is expected, the improvement of Symbolic WM is not trivial. We find marginal improvement when iterating the rule set over the same error clusters again and again. It indicates that as Neural WM improves, it exposes new, more subtle failure modes that Symbolic WM effectively captures in the second iteration. This simultaneous growth demonstrates that our reciprocal refinement prevents the modules from becoming redundant, instead pushing each to address the other’s remaining blind spots. Each part of our framework contributes to the final performance. Even after the individual components are refined, their combination yields performance superior to either in isolation. For both backbones, the final performance of NeSyS surpasses the one of each single WM in each phase. This additive gain confirms that the two modules remain fundamentally complementary: Symbolic WM secures the deterministic constraints while Neural WM captures the semantic information, and their integration covers the intersection more robustly than any single modality. Table 3: World modeling accuracy on Plancraft. Bold numbers highlight the best performance for each backbone model. | World Model Baseline Models Llama3.1-8B | Smelt 90.6 | Move-Easy 91.2 | Move-Medium 69.5 | Move-Hard 36.5 | Average 72.6 | | --- | --- | --- | --- | --- | --- | | Qwen3-8B | 48.4 | 90.0 | 68.4 | 33.9 | 68.0 | | Qwen3-14B | 56.3 | 91.5 | 64.4 | 35.7 | 68.9 | | Phi-4-mini | 75.0 | 90.5 | 72.4 | 33.0 | 70.8 | | GPT-oss-20B | 62.5 | 81.3 | 70.7 | 27.4 | 63.7 | | GPT-5-mini | 93.8 | 91.5 | 60.3 | 47.0 | 73.8 | | Llama3.2-1B | | | | | | | SFT (100% data) | 31.3 | 98.8 | 73.6 | 67.0 | 80.5 | | Neural WM (Phase 1) | 54.7 | 88.1 | 69.5 | 28.7 | 66.4 | | Symbolic WM (Phase 1) | 45.3 | 93.4 | 47.7 | 48.7 | 69.2 | | NeSyS (Phase 1) | 71.9 | 83.9 | 92.5 | 69.6 | 81.0 | | Neural WM (Phase 2) | 34.4 | 97.8 | 75.3 | 47.0 | 75.4 | | Symbolic WM (Phase 2) | 81.3 | 89.1 | 87.9 | 62.2 | 81.2 | | NeSyS (ours, 35% data) | 98.4 | 95.1 | 93.7 | 67.0 | 87.7 | | Qwen3-4B | | | | | | | SFT (100% data) | 87.5 | 99.8 | 94.8 | 70.0 | 90.1 | | Neural WM (Phase 1) | 57.8 | 89.3 | 69.5 | 31.3 | 67.9 | | Symbolic WM (Phase 1) | 85.9 | 79.3 | 82.2 | 66.1 | 76.9 | | NeSyS (Phase 1) | 93.8 | 83.0 | 92.5 | 67.4 | 81.6 | | Neural WM (Phase 2) | 87.5 | 100.0 | 93.1 | 51.7 | 85.1 | | Symbolic WM (Phase 2) | 81.3 | 89.8 | 86.8 | 45.7 | 77.0 | | NeSyS (ours, 35% data) | 92.2 | 97.1 | 96.6 | 65.6 | 88.4 | ### 5.3 Webshop We further evaluate the methods on Webshop, which is an e-commerce website environment that requires the agent to navigate webpages, search and purchase items that satisfy particular requirements. It demands a distinct combination of capabilities: the rigid adherence to web navigation protocols and the semantic understanding required to predict and interpret product details. We divide the tasks into 6 types by actions: - Search (search[xxx]), where the model needs to predict the possible search result. - Layout (click[item - xxx]), where the model needs to predict the layout of the item page. - Detail (click[description/features/reviews]), where the model needs to predict the detailed descriptions of the product or the detailed user feedback of the product. These buttons are standard for all product pages, and the result format is fixed. - Decision (click[buy now]), where the model needs to decide whether we successfully bought the product that meets the description. All the questions in this category have only 2 choices: Success and Fail. - Page Change (click[¡ prev/next ¿/back to search]), where the model needs to understand the meaning of these button and the format of the target pages. - Other, corresponding to all other actions, mainly include choosing the size/specification of the products. Table 2 summarizes the result on this task. Symbolic rules can perform much better than LLMs on specific tasks. The “Search” and “Decision” tasks require exact string matching and logic verification that probabilistic models struggle to approximate. As a result, aside from gpt-5-mini, all neural models get zero accuracy on “Search” and performs random guess on “Decision”. In contrast, Symbolic WM achieves 100% and ¿66% respectively on these tasks, even in Phase 1. This provides the clearest evidence for the necessity of neuro-symbolic synergy: simple Python rules can solve deterministic structural constraints that completely baffle fully fine-tuned LLMs. Synergy still exists even for rule-solvable tasks. While “Layout” is a task that could be completely resolved by rules (Phase 2 Symbolic WM achieves 100% for Qwen), it is very difficult to learn rules from the whole dataset (Phase 1 Symbolic WM only gets 23.5%). However, by learning from the mistakes of a fine-tuned Neural WM, the Phase 2 Symbolic WM improves significantly for both Llama and Qwen. In other words, Symbolic WM learns more from the data that Neural WM struggles to learn, verifying our hypothesis of neural-symbolic synergy. Our framework can benefit from a weak Neural WM. We observe a notable anomaly in the Llama Phase 1 results where NeSyS (80.9%) slightly underperforms the standalone Symbolic WM (83.4%). This degradation stems from the extremely poor quality of the initialized Neural WM (22.2% average, with 0.0% on structural tasks), which effectively acts as a noise generator that dilutes the precise signals from the symbolic rules during probability re-ranking. However, this sensitivity is limited to extreme cases. As long as the neural model achieves a baseline level of competency—as seen with the stronger Qwen Phase 1 model or the fine-tuned Llama Phase 2 model—the synergy is restored. For instance, once the Llama Neural WM improves to 45.9% in Phase 2, the combined framework (92.2%) successfully overtakes Symbolic WM (91.5%), demonstrating that NeSyS can leverage even a moderately capable neural model to achieve state-of-the-art results. ### 5.4 Plancraft Finally, we evaluate our approach on Plancraft, a Minecraft-based environment that constrains interaction to the Minecraft crafting GUI, requiring agents to use items in the backpack to craft certain items. Unlike other 2 environments, Plancraft requires world models to possess game-specific knowledge, i.e., the crafting recipes (or infer them from common sense), and strictly adhere to the rules of the game. The tasks are categorized into Smelt, involving furnace interactions, and Move tasks, which require the model to predict the result of moving certain inventory items. We further divide the move tasks into 3 difficulty levels: - Easy, including simple moves with no side effect. - Medium, corresponding to moves that generate the product required to reach the goal (not necessarily the final product). This is easier than hard as the goal is part of the belief state. - Hard, including moves that generate a side product. Table 3 presents the results. Training instability highlights the necessity of hybrid modeling. A critical observation in Plancraft is the vulnerability of neural fine-tuning to catastrophic forgetting. For the Llama backbone, any form of training causes a significant performance drop in the Smelt task: the Neural WM (Phase 1) model starts at 54.7%, but SFT (100% data) plummets to 31.3%. We hypothesize that the distribution of crafting tasks in the training set may dominate the gradient updates, causing the model to overwrite its understanding of furnace logic. In stark contrast, NeSyS achieves the highest performance on this task (98.4%), as the symbolic module is immune to distribution shift. Similarly, for Qwen, NeSyS also improves “Smelt” performance to 92.2%, surpassing the SFT baseline of 87.5%. Medium-level tasks benefit most from neuro-symbolic synergy. The “Move-Medium” tasks represent the sweet spot for our approach, requiring a balance of recipe knowledge (symbolic) and goal-oriented planning (neural). Here, NeSyS achieves a remarkable 96.0% accuracy for Llama, vastly outperforming both Neural WM (77.0%) and Symbolic WM (73.0%) in isolation. In contrast, “Move-Easy” tasks are trivial enough for the LLM to solve alone (98.8% SFT accuracy), while “Move-Hard” tasks involve complex side-effects that defy simple rule encoding. The substantial margin in the medium difficulty tasks suggests that our framework delivers the highest value when the problem complexity sits specifically at the intersection of rigid logic and flexible planning, where neither the neural nor the symbolic expert is sufficient on its own. ### 5.5 Analysis Ablation studies on data selection methods. We compare the rule-guided data selection method we used in the main experiments against a random selection baseline. Figure 4 shows that Neural WMs trained with two data selection methods perform similar without symbolic rules, but rule-guided method consistently perform better when paired with Symbolic WM. Additionally, we observe that the performance of Neural WM keeps improving until using all the training samples, but this is not true for NeSyS. These two observations further prove the complementary strengths of Neural and Symbolic WM in our framework, and the effectiveness of the proposed rule-guided data selection mechanism. *[Error downloading image: 2602.10480v3/images/random_vs_rule_filtering.png]* Figure 4: Comparing rule-guided data selection (ours) vs. random data selection under different training budgets on Plancraft. The backbone model is Llama-3.2-1B-Instruct. The shadow in the figure highlights the performance gain of our strategy. Exploration of training a lightweight router. Building on the neuro-symbolic synergy observed in our main experiments, we explored the potential of a routing mechanism to dynamically select the optimal configuration (i.e., Neural vs. Symbolic WM, phase 1 vs. phase 2 models, and the scaling factor $\gamma$ ). We compared a lightweight XGBoost router, trained on the development set, against an ’oracle’ router that acts as a theoretical upper bound by always selecting the optimal parameters. As detailed in Appendix D, the XGBoost router yielded marginal gains over the baseline NeSyS. However, the oracle router achieved near-perfect scores, indicating that while simple routing is insufficient, developing a more sophisticated selection mechanism remains a high-potential direction for future work. ## 6 Conclusion In this work, we introduced NeSyS, a novel framework that bridges the gap between the broad semantic reasoning of LLMs and the strict logical consistency required for rigorous world modeling. By treating executable symbolic rules as energy functions that directly modify the LLM’s output distribution, our approach enforces deterministic constraints without relying on the model’s variable instruction-following capabilities. Empirical results validate NeSyS ’ consistent effectiveness and efficiency across diverse domains and model scales. While our current implementation effectively combines neural and symbolic signals, our analysis suggests that more sophisticated, dynamic routing between these modules represents a high-potential direction for future work. ## Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. ## References - Abouelenin et al. (2025) Abouelenin, A., Ashfaq, A., Atkinson, A., Awadalla, H., Bach, N., Bao, J., Benhaim, A., Cai, M., Chaudhary, V., Chen, C., et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. - Agarwal et al. (2025) Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. - Agravante et al. (2023) Agravante, D. J., Kimura, D., Tatsubori, M., Munawar, A., and Gray, A. Learning neuro-symbolic world models with conversational proprioception. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 648–656, 2023. - Balloch et al. (2023) Balloch, J., Lin, Z., Wright, R., Peng, X., Hussain, M., Srinivas, A., Kim, J., and Riedl, M. O. Neuro-symbolic world models for adapting to open world novelty. arXiv preprint arXiv:2301.06294, 2023. - Beurer-Kellner et al. (2024) Beurer-Kellner, L., Fischer, M., and Vechev, M. Guiding llms the right way: Fast, non-invasive constrained generation. arXiv preprint arXiv:2403.06988, 2024. - Cano et al. (2025) Cano, L. H., Perroni-Scharf, M., Dhir, N., Ramamurthy, A., and Solar-Lezama, A. Neurosymbolic world models for sequential decision making. In Forty-second International Conference on Machine Learning, 2025. - Chae et al. (2024) Chae, H., Kim, N., Ong, K. T.-i., Gwak, M., Song, G., Kim, J., Kim, S., Lee, D., and Yeo, J. Web agents with world models: Learning and leveraging environment dynamics in web navigation. arXiv preprint arXiv:2410.13232, 2024. - Dagan et al. (2024) Dagan, G., Keller, F., and Lascarides, A. Plancraft: an evaluation dataset for planning with llm agents. arXiv preprint arXiv:2412.21033, 2024. - De Giorgis et al. (2025) De Giorgis, S., Gangemi, A., and Russo, A. Neurosymbolic graph enrichment for grounded world models. Information Processing & Management, 62(4):104127, 2025. - Duan et al. (2024) Duan, J., Zhang, R., Diffenderfer, J., Kailkhura, B., Sun, L., Stengel-Eskin, E., Bansal, M., Chen, T., and Xu, K. Gtbench: Uncovering the strategic reasoning capabilities of llms via game-theoretic evaluations. Advances in Neural Information Processing Systems, 37:28219–28253, 2024. - Grattafiori et al. (2024) Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint, 2024. URL https://arxiv.org/abs/2407.21783. - Levy et al. (2025) Levy, G., Colas, C., Oudeyer, P.-Y., Carta, T., and Romac, C. Worldllm: Improving llms’ world modeling using curiosity-driven theory-making. arXiv preprint arXiv:2506.06725, 2025. - Li et al. (2024) Li, X., Gao, M., Zhang, Z., Yue, C., and Hu, H. Rule-based data selection for large language models. arXiv preprint arXiv:2410.04715, 2024. - Liang et al. (2024) Liang, Y., Kumar, N., Tang, H., Weller, A., Tenenbaum, J. B., Silver, T., Henriques, J. F., and Ellis, K. Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning. arXiv preprint arXiv:2410.23156, 2024. - Ma & Hu (2025) Ma, F. and Hu, A. J. Logically constrained decoding. In Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025), pp. 150–167, 2025. - Reimers & Gurevych (2019) Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084. - Sehgal et al. (2023) Sehgal, A., Grayeli, A., Sun, J. J., and Chaudhuri, S. Neurosymbolic grounding for compositional world models. arXiv preprint arXiv:2310.12690, 2023. - Singh et al. (2025) Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. - Wang et al. (2022) Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P. Scienceworld: Is your agent smarter than a 5th grader?, 2022. URL https://arxiv.org/abs/2203.07540. - Willard & Louf (2023) Willard, B. T. and Louf, R. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702, 2023. - Wu et al. (2024) Wu, T.-H., Lian, L., Gonzalez, J. E., Li, B., and Darrell, T. Self-correcting llm-controlled diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6327–6336, 2024. - Yang et al. (2024a) Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv preprint, 2024a. URL https://arxiv.org/abs/2407.10671. - Yang et al. (2024b) Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint, 2024b. URL https://arxiv.org/abs/2412.15115. - Yang et al. (2024c) Yang, C., Wang, X., Jiang, J., Zhang, Q., and Huang, X. Evaluating world models with llm for decision making. arXiv preprint arXiv:2411.08794, 2024c. - Yao et al. (2022) Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022. - Zhou et al. (2023) Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023. - Zhou et al. (2024) Zhou, S., Zhou, T., Yang, Y., Long, G., Ye, D., Jiang, J., and Zhang, C. Wall-e: World alignment by rule learning improves world model-based llm agents. arXiv preprint arXiv:2410.07484, 2024. - Zhou et al. (2025) Zhou, S., Zhou, T., Yang, Y., Long, G., Ye, D., Jiang, J., and Zhang, C. Wall-e 2.0: World alignment by neurosymbolic learning improves world model-based llm agents. arXiv preprint arXiv:2504.15785, 2025. ## Appendix A Evaluation Prompt When evaluating on ScienceWorld, we include recent trajectories and a memory of each visited location before the beginning of the trajectories, and ask for predicting the next observation, reward and difference of inventory. Here is the prompt we used: Prompt used for Evaluating on ScienceWorld System: You are a ScienceWorld transition model. Given the compressed history and ONE action, output exactly three lines in this order: predicted_observation: ¡text¿ predicted_reward: ¡float in [-1,1]¿ predicted_inventory_diff: ¡one or more lines with ’+ ’ or ’- ’ prefixes only¿ Do not output any additional text. User: Task Description: {task description} Memory (last look-around per visited location): {memory} action: { $a_{k}$ } observation: { $s_{k+1}$ } reward: { $r_{k}$ } … current_step_action: { $a_{t}$ } Here is the prompt we used for evaluating on Webshop: Prompt used for Evaluating on Webshop System: You are a transition model for the WebShop text environment. Given the current page text (state) and the action taken, predict the next page text. Your answer must be ONLY the next state’s text, with no extra commentary. User: Current state (page text): {state} Action taken: { $a_{t}$ } Question: What is the next state’s page text? Answer with ONLY the next state’s text. Here is the prompt we used for evaluating on Plancraft: Prompt used for Evaluating on Plancraft System: You are a Minecraft transition model. Given a current state and ONE action, return the next state text in the exact format used by the data (same headers and lists). Do not output actions. Enforce environment rules: cannot move/smelt into [0]; crafting outputs appear at [0] and must be moved to an inventory slot to complete. Output only the next state text. User: {state} Action: { $a_{t}$ } ## Appendix B Rule induction Prompts Here are the prompt we used for rule inductions for each environment. Example programs are omitted for clarity, they are essentially python programs that implement the example rules and can be found in our published code. Rule induction for Webshop User: Analyze the following WebShop transition model error cases and summarize one actionable improvement rule. Follow these guidelines: [Error Cases] {cases} [WebShop Format] • The state text is the current WebShop page content. • The action is an environment action such as: – click[buy now] – click[< prev] – search[...] • The choice is a candidate next state or a terminal token: Success or Fail. [Analysis Requirements] 1. Identify shared action types across cases. 2. Infer what the correct next page or terminal result should be. 3. Formulate one generalizable and checkable rule returning a score in $[-1,1]$ . 4. Apply the rule only when its conditions match; otherwise return 0.0. 5. Phrase the rule after ### Rule ###. 6. Write a Python program after ### Program ###. [Example Rules and Programs] 1) Example rule (Buy Now missing =¿ Fail): If action is exactly click[buy now] but the current page text does NOT contain a Buy Now button, then the correct terminal result is Fail. Return 1 if choice is Fail and -1 otherwise. Example Program: (Omitted) 2) Example rule (¡ Prev =¿ back to results): If action is click[¡ prev] and the current page text contains a ¡ Prev button, then the next page is usually a search results list page that contains ’Total results’ and a ’Next ¿’ button. Example Program: (Omitted) Rule induction on Plancraft User: Analyze the following model error cases and summarize one actionable improvement rule. Follow these guidelines: [Error Cases] {cases} [Analysis Requirements] 1. Try to find patterns in the questions. What do they have in common? Are the actions of the same type? Do the states share similarities? 2. Try to find patterns in the correct answers. What are their shared characteristics? 3. Try to find patterns in the incorrect answers. What makes them incorrect? 4. Formulate one generalizable rule for the presented error cases. The rule should be detailed enough to be programmed and used to score each candidate choice. It should apply only when the shared patterns are observed, encouraging correct patterns and discouraging incorrect ones. 5. If a detailed rule cannot be found for all error cases, describe a rule for a subset rather than being vague. 6. Phrase the rule after ### Rule ###. 7. Write a Python program after ### Program ###. The program should define a function rule_reward(state, action, choice) that returns a float in $[-1,1]$ , indicating how likely the choice is correct. [Example Rule and Programs] Example rule 1. For a move action ”move: from [I?] to [A/B/C?] with quantity q”, the next state should reflect only the intended move. Penalize choices that change unrelated item counts. Example program: (Omitted) Example rule 2. For illegal actions other than move or smelt, the next state should not change. Example program: (Omitted) Rule induction for ScienceWorld User: Analyze the following ScienceWorld model error cases and summarize one actionable improvement rule. Follow these guidelines: [Error Cases] {cases} [ScienceWorld Format] • The state text is a compressed history including task description, optional memory, per-step history (action, observation, reward), and inventory or inventory_diff snippets. It ends with current_step_action: ¡action¿. • The choice must contain exactly the following lines (order fixed): – predicted_observation: ¡text¿ – predicted_reward: ¡float in [-1,1]¿ – predicted_inventory_diff: ¡zero or more +/- lines¿ [Analysis Requirements] 1. Identify common action types (e.g., open X, pick up Y, use thermometer on Z). 2. Infer what consistent predictions should look like for these actions. 3. Design a rule that checks these consistencies by parsing the choice and, if needed, extracting the current action from the state. 4. The rule should be specific and return a score in $[-1,1]$ . 5. Phrase the rule after ### Rule ###. 6. Write a Python program after ### Program ### defining rule_reward(state, action, choice). [Example Rules and Programs] 1)Example rule (Pick up): If action starts with ’pick up ¡obj¿’, then: - predicted_observation contains ’You move the ¡obj¿ to the inventory.’ - predicted_inventory_diff contains a line starting with ’+ ’ and mentioning ¡obj¿. Example Program: (Omitted) 2) Example rule (Open): If action matches ’open ¡obj¿’, require: - predicted_observation contains ’The ¡obj¿ is now open.’ (or ’already open’) - predicted_inventory_diff is empty or contains no changes Example Program: (Omitted) 3) Example rule (Thermometer): If action contains ’use thermometer’ and ’on ¡target¿’, then: - predicted_observation contains ’the thermometer measures a temperature of’ - predicted_inventory_diff should be empty Example Program: (Omitted) We use the following prompt for reflection in rule induction: Prompt for reflection in rule induction User: The following rule is causing negative impacts on some questions that were originally answered correctly. Please refine the rule to avoid these negative impacts while maintaining its beneficial effects. [Current Rule Description] {rule_description} [Current Rule Program] {current_rule} [Negative Impact Cases] {negative_impacted cases} [Refinement Requirements] 1. Analyze why the current rule is causing originally correct answers to become wrong. 2. Identify the specific conditions or patterns responsible for the negative impact. 3. Refine the rule to be more precise and avoid these false positives. 4. Maintain the beneficial effects the rule was designed to achieve. 5. Make the refined rule more conservative to avoid breaking correct predictions. 6. Write the refined rule after ### Rule ###. 7. Write the refined Python program after ### Program ###. The program should define rule_reward(state, action, choice) and return a float in $[-1,1]$ . [Focus Areas for Refinement] • Add more specific conditions to prevent false positives. • Consider edge cases that may be incorrectly handled. • Make the rule more conservative in its judgments. • Add additional validation checks before applying penalties or rewards. ## Appendix C Clustering Details We cluster the error cases based on the embeddings of the following part of the case respectively: - question - correct answer - wrongly selected answer - action - task name To convert text to embedding, we first concatenate a tf-idf embedding of the selected text with the embedding generated by all-MiniLM-L6-v2 from sentence transformers (Reimers & Gurevych, 2019). We reduce the dimension to 50 with a truncated SVD, then to 5 with a UMAP. We use the OPTICS algorithm with min samples=3, xi=0.05 and min cluster size=0.1 for clustering. ## Appendix D Router Training We train the XGBoost router with the following features: - Scores provided by each rule - Logprob of each options provided by the base Neural WM - Length of the question - Length of the action - task name We try all combinations of the features and select the model with best dev set accuracy. We use a fixed set of hyperparameters: n estimators=2500, max depth=6, learning rate=0.01. The detailed result is shown in Table 4. The analysis can be found at Section 5.5. Table 4: Performance of an oracle router and an actually trained XGBoost router. | NeSyS XGBoost router Oracle router | 68.3 68.7 82.9 | 92.2 92.4 94.3 | 87.7 88.6 95.8 | 82.7 83.2 91.0 | | --- | --- | --- | --- | --- | | Qwen3-4B | | | | | | NeSyS | 71.0 | 92.6 | 88.4 | 84.0 | | XGBoost router | 71.0 | 91.3 | 91.5 | 84.6 | | Oracle router | 84.2 | 99.9 | 98.2 | 94.1 | ## Appendix E Benchmark Creation Details As discussed in Section 5.1, we create the benchmarks based on the official expert trajectories provided by the environment. Here we enclose the details for each environment. For ScienceWorld, we sample at most 50 dev/test variations from each pre-defined task (there are 30 predefined tasks in the environment), and at most 5 positions from each variation. We create one question at each position. For Webshop, we do a random 90:5:5 train:dev:test split on the provided trajectories. We use every time step in the dev/test trajectories to create a question. Since all the trajectories end up successfully buying the desired item, we randomly select middle positions of each trajectory and insert a “click[buy now]” action, which will result in a ”Fail” state. Similar operation is performed on the training set. For Plancraft, we sample at most 6 steps from each provided dev/test trajectories. We enriched the training trajectory by - Using side products in existing trajectory as goal and truncate the trajectory up to that position. - Starting from the existing setups and generating new trajectories by looping through all possible recipes. - Modifying the starting inventory of existing trajectories by removing a required ingredient or swapping it with a similar item. After we sample the questions and the correct answers, we use Llama-3.2-1b-instruct to answer questions and use the wrong ones as the distractors. ## Appendix F Rigorous Definition of POMDP In Section 3, we defined the problem in a informal way for better understanding. Here we provide the rigorous definition of the POMDP problem we are studying. The problem is usually written as $\mathcal{M}=(S,A,T,R,\Omega,O)$ , where $S$ denotes the latent state space, $A$ the action space, $T:S\times A\to\Delta(S)$ the transition kernel, $R:S\times A\to\mathbb{R}$ the reward function, $\Omega$ the observation space, and $O:S\times A\to\Delta(\Omega)$ the observation kernel. In this work, both actions $a_{t}\in A$ and observations $\omega_{t}\in\Omega$ are represented as natural-language strings. A world model is defined as a predictive model that estimates the next observation $\omega_{t}$ and reward $r_{t}$ conditioned on the current action $a_{t}$ and belief state $b_{t}$ . Following prior LLM-based agent work, we approximate the belief state by a textual context constructed from the task description $g$ and a truncated history of recent observations, actions, and rewards that fit within the model’s context window.