2511.22904v1

# Language-conditioned world model improves policy generalization by reading environmental descriptions **Authors**: - Anh (Joe) Nguyen (Oregon State University) - &Stefan Lee (Oregon State University) LAW 2025: Bridging Language, Agent, and World Models ## Abstract To interact effectively with humans in the real world, it is important for agents to understand language that describes the dynamics of the environment—that is, how the environment behaves —rather than just task instructions specifying what to do. For example, a cargo-handling robot might receive a statement like "the floor is slippery so pushing any object on the floor will make it slide faster than usual". Understanding this dynamics-descriptive language is important for human-agent interaction and agent behavior. Recent work [20, 40, 6] address this problem using a model-based approach: language is incorporated into a world model, which is then used to learn a behavior policy. However, these existing methods either do not demonstrate policy generalization to unseen language or rely on limiting assumptions. For instance, assuming that the latency induced by inference-time planning is tolerable for the target task or that expert demonstrations are available. Expanding on this line of research, we focus on improving policy generalization from a language-conditioned world model while dropping these assumptions. We propose a model-based reinforcement learning approach, where a language-conditioned world model is trained through interaction with the environment, and a policy is learned from this model—without planning or expert demonstrations. Our method proposes L anguage-aware E ncoder for D reamer W orld M odel (LED-WM) built on top of DreamerV3 [13]. LED-WM features an observation encoder that uses an attention mechanism to explicitly ground language descriptions to entities in the observation. We show that policies trained with LED-WM generalize more effectively to unseen games described by novel dynamics and language compared to other baselines in several settings in two environments: MESSENGER and MESSENGER-WM. To highlight how the policy can leverage the trained world model before real-world deployment, we demonstrate the policy can be improved through fine-tuning on synthetic test trajectories generated by the world model. ## 1 Introduction We envision a future where humans can seamlessly command AI agents through natural language to automate repetitive tasks in the real world. Traditionally, language has been used to specify task instructions, such as telling a navigation robot to "go to the door" [2, 17, 1]. However, language can also offer valuable information about environments. Such environmental description not only makes human interaction more natural, but also provides important contextual information about how the environment changes over time. It informs the agent about how the environment behaves —its dynamics, the current state of the world, and how various entities interact with each other and with the agent—not just what to do. <details> <summary>x1.png Details</summary>  ### Visual Description ## Grid-Based Scenario Diagram: Strategic Movement and Objectives ### Overview The image depicts a 10x10 grid-based scenario with symbolic elements representing entities and objectives. Key components include a researcher, planes, a robot, arrows, and a text box with contextual instructions. The grid appears to represent a strategic movement or decision-making environment. ### Components/Axes - **Grid Structure**: 10x10 grid with no labeled axes or numerical scales. - **Icons**: - **Researcher**: Located at (2,8) (top-right quadrant), depicted as a human figure with a magnifying glass. - **Planes**: - Three planes in the top-left quadrant (positions (2,2), (2,3), (3,2)), with two darker variants. - Three planes in the middle-right quadrant (positions (5,5), (5,6), (6,5)). - **Robot**: Positioned at (7,3) (bottom-left quadrant), depicted as a simple robotic figure. - **Arrows**: - Vertical dashed arrow from the researcher (2,8) pointing upward. - Vertical dashed arrow from the robot (7,3) pointing upward. - **Text Box**: Located to the right of the grid, containing three bullet points: 1. "The plane going away from you carries out a message" 2. "The researcher who doesn't move is final goal" 3. "The ferry chasing you is an enemy" ### Detailed Analysis - **Researcher (2,8)**: Stationary entity marked as the "final goal." The upward arrow suggests a directional priority or objective. - **Planes**: - Top-left planes (2,2), (2,3), (3,2): Two darker variants may indicate different states (e.g., active vs. inactive). - Middle-right planes (5,5), (5,6), (6,5): Positioned symmetrically, possibly representing a coordinated movement or message transmission. - **Robot (7,3)**: Dashed upward arrow implies a planned path toward the researcher. Its position suggests it is the starting point for navigation. - **Arrows**: Dashed lines indicate non-physical or strategic directions rather than physical movement. - **Text Box**: Provides critical context: - Planes moving away carry messages, implying communication or data transfer. - The researcher’s immobility defines the endpoint. - The "ferry" (not visually present) is an enemy, suggesting a threat model. ### Key Observations 1. **Pathfinding**: The robot’s dashed arrow suggests a calculated route to the researcher, avoiding obstacles (e.g., the ferry). 2. **Message Transmission**: Planes moving away from the viewer may symbolize data dissemination or mission objectives. 3. **Threat Avoidance**: The absence of a visual ferry but its mention in the text box introduces ambiguity, requiring inference about its role. 4. **Symbolic Hierarchy**: The researcher’s prominence (magnifying glass, upward arrow) emphasizes its importance over other elements. ### Interpretation This diagram likely represents a simplified strategic scenario, possibly for a game, simulation, or decision-making exercise. The grid serves as a spatial framework for navigating objectives (researcher as goal) while managing risks (ferry as enemy). The planes’ dual roles—carrying messages and potentially acting as obstacles—add complexity. The robot’s path (dashed line) contrasts with the researcher’s static position, highlighting the dynamic vs. static elements of the scenario. The text box’s instructions clarify ambiguities (e.g., the ferry’s role), suggesting the diagram is part of a larger system requiring contextual interpretation. </details> Figure 1: An example of dynamics-descriptive language in a game play. The observation includes a 10 $\times$ 10 grid-world with three entities represented by their associated symbols: (ferry - <details> <summary>figures/ferry.png Details</summary>  ### Visual Description ## Icon: Stylized Boat on Water ### Overview The image depicts a minimalist, abstract representation of a boat floating on water. The design uses geometric shapes and a limited color palette to convey the subject. No textual elements, numerical data, or explicit labels are present. ### Components/Axes - **Boat Structure**: - **Hull**: A large, dark blue (hex: #00008B) trapezoidal shape representing the boat’s base. - **Cabin**: A smaller, light blue (hex: #ADD8E6) rectangular structure atop the hull, with a white (hex: #FFFFFF) rectangular superstructure. - **Mast**: A vertical white line extending from the cabin, suggesting a sail or antenna. - **Water**: A wavy, light blue (hex: #87CEEB) band at the bottom, simulating ocean waves. ### Detailed Analysis - **Color Usage**: - Dark blue dominates the hull, symbolizing depth or stability. - Light blue and white for the cabin and mast imply openness or functionality. - **Shape Simplification**: - The hull’s trapezoidal form abstracts the boat’s structure, prioritizing recognition over realism. - The cabin’s rectangular shape contrasts with the hull’s angularity, creating visual hierarchy. - **Water Representation**: - Wavy lines in light blue mimic motion, though no explicit wave frequency or amplitude is quantified. ### Key Observations - No textual labels, legends, or numerical values are present. - The design adheres to a flat, iconographic style, avoiding gradients or shading. - The boat’s orientation is horizontal, with the mast aligned vertically. ### Interpretation This icon likely serves as a universal symbol for maritime themes, navigation, or transportation. The absence of text or data suggests it is intended for broad, non-technical audiences. The color choices and geometric abstraction prioritize clarity and scalability, making it suitable for digital interfaces or signage. No outliers or anomalies exist, as the design is intentionally minimalist. **Note**: The image contains no factual or numerical data. The description above is based solely on visual analysis of design elements and symbolic representation. </details> ), (plane - <details> <summary>figures/plane.png Details</summary>  ### Visual Description ## Icon: Stylized Airplane Symbol ### Overview The image depicts a minimalist, stylized airplane icon. It features a light blue fuselage with darker blue wings, outlined in black. The wings are positioned diagonally across the body, creating a dynamic, abstract representation. No textual elements, labels, or numerical data are present. ### Components/Axes - **Fuselage**: Central elongated shape in light blue (#ADD8E6), outlined in black. - **Wings**: Two triangular shapes in darker blue (#0000FF), outlined in black, intersecting the fuselage diagonally. - **Tail**: Small triangular extension at the rear of the fuselage, matching the light blue color. - **Background**: Plain white (#FFFFFF), providing high contrast with the black outlines. ### Detailed Analysis - **Color Scheme**: - Fuselage: Light blue (#ADD8E6) - Wings: Dark blue (#0000FF) - Outlines: Black (#000000) - **Design Elements**: - Wings are asymmetrically placed, crossing the fuselage at a 45-degree angle. - No shading, gradients, or additional details (e.g., windows, engines). - Simplified geometric shapes for clarity and recognizability. ### Key Observations - The icon prioritizes simplicity over realism, using bold outlines and flat colors. - The crossed-wing design may symbolize motion or flight dynamics. - No textual or numerical data is embedded in the image. ### Interpretation This icon likely serves as a universal symbol for aviation, travel, or flight-related applications (e.g., apps, websites, signage). The use of contrasting colors (light blue vs. dark blue) enhances visibility, while the black outline ensures scalability for small sizes. The absence of text aligns with standard iconography practices, relying on visual recognition rather than labels. The crossed wings could imply speed or agility, though this is speculative without contextual data. **Note**: The image contains no factual or numerical information. All descriptions are based on visual analysis of design elements and symbolic conventions. </details> ), (researcher - <details> <summary>figures/scientist.png Details</summary>  ### Visual Description ## Icon/Symbol: Cartoon Scientist Using Microscope ### Overview The image depicts a stylized, cartoon-like illustration of a person engaged in scientific observation using a microscope. The figure is shown in profile, facing left, with a focus on their interaction with the microscope. No textual labels, numerical data, or explicit annotations are present. ### Components/Axes - **Primary Subject**: A humanoid figure with curly brown hair, wearing a light gray lab coat and blue safety goggles. - **Microscope**: A simplified, stylized microscope with a gray body, yellow arm, and black eyepiece. - **Background**: Plain white, with no additional contextual elements. - **Color Palette**: - Brown (hair) - Gray (lab coat, microscope body) - Blue (safety goggles) - Yellow (microscope arm) - Black (eyepiece, outlines) ### Detailed Analysis - **Figure Posture**: The figure leans forward, with their right hand positioned near the microscope’s stage, suggesting active observation. - **Microscope Design**: The microscope is rendered with minimal detail, emphasizing its functional components (eyepiece, arm, stage) rather than realism. - **Absence of Text**: No labels, legends, or textual annotations are visible in the image. ### Key Observations 1. The illustration prioritizes symbolic representation over realism, using bold outlines and flat colors. 2. The lack of textual elements suggests the image is intended for generic use (e.g., educational materials, icons) rather than data visualization. 3. The figure’s gender is ambiguous due to the absence of facial features beyond hair and goggles. ### Interpretation This icon likely represents a generic scientist or researcher in a laboratory setting. The absence of specific identifiers (e.g., race, gender, institutional branding) implies it is designed for broad applicability. The use of a microscope underscores themes of scientific inquiry, experimentation, or analysis. The simplicity of the design suggests it could serve as a visual metaphor for education, discovery, or technical work in fields like biology, chemistry, or materials science. **Note**: No factual or numerical data is present in the image. The description focuses on symbolic and design elements only. </details> ) and one agent (depicted by <details> <summary>figures/bot.png Details</summary>  ### Visual Description ## Icon: Robot Symbol ### Overview The image depicts a minimalist, black-and-white line-drawn robot icon. It features a square body, two circular eyes, a circular antenna atop the head, and two rounded handles on the sides. No textual elements, numerical data, or color gradients are present. ### Components/Axes - **Body**: A centrally positioned square with rounded corners. - **Eyes**: Two identical black circles with white centers, placed symmetrically in the upper half of the body. - **Antenna**: A single vertical line extending upward from the center of the body, terminating in a small black circle. - **Handles**: Two identical rounded rectangles attached to the left and right sides of the body, resembling arms. - **Color**: Pure black lines on a white background. ### Detailed Analysis - No numerical values, labels, or legends are visible. - The design adheres to a flat, simplistic style typical of modern UI icons. - The antenna’s position (top-center) and handles’ placement (symmetrical sides) suggest a humanoid robot archetype. ### Key Observations - The absence of text or data points indicates the icon is purely symbolic, not data-driven. - The use of circles for eyes and antenna aligns with common anthropomorphic robot representations. - Symmetry in design elements (eyes, handles) emphasizes balance and simplicity. ### Interpretation This icon likely serves as a visual shorthand for robotics, AI, or automation in user interfaces, educational materials, or branding. Its simplicity prioritizes recognizability over detail, making it versatile for small-scale applications (e.g., app icons, signage). The lack of complexity suggests it is not intended for technical schematics but rather for general symbolic use. </details> ). The observation also has a manual on the right, which describes the dynamics of the game. The agent can navigate the grid using five actions: left, right, up, down, and stay. The agent can only interact with entities when it is in the same grid cell as the entity. The agent’s task is to identify roles of all entities from the manual, go to the messenger, then go to the goal, while avoiding the enemy. Shaded icons indicate one possible scenario of entity movement over time. By observing entity movement patterns and grounding language to entities based on their behaviors, the agent can infer the roles assigned to each entity: (ferry-enemy), (plane-messenger), and (researcher-goal). The agent can then execute an appropriate plan to complete the task. The dashed line in the grid shows such a possible plan. We illustrate dynamics-descriptive language by using a simple 2D grid-based game in Figure ˜ 1, instantiated by MESSENGER S2 [14]. This is the setting of our testbed environments and will be detailed in Section ˜ 3.1 Each game instance consists of several entities, an agent positioned in a grid-world observation, and a language manual. Each entity has a role among messenger, goal, and enemy. The agent acts as a courier, tasked with picking up a message from the messenger and delivering it to the goal while avoiding the enemy. The manual provides descriptions of the entity attributes, helping the agent understand the environment’s dynamics: what the roles of entities are and how the environment changes as the agent interacts with them. To succeed, the agent must interpret the language manual, identify the entities, and infer their respective roles based on observed behaviors. Language is valuable because it allows for the description of novel games by recombining known concepts. For instance, consider Figure ˜ 1 as the training reference game and the following example manual: The ship going away from you is the goal you need to go to. The stationary plane is an enemy. The scientist won’t move and has an important message. The example manual describes an unseen dynamics game with known concepts derived from the reference game. To succeed in this environment—where the dynamics have changed but the rules remains the same—the agent must adopt a different behavior than in the reference game. This manual also produces novel surface-level language through synonyms (e.g. "researcher" vs "scientist") and paraphrases ("won’t move" vs "stationary"), We want to study language grounding and how it affects agent generalizability. Therefore, we abstract away our observation to a discrete grid-world, thus simplifying perception complexity, similar to existing work [14, 26, 20, 40, 6]. Our goal is to develop an agent capable of understanding dynamics-descriptive language by grounding it to discrete entities. More importantly, we aim for the agent to generalize to unseen games described by unseen dynamics and/or novel language, allowing it to adapt agent behavior to new environmental changes. In the current literature, there are two main approaches to building such an agent: model-free and model-based approach. Model-free methods [14, 26] directly map language to a policy. Language grounding is thus based entirely on policy learning signals, without modeling the environment dynamics. This might be challenging for agent to learn complex mapping from dynamics-descriptive language to action. Meanwhile, model-based methods like EMMA-LWM [40], Reader [6], and Dynalang [20] build a world model [11] simulating trajectories, which are then used to train a policy. Dynamics-descriptive language is incorporated into the world model, enabling it to use language to predict environmental changes. However, these existing works have some limitations. EMMA-LWM requires expert demonstrations—a constraint that may not be always feasible for real-life tasks. Reader assumes inference-time latency is tolerable for the target tasks. This is because Reader uses a Monte Carlo Tree Search (MCTS) to look ahead and generate a full plan. This approach may not be practical for applications that require quick policy responses. Last, we show that the policy learned from Dynalang fails to generalize over unseen games in Section ˜ 5.1. To address these limitations, we adopt a model-based reinforcement learning (MBRL) approach that builds a language-grounded world model from interaction with the environment, and then use this world model to train a policy. In contrast to previous methods, our approach does not require expert demonstrations, avoids expensive inference-time planning, and can generalize to unseen games. We propose L anguage-aware E ncoder for D reamer W orld M odel (LED-WM), building on a MBRL framework: DreamerV3 [13]. LED-WM introduces a new encoder for DreamerV3 that explicitly grounds entities to their language descriptions, using a simple yet effective attention mechanism. In this paper, we make the following contributions: - We show that a language-conditioned MBRL without an explicit language grounding to entities, instantiated by Dynalang [20], fails to generalize over unseen games (see Section ˜ 5.1). - By using an attention mechanism in LED-WM to do language grounding, we show that a policy trained from LED-WM can generalize over unseen games better than model-free and model-based baselines in several settings MESSENGER and MESSENGER-WM (see Section ˜ 4). - We demonstrate that given a trained LED-WM, we can improve a trained policy by fine-tuning it in synthetic test trajectories generated by the world model (see Section ˜ 5.2). ## 2 Background Problem formulation. We define our problem as a language-conditioned Markov Decision Process, represented by a tuple with common notations: $(\mathcal{S},\mathcal{A},r,T,\gamma,H)$ . $\mathcal{S}$ represents the state space where each state has a 10 $\times$ 10 grid-world observation containing entity symbols and an agent (e.g. <details> <summary>figures/ferry.png Details</summary>  ### Visual Description ## Icon: Stylized Boat on Water ### Overview The image depicts a minimalist, abstract representation of a boat floating on water. The design uses geometric shapes and a limited color palette to convey the subject. No textual elements, numerical data, or explicit labels are present. ### Components/Axes - **Boat Structure**: - **Hull**: A large, dark blue (hex: #00008B) trapezoidal shape representing the boat’s base. - **Cabin**: A smaller, light blue (hex: #ADD8E6) rectangular structure atop the hull, with a white (hex: #FFFFFF) rectangular superstructure. - **Mast**: A vertical white line extending from the cabin, suggesting a sail or antenna. - **Water**: A wavy, light blue (hex: #87CEEB) band at the bottom, simulating ocean waves. ### Detailed Analysis - **Color Usage**: - Dark blue dominates the hull, symbolizing depth or stability. - Light blue and white for the cabin and mast imply openness or functionality. - **Shape Simplification**: - The hull’s trapezoidal form abstracts the boat’s structure, prioritizing recognition over realism. - The cabin’s rectangular shape contrasts with the hull’s angularity, creating visual hierarchy. - **Water Representation**: - Wavy lines in light blue mimic motion, though no explicit wave frequency or amplitude is quantified. ### Key Observations - No textual labels, legends, or numerical values are present. - The design adheres to a flat, iconographic style, avoiding gradients or shading. - The boat’s orientation is horizontal, with the mast aligned vertically. ### Interpretation This icon likely serves as a universal symbol for maritime themes, navigation, or transportation. The absence of text or data suggests it is intended for broad, non-technical audiences. The color choices and geometric abstraction prioritize clarity and scalability, making it suitable for digital interfaces or signage. No outliers or anomalies exist, as the design is intentionally minimalist. **Note**: The image contains no factual or numerical data. The description above is based solely on visual analysis of design elements and symbolic representation. </details> , <details> <summary>figures/plane.png Details</summary>  ### Visual Description ## Icon: Stylized Airplane Symbol ### Overview The image depicts a minimalist, stylized airplane icon. It features a light blue fuselage with darker blue wings, outlined in black. The wings are positioned diagonally across the body, creating a dynamic, abstract representation. No textual elements, labels, or numerical data are present. ### Components/Axes - **Fuselage**: Central elongated shape in light blue (#ADD8E6), outlined in black. - **Wings**: Two triangular shapes in darker blue (#0000FF), outlined in black, intersecting the fuselage diagonally. - **Tail**: Small triangular extension at the rear of the fuselage, matching the light blue color. - **Background**: Plain white (#FFFFFF), providing high contrast with the black outlines. ### Detailed Analysis - **Color Scheme**: - Fuselage: Light blue (#ADD8E6) - Wings: Dark blue (#0000FF) - Outlines: Black (#000000) - **Design Elements**: - Wings are asymmetrically placed, crossing the fuselage at a 45-degree angle. - No shading, gradients, or additional details (e.g., windows, engines). - Simplified geometric shapes for clarity and recognizability. ### Key Observations - The icon prioritizes simplicity over realism, using bold outlines and flat colors. - The crossed-wing design may symbolize motion or flight dynamics. - No textual or numerical data is embedded in the image. ### Interpretation This icon likely serves as a universal symbol for aviation, travel, or flight-related applications (e.g., apps, websites, signage). The use of contrasting colors (light blue vs. dark blue) enhances visibility, while the black outline ensures scalability for small sizes. The absence of text aligns with standard iconography practices, relying on visual recognition rather than labels. The crossed wings could imply speed or agility, though this is speculative without contextual data. **Note**: The image contains no factual or numerical information. All descriptions are based on visual analysis of design elements and symbolic conventions. </details> , <details> <summary>figures/scientist.png Details</summary>  ### Visual Description ## Icon/Symbol: Cartoon Scientist Using Microscope ### Overview The image depicts a stylized, cartoon-like illustration of a person engaged in scientific observation using a microscope. The figure is shown in profile, facing left, with a focus on their interaction with the microscope. No textual labels, numerical data, or explicit annotations are present. ### Components/Axes - **Primary Subject**: A humanoid figure with curly brown hair, wearing a light gray lab coat and blue safety goggles. - **Microscope**: A simplified, stylized microscope with a gray body, yellow arm, and black eyepiece. - **Background**: Plain white, with no additional contextual elements. - **Color Palette**: - Brown (hair) - Gray (lab coat, microscope body) - Blue (safety goggles) - Yellow (microscope arm) - Black (eyepiece, outlines) ### Detailed Analysis - **Figure Posture**: The figure leans forward, with their right hand positioned near the microscope’s stage, suggesting active observation. - **Microscope Design**: The microscope is rendered with minimal detail, emphasizing its functional components (eyepiece, arm, stage) rather than realism. - **Absence of Text**: No labels, legends, or textual annotations are visible in the image. ### Key Observations 1. The illustration prioritizes symbolic representation over realism, using bold outlines and flat colors. 2. The lack of textual elements suggests the image is intended for generic use (e.g., educational materials, icons) rather than data visualization. 3. The figure’s gender is ambiguous due to the absence of facial features beyond hair and goggles. ### Interpretation This icon likely represents a generic scientist or researcher in a laboratory setting. The absence of specific identifiers (e.g., race, gender, institutional branding) implies it is designed for broad applicability. The use of a microscope underscores themes of scientific inquiry, experimentation, or analysis. The simplicity of the design suggests it could serve as a visual metaphor for education, discovery, or technical work in fields like biology, chemistry, or materials science. **Note**: No factual or numerical data is present in the image. The description focuses on symbolic and design elements only. </details> , <details> <summary>figures/bot.png Details</summary>  ### Visual Description ## Icon: Robot Symbol ### Overview The image depicts a minimalist, black-and-white line-drawn robot icon. It features a square body, two circular eyes, a circular antenna atop the head, and two rounded handles on the sides. No textual elements, numerical data, or color gradients are present. ### Components/Axes - **Body**: A centrally positioned square with rounded corners. - **Eyes**: Two identical black circles with white centers, placed symmetrically in the upper half of the body. - **Antenna**: A single vertical line extending upward from the center of the body, terminating in a small black circle. - **Handles**: Two identical rounded rectangles attached to the left and right sides of the body, resembling arms. - **Color**: Pure black lines on a white background. ### Detailed Analysis - No numerical values, labels, or legends are visible. - The design adheres to a flat, simplistic style typical of modern UI icons. - The antenna’s position (top-center) and handles’ placement (symmetrical sides) suggest a humanoid robot archetype. ### Key Observations - The absence of text or data points indicates the icon is purely symbolic, not data-driven. - The use of circles for eyes and antenna aligns with common anthropomorphic robot representations. - Symmetry in design elements (eyes, handles) emphasizes balance and simplicity. ### Interpretation This icon likely serves as a visual shorthand for robotics, AI, or automation in user interfaces, educational materials, or branding. Its simplicity prioritizes recognizability over detail, making it versatile for small-scale applications (e.g., app icons, signage). The lack of complexity suggests it is not intended for technical schematics but rather for general symbolic use. </details> in Figure ˜ 1). Each state also has a language manual $L$ , describing environment dynamics: transition function $T(s^{\prime}|s,a)$ and reward function $r(s,a)$ . $L$ consists of $N$ sentences associated with $N$ entities, where each sentence describes the dynamics of each entity. An example of a state is shown in Figure ˜ 1. Action space $\mathcal{A}=\{\texttt{up, down, right, left, stay}\}$ is discrete. The agent must take a sequence of actions $a_{t}\in\mathcal{A}$ over a horizon $H$ , where time step $t\in[1..H]$ , resulting in a state-action trajectory $(s_{1},a_{1},\ldots,s_{H},a_{H})$ . Our goal is to find a policy $\pi:\mathcal{S}\times L\rightarrow\mathcal{A}$ that maximizes the expected sum of discounted rewards: $\mathbb{E}_{\pi,L}\left[\sum_{t=1}^{H}\gamma^{t-1}r(s_{t},a_{t})\right].$ World model DreamerV3. We base our world model on DreamerV3 [13], which uses Recurrent State-Space Model (RSSM) [12] to build a recurrent world model. DreamerV3 receives a sequence of observations and predicts latent representations of future observations given actions. Specifically, at a time step $t$ , DreamerV3 receives an observation $x_{t}$ , an action $a_{t}$ , and history information $h_{t}$ . These inputs are compressed into a latent representation $z_{t}$ and fed to RSSM with the action $a_{t}$ to predict the next latent representation $z_{t+1}$ . The world model has the following components: $$ \displaystyle\raisebox{25.00003pt}{ $\text{RSSM}~~\begin{cases}\hphantom{A}\\[-6.0pt] \hphantom{A}\\[-6.0pt] \hphantom{A}\end{cases}$}\begin{aligned} &\text{Sequence model:}&\quad h_{t}&=f_{\phi}(h_{t-1},z_{t-1},a_{t-1})\\ &\text{Encoder:}&\quad z_{t}&\sim q_{\phi}(z_{t}\mid h_{t},x_{t})\\ &\text{Dynamics predictor:}&\quad\hat{z}_{t}&\sim p_{\phi}(\hat{z}_{t}\mid h_{t})\\ &\text{Reward predictor:}&\quad\hat{r}_{t}&\sim p_{\phi}(\hat{r}_{t}\mid h_{t},z_{t})\\ &\text{Continue predictor:}&\quad\hat{c}_{t}&\sim p_{\phi}(\hat{c}_{t}\mid h_{t},z_{t})\\ &\text{Decoder:}&\quad\hat{x}_{t}&\sim p_{\phi}(\hat{x}_{t}\mid h_{t},z_{t})\\ \end{aligned} \tag{1} $$ In this work, we propose to change the encoder of DreamerV3 to better leverage language grounding to learn a more robust world model. ## 3 Environment setup We adopt MESSENGER [14] and MESSENGER-WM [40] as our test bed environments. Both environments have the same setup as the example game in Figure ˜ 1. To succeed in the game, the agent must understand the language manuals $L$ and use reward and transitional signals to ground the roles, entity names and movement types to the entity symbols in the observation. ### 3.1 MESSENGER Overview. As shown in Figure ˜ 1, MESSENGER [14] is a 10 $\times$ 10 grid-world environment. We refer the readers to Figure ˜ 1 for game rules and setup, and Section ˜ C.1.1 for environment dynamics and action. Each game includes a language manual and an observation containing entities and a single agent. For more details about language grounding to entities, we refer the readers to Section ˜ C.1.2. Evaluation settings. MESSENGER offers four stages (stage S1, S2, S2-dev, S3) with different levels of generalization for test games. Each stage has its own training set, test set, and development set, all of which are described in detail in Section ˜ C.1. ### 3.2 MESSENGER-WM Overview. While providing multiple stages to evaluate policy generalization over out-of-distribution dynamics, MESSENGER does not include a setting for compositional generalization dynamics. To bridge this gap, MESSENGER-WM [40], derived from MESSENGER S2, enables evaluation at compositional generalization for world model and policy. Together, these two environments offer a comprehensive framework for assessing generalization, under varying levels of unseen games. Evaluation settings. MESSENGER-WM has three different evaluation settings with different levels of generalization: NewCombo, NewAttr, and NewAll. All settings share the same training set. More details about the evaluation settings are provided in Section ˜ C.3.2 and the original paper [40]. ### 3.3 Evaluating generalization to unseen games Table 1: Summary of generalization capabilities over unseen games in MESSENGER and MESSENGER-WM across stages. Examples with visualizations are provided in Section ˜ C.2. | | MESSENGER | MESSENGER-WM | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | | S1 | S2 | S2-dev | S3 | New | New | New | | | Combo | Attr | All | | | | | | | Novel combinations of known entities | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | | Novel language (synonyms and paraphrase) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | Novel entity-role assignments | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | | Novel entity-movement-role assignments | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | | Novel game dynamics of known movement behaviors | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | | Novel game dynamics from one training dynamic | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | Together, MESSENGER and MESSENGER-WM offer different levels of generalization in test games. We summarize these in Table ˜ 1 and below: - Novel language: the test manual uses synonyms and paraphrases to create novel language through surface structure. - Novel combinations of known entities: the test game involve entities that appear in training set but never appear together in one training game. - Novel entity-role assignments: at least one entity in the test game has a different role from its roles in training games. - Novel entity-role-movement assignment: at least one entity in the test game has a novel combination of entity-role-movement assignment. - Novel combinations of known movement behaviors (novel game dynamics): the test game has a novel movement combinations of entities, e.g. (chaser-chaser-chaser) for three entities. - Novel game dynamics from one training dynamic: Game dynamic is defined by the combination of entity movements. In the training set, there is only one such combination (chasing-fleeing-stationary) across all training games. The test game meanwhile has a novel dynamic, e.g. (chaser-chaser-chaser). This is also the difference between MESSENGER-WM and MESSENGER, which can be found more detailed in Section ˜ C.3.1 See Section ˜ C.2 for visualizations of these settings. ## 4 Method: L anguage-aware E ncoder for D reamer W orld M odel (LED-WM) To generalize policy across unseen games, we aim to develop a world model capable of doing language grounding to entities in a game. Inspired by EMMA [14], we propose L anguage-aware E ncoder for D reamer (LED), which uses cross-modal attention to align game entities with sentences. The resulting vectors are then placed back into their original entity locations, producing a language-aware grid observation. This grid is passed through a CNN encoder to extract observation features, which are used by the other components of DreamerV3. We call this overall model LED- W orld M odel (LED-WM). We provide an overview of the encoder LED and the world model LED-WM in Figure ˜ 2 and describe each component in the following sections. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: LED-World Model (LED-WM) Architecture ### Overview This diagram illustrates the architecture of the LED-World Model (LED-WM), a language-aware encoder system for the Dreamer framework. It integrates natural language instructions, grid-based observations, and temporal embeddings to guide decision-making in a simulated environment. The model processes inputs through a T5 Encoder, attention mechanisms, and a CNN to produce a language-guided grid output for DreamerV3. ### Components/Axes 1. **Language Manual (Component 1)** - Textual instructions: - "The plane doesn't move and has a message" - "The scientist going closer to you is the goal" - "The ship away from you is an enemy" 2. **Grid Observation (Component 2)** - Grid layout with symbols: - Plane (✈️) at top-left - Scientist (👨‍🔬) at bottom-left - Ship (🚢) at bottom-right - Dashed line path connecting entities 3. **Language-aware Encoder (LED) (Component 3)** - **T5 Encoder**: Processes language manual inputs (`v_i`, `k_i`) - **Softmax Attention**: Combines entity embeddings (`q_i`) with positional history - **Entity Embedding**: Represents grid entities (plane, scientist, ship) - **Positional History**: Tracks movement/relationships over time - **Time Embedding**: Adds temporal context (`z_t`) 4. **Language-guided Grid (Component 4)** - 3D grid with colored bars: - `e1` (pink), `e2` (blue), `e3` (green) - **CNN**: Processes grid data to extract features 5. **DreamerV3 (Component 5)** - Final output module integrating temporal and spatial data **Legend Colors**: - Purple: T5 Encoder - Green: Entity Embedding - Blue: Positional History - Orange: CNN - Gray: Time Embedding ### Detailed Analysis - **Language Manual**: Explicitly defines agent goals (scientist as goal, ship as enemy) and static entities (non-moving plane). - **Grid Observation**: Visualizes spatial relationships between entities via dashed paths. - **T5 Encoder**: Converts textual instructions into embeddings (`v_i`, `k_i`) for contextual understanding. - **Softmax Attention**: Merges entity embeddings (`q_i`) with positional history to prioritize relevant grid regions. - **CNN**: Extracts spatial features from the language-guided grid (`e1`, `e2`, `e3`). - **Time Embedding**: Injects temporal context (`z_t`) to model dynamic interactions. ### Key Observations - The model emphasizes **language grounding** by linking textual instructions (e.g., "scientist is the goal") to grid entities. - **Attention mechanisms** dynamically weight grid regions based on linguistic and positional context. - **CNN output** (`e1`, `e2`, `e3`) likely represents encoded spatial relationships for decision-making. - **DreamerV3** integrates temporal (`z_t`) and spatial features to guide agent behavior. ### Interpretation The LED-WM bridges natural language and grid-based perception by: 1. Translating instructions into embeddings via the T5 Encoder. 2. Using attention to focus on relevant grid regions (e.g., scientist as goal). 3. Combining spatial features (via CNN) with temporal context to inform DreamerV3's actions. This architecture suggests a **multimodal reinforcement learning system** where language instructions directly shape agent navigation and goal-oriented behavior in a dynamic environment. The absence of numerical data implies the diagram focuses on architectural flow rather than quantitative performance metrics. </details> Figure 2: Overview of our proposed world model LED-WM. The world model input consists of: <details> <summary>x7.png Details</summary>  ### Visual Description Icon/Small Image (23x22) </details> a language manual $L$ , <details> <summary>x8.png Details</summary>  ### Visual Description Icon/Small Image (22x22) </details> a grid-world observation representing entity and agent symbols, and <details> <summary>x9.png Details</summary>  ### Visual Description Icon/Small Image (22x22) </details> the current time step $t$ . Entity, agent symbols, and time step are encoded using learned embeddings, while $L$ is encoded via a frozen T5 encoder. To represent each entity, we employ a multi-layer perceptron (MLP) that processes the entity embedding and its temporal information, capturing its movement pattern relative to the agent, to produce a query vector. We apply an attention network between the query vectors and the sentence embeddings to align each entity with its corresponding sentence. The resulting vectors are then put into their respective entity positions. This produces <details> <summary>x10.png Details</summary>  ### Visual Description Icon/Small Image (22x22) </details> a language-grounded grid $G_{l}$ , which is then processed by a CNN. The extracted feature vector is flattened and concatenated with the time embedding to form final observation representation $x_{t}$ . ### 4.1 Observational inputs The input to the world model consists of a natural language manual $L$ , a grid observation $o_{t}$ of size $10\times 10$ , containing symbolic entities, and the current time step $t$ . The manual <details> <summary>x11.png Details</summary>  ### Visual Description Icon/Small Image (24x23) </details> comprises $N$ sentences, each describing the dynamic of one of the $N$ entities in the observation. Following Lin et al. [20], $N$ sentences in $L$ are encoded using a T5 encoder [30], resulting in $N$ frozen sentence embeddings, denoted by $s_{1},s_{2},...,s_{N}$ . In the grid <details> <summary>x12.png Details</summary>  ### Visual Description Icon/Small Image (23x23) </details> , the $N$ entities and the agent, represented by entity symbols, are encoded using a learned entity embedding vectors initialized with random weights. This ensures the agent does not have prior knowledge about entity identities, requiring it to infer entities based on the language. This results in $N$ symbol embeddings $sb_{1},sb_{2},...,sb_{N}\in\mathbb{R}^{d_{sb}}$ and a single agent embedding $a\in\mathbb{R}^{d_{sb}}$ . The current time step $t$ <details> <summary>x13.png Details</summary>  ### Visual Description Icon/Small Image (23x23) </details> is encoded as $time_{t}$ using a learned time embedding, also initialized with random weights. To build position history of each entity $i$ , we capture temporal dynamics by constructing an array $D_{i}$ temporally, with length corresponding to the maximum possible steps in the environment and initial values of $-1$ . At time step $t$ , let the 2D coordinate of the entity $i$ be $p^{t}_{i}$ and that of the agent be $p^{t}_{a}$ . To determine the relative direction of the entity’s movement with respect to the agent, we compute the dot product: $$ D_{i}^{t}=\frac{p^{t}_{i}-p^{t}_{a}}{\|p^{t}_{i}-p^{t}_{a}\|}\cdot\frac{p^{t}_{i}-p^{t-1}_{i}}{||p^{t}_{i}-p^{t-1}_{i}||},\forall i\in[1..N],\forall t, \tag{2} $$ where the first term is a normalized vector from the agent to the entity $i$ , and the second term is a normalized velocity vector of the entity $i$ . This dot product quantifies the alignment between the entity’s direction of motion and its position relative to the agent at each time step $t$ . ### 4.2 LED: Building a language-aware encoder We construct a language-grounded grid representation that aligns the language manual $L$ , which consists of $N$ sentence embeddings, with the observation $o_{t}$ , which includes $N$ entity embeddings and one agent embedding. To align the sentence embeddings with the entity embeddings, we use an attention network. The values are obtained through a linear transformation of the sentence embeddings $s_{i}$ . Meanwhile, the queries are obtained through a multi-layer perception (MLP) applied to the entity embeddings $sb_{i}$ and temporal array $D_{e}$ . Likewise, the keys are obtained through an MLP applied to the sentence embeddings $s_{i}$ : $$ \displaystyle q_{i} \displaystyle=\text{MLP}([sb_{i},D_{e}]), \displaystyle k_{i} \displaystyle=\text{MLP}(s_{i}), \displaystyle v_{i} \displaystyle=W_{v}s_{i}, \displaystyle q_{i} \displaystyle\in\mathbb{R}^{d}, \displaystyle k_{i} \displaystyle\in\mathbb{R}^{d}, \displaystyle v_{i} \displaystyle\in\mathbb{R}^{d_{\text{val}}}, \tag{3} $$ where $d$ and $d_{val}$ denote the dimensions of the query/key and value vectors, respectively. We then apply scaled dot-product attention [35]. Given $K\in R^{N\times d}$ as the key matrix where the row $i$ of $K$ is $k_{i}^{T}$ , attention scores $\gamma_{i}\in R^{N}$ and resulting vector $e_{i}\in R^{d_{a}}$ for each entity $i$ are calculated as: $$ \displaystyle\gamma_{i}=softmax\left(\frac{q_{i}\cdot K}{\sqrt{d}}\right),\quad e_{i}=\sum_{j=1}^{N}\gamma_{ij}v_{j}, \tag{5} $$ This attention aligns entity symbols in the observation with sentences in the manual based on attribute language descriptions such as movement (e.g., chaser, moving away, stationary) and entity name (e.g., dog, wizard). The resulting $e_{i}$ from the attention is able to represent an associated role for entity $i$ such as enemy, messenger, goal, which is vital information for world model and policy learning. To retain the spatial information of entities, we place the resulting vectors $e_{i}$ back into the original positions of their corresponding entities in the grid observation. This produces <details> <summary>x14.png Details</summary>  ### Visual Description Icon/Small Image (23x23) </details> a language-aware grid observation $G_{l}$ of size $h\times w\times d_{val}$ . We then use a CNN encoder to extract a feature map, which is subsequently flattened and concatenated with the time embedding $\text{time}_{t}$ . The combined representation is processed through an MLP to obtain the final feature representation $x_{t}$ for the observation $o_{t}$ at time step $t$ : $$ \displaystyle x_{t} \displaystyle=\text{MLP}(\text{Flatten}([\text{CNN}(G_{l})),\text{time}_{t}]) \tag{6} $$ Denoting $\phi$ as the parameters of LED-WM, we can find stochastic variable $z_{t}$ as the function of $x_{t}$ : $$ \displaystyle z_{t}\sim q_{\phi}(z_{t}|h_{t},x_{t}), \tag{7} $$ which now replaces the encoder in DreamerV3, as shown in Equation ˜ 1. ### 4.3 LED-WM: Combining LED with Dreamerv3 We replace DreamerV3’s encoder with LED, resulting in our world model LED-WM. We adopt world model and policy learning from DreamerV3. However, we make the following changes to the original architecture to improve policy generalization and sample efficiency: we omit the reconstruction decoder (Decoder in Equation ˜ 1) and adopt multi-step prediction for reward and continue prediction [15, 27]. For more details, we refer the readers to Appendix ˜ D for world model loss and Appendix ˜ E for training procedure. ## 5 Experiments We want to answer the following questions: 1) Can a policy trained on our world model LED-WM generalize to unseen games? (see Section ˜ 5.1), and 2) Can the world model LED-WM generalize to unseen games? (see Section ˜ 5.2) To answer these, we use two environments: MESSENGER and MESSENGER-WM, which are detailed in Section ˜ 3. We detail the training settings in Appendix ˜ A. ### 5.1 Policy generalization trained from LED-WM #### 5.1.1 Policy baselines As baselines, we adopt the following model-free (EMMA,CRL) and model-based (Dynalang, EMMA-LWM) methods: - EMMA [14] uses attention between entities and sentences to generate language-conditioned observation to the policy. The policy is trained via curriculum learning where the agent is initialized with parameters learned from previous easier game settings. We report EMMA with curriculum learning from the original paper and EMMA without curriculum learning from [40]. - CRL [26] develops a specialized constraint for MESSENGER to overcome spurious correlations between entity identities and their roles in the training data. It has the state-of-the-art win rate performance in test environments of MESSENGER. - Dynalang [20] use soft actor-critic for policy learning. Because the paper does not report policy generalization performance in MESSENGER, we first reproduce Dynalang using published code and train to convergence according to published hyperparameters and training steps. We then report its policy performance on test environment of MESSENGER in Table ˜ 2. - EMMA-LWM [40] built a language-conditioned world model. A policy is trained with simulated trajectories from this world model through online imitation learning and filtered behavior cloning. Both methods require expert demonstrations. Online imitation learning is where the expert supervises the optimal action to take in simulated states (from the world model). Meanwhile, in filtered behavior cloning, the expert uses only states from its own expert plan. The agent then only chooses plans that achieve the highest returns according to the world models to imitate. #### 5.1.2 Evaluation metrics - Win Rates for MESSENGER: To make our comparison consistent with reported results from EMMA [14] and CRL [26], we adopt win rate as the metric in MESSENGER. Win rate is calculated as the average number of games won by the agent over 1000 episodes. - Average Sum of Scores for MESSENGER-WM: Likewise to be consistent with EMMA-LWM [40] studying MESSENGER-WM, we adopt average sum of scores as the metric. For each game configuration, we run the policy for 60 trials We find that 60 trials are enough to find a stable average sum of scores to evaluate a policy given a particular game configuration. and compute the average sum of scores. This process is repeated for 1000 games, and we report the average sum across all games. Table 2: Policy generalization in MESSENGER in terms of win rate. Note that other methods (Dynalang, CRL and LED-WM) do not use curriculum training. Results of Dynalang and LED-WM (∗) are rounded to second decimal place, while results for CRL and EMMA are taken from their original papers. Results are recorded across five training seeds. | Dynalang ∗ CRL EMMA (w/o curriculum) | 0.03 ± 0.02 88 ± 2.5 85 ± 1.4 | 0.04 ± 0.05 76 ± 5 45 ± 12 | – – – | 0.03 ± 0.05 32 ± 1.9 10 ± 0.8 | | --- | --- | --- | --- | --- | | EMMA(w/ curriculum) | 88 ± 2.3 | 95 ± 0.4 | – | 22 ± 3.8 | | LED-WM (Ours) ∗ | 100 ± 0 | 51.6 ± 2.7 | 96.6 ± 1.0 | 34.97 ± 1.73 | #### 5.1.3 Results We report the win rate performance of our method and other baselines for MESSENGER in Table ˜ 2 and the average sum of scores for MESSENGER-WM in Table ˜ 3. In MESSENGER-WM, LED-WM outperforms EMMA-LWM in all settings without using any expert demonstrations. In MESSENGER, Dynalang fails to generalize to unseen games. We hypothesize that this is because Dynalang lacks an explicit mechanism to ground language to each entity. Meanwhile, LED-WM is better than other baselines in S1 and comparable to CRL in S3. However, LED-WM underperforms CRL in S2, where the agent is trained on only one movement combination chasing-fleeing-stationary but is evaluated over different unseen movement combinations (unseen dynamics - see Table ˜ 1). In contrast, LED-WM performs well on S2-dev, where its setting is similar to S2, but its test dynamics are the same as the training games. We hypothesize that this occurs because CRL incorporates an explicit mechanism to mitigate the data bias in S2 that there is only one movement combination in the training data and spurious correlations between entity identities and their roles. For instance, the assumption "a dog is always a goal". Therefore, this mechanism might enhance generalization in test scenarios where the dog is either a friend or an enemy. Incorporating such a mechanism in LED-WM might be a promising direction for future work. Table 3: Policy generalization in MESSENGER-WM in terms of average sum of scores. EMMA-LWM results are taken from its original paper [40]. Results are recorded across five training seeds. | EMMA-LWM Online IL Filtered BC (near-optimal) | 1.01 $\pm$ 0.12 1.18 $\pm$ 0.10 | 0.96 $\pm$ 0.17 0.75 $\pm$ 0.20 | 0.62 $\pm$ 0.21 0.44 $\pm$ 0.18 | | --- | --- | --- | --- | | Filtered BC (suboptimal) | 0.98 $\pm$ 0.13 | 0.29 $\pm$ 0.25 | 0.13 $\pm$ 0.19 | | LED-WM (Ours) | 1.31 $\pm$ 0.05 | 1.15 $\pm$ 0.08 | 1.16 $\pm$ 0.02 | ### 5.2 World model generalization To evaluate the generalization of a world model, one pragmatic metric is to measure how its generated rollouts on unseen dynamics benefit policy learning. If the world model can generalize to unseen dynamics in test games, which effectively simulates these dynamics, a policy finetuned on these rollouts should improve in new games. Finetuning procedure. Given a trained LED-WM, a trained policy from LED-WM, and a test game, LED-WM takes the initial observation and the manual as input to generate 60 synthetic trajectories. These trajectories are then used to determine whether the policy should be finetuned on this game. We estimate the value of the trained policy from the world model and finetune the policy if the estimated value is smaller than a pre-defined threshold. For each gradient update on the policy finetune, we generate 60 synthetic trajectories. We repeat this process in 2000 optimization steps. We illustrate the finetuning procedure in Appendix ˜ F with a Python-like format. Table 4: World model generalization over S2-dev and S3 (MESSENGER) through finetune procedure. Finetune results are recorded in average sum of scores across five seeds. | LED-WM (Ours) After finetune | 1.500 $\pm$ 0 - | - - | 1.4478 $\pm$ 0.01 1.4513 $\pm$ 0.01 | –0.11 $\pm$ 0.05 -0.01 $\pm$ 0.12 | | --- | --- | --- | --- | --- | Evaluation metrics and results. We adopt the average sum of scores due to its robustness to the stochasticity of the environment. We show policy finetune results in Table ˜ 4 for MESSENGER. In MESSENGER, we show that the finetuning procedure improves the trained policy in S2-dev, In S2-dev, we use Wilcoxon signed-rank [37] and hierarchical bootstrap sampling [8] corresponding to two levels of hierarchies in our experiments (episodes and run trials): at a 95% confidence level, hierarchical sampling indicates an improvement between 0.014 and 0.019. and in S3, demonstrating that the world model is generalizable to test trajectories. However, the absolute policy improvement is still limited in our experiments. ## 6 Related work Due to space limit, we provide a detailed related work in Appendix ˜ B. In this section, we briefly review related work on language-conditioned dynamics using model-based approach. Recent efforts focus on integrating dynamics-descriptive language into world models, resulting in language-conditioned world models. Dynalang [20] shows that such world model improves policy’s sample efficiency compared to model-free approaches. However, it does not demonstrate policy generalization in unseen games. Reader [6] shows that a MCTS planner can generalize to unseen games using a language-conditioned world model. Despite this, its environment (RTFM [43]) does not require language grounding to entities. Zhang et al. [40] introduce MESSENGER-WM, a compositional benchmark based on MESSENGER, and EMMA-LWM—a policy can generalize over unseen games from a language-based world model and expert demonstrations. Though sharing this same goal of policy generalization with our work, these studies rely on limiting assumptions. Planning with an MCTS tree in Reader, involves incurring computational cost to generate plans in inference time. This approach may not be practical for applications that require quick policy responses. On the other hand, EMMA-LWM requires expert demonstrations to use imitation learning and behavior cloning. This assumption may not always be feasible for every application. In contrast, our work lifts these assumptions and demonstrates policy generalization over unseen games in two environments that require language grounding: MESSENGER and MESSENGER-WM. ## 7 Conclusion We develop an agent that can understand dynamics-descriptive language in interactive tasks. We adopt a model-based reinforcement learning (MBRL) approach, where a language-conditioned world model is trained through interactions with the environment, and a policy is learned from this world model. Unlike existing works, we do not require expert demonstrations or expensive planning during inference. Our method proposes Language-aware Encoder for Dreamer World Model (LED-WM). LED-WM adopts an attention mechanism to explicitly align language description to entities in the observation. We show that policies trained with LED-WM can generalize better to unseen games than existing baselines. We can also further improve the trained policy through fine-tuning on synthetic test trajectories generated by the world model. ## 8 Acknowledgment We thank everyone from VIRL lab (Oregon State University), especially Skand and Akhil for their valuable feedback and discussions. The first author was personally supported by Amanda Putiza, Nguyen Thi Ngoc Anh, Ngo Thi Bich Lan, Tran Thanh Nhu, Bui Thuy Tien, and Nguyen Hoang Kieu Anh. This work is supported by NSF CAREER Award 2339676. We also thank the anonymous reviewers for their valuable feedback and suggestions. ## References - Anderson et al. [2017] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. arXiv [cs.CV], November 2017. - Bisk et al. [2016] Yonatan Bisk, Deniz Yuret, and Daniel Marcu. Natural language communication with robots. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 751–761. Association for Computational Linguistics, June 2016. - Cao et al. [2023] Tianshi Cao, Jingkang Wang, Yining Zhang, and Sivabalan Manivasagam. Zero-shot compositional policy learning via language grounding. arXiv, April 2023. - Cheng et al. [2023] Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. LLF-bench: Benchmark for interactive learning from language feedback. arXiv, December 2023. - Chevalier-Boisvert et al. [2019] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: A platform to study the sample efficiency of grounded language learning. arXiv, December 2019. - Dainese et al. [2023] Nicola Dainese, Pekka Marttinen, and Alexander Ilin. Reader: Model-based language-instructed reinforcement learning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page 16583–16599, Singapore, December 2023. Association for Computational Linguistics. - Dainese et al. [2024] Nicola Dainese, Matteo Merler, Minttu Alakuijala, and Pekka Marttinen. Generating code world models with large language models guided by monte carlo tree search. arXiv, October 2024. - Davison and Hinkley [2013] A C Davison and D V Hinkley. Cambridge series in statistical and probabilistic mathematics: Bootstrap methods and their application series number 1. Cambridge University Press, Cambridge, England, June 2013. - Du et al. [2023] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv, November 2023. - Goyal et al. [2019] Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv, May 2019. - Ha and Schmidhuber [2018] David Ha and Jürgen Schmidhuber. World models. March 2018. - Hafner et al. [2018] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv [cs.LG], November 2018. - Hafner et al. [2024] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models - new 2024. arXiv, April 2024. - Hanjie et al. [2021] Austin W Hanjie, Victor Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. arXiv, June 2021. - Hansen et al. [2023] Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. arXiv, October 2023. - Kaiser et al. [2020] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. arXiv, February 2020. - Krantz and Lee [2022] Jacob Krantz and Stefan Lee. Sim-2-sim transfer for vision-and-language navigation in continuous environments. arXiv, April 2022. - Krantz et al. [2022] Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason. Iterative vision-and-language navigation. arXiv [cs.CV], October 2022. - Krantz et al. [2023] Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason. Iterative vision-and-language navigation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14921–14930, Vancouver, BC, Canada, June 2023. IEEE. - Lin et al. [2023] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. arXiv [cs.CL], July 2023. - Liu et al. [2023] Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv, May 2023. - McCallum et al. [2023] Sabrina McCallum, Max Taylor-Davies, Stefano V Albrecht, and Alessandro Suglia. Is feedback all you need? leveraging natural language feedback in goal-conditioned reinforcement learning. arXiv, December 2023. - Mehta et al. [2024] Nikhil Mehta, Milagro Teruel, Patricio Figueroa Sanz, Xin Deng, Ahmed Hassan Awadallah, and Julia Kiseleva. Improving grounded language understanding in a collaborative environment by interacting with agents through help feedback. arXiv, February 2024. - Nguyen et al. [2022] Khanh Nguyen, Yonatan Bisk, and Hal Daumé, III. A framework for learning to request rich and contextually useful information from humans. arXiv, June 2022. - Parakh et al. [2023] Meenal Parakh, Alisha Fong, Anthony Simeonov, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Lifelong robot learning with human assisted language planners. arXiv, October 2023. - Peng et al. [2023] Shaohui Peng, Xing Hu, Rui Zhang, Jiaming Guo, Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, and Yunji Chen. Conceptual reinforcement learning for language-conditioned tasks. arXiv, March 2023. - Peri et al. [2024] Skand Peri, Iain Lee, Chanho Kim, Li Fuxin, Tucker Hermans, and Stefan Lee. Point cloud models improve visual robustness in robotic learners. arXiv, April 2024. - Piriyakulkij et al. [2025] Wasu Top Piriyakulkij, Yichao Liang, Hao Tang, Adrian Weller, Marta Kryven, and Kevin Ellis. PoE-world: Compositional world modeling with products of programmatic experts. arXiv, May 2025. - Poudel et al. [2023] Rudra P K Poudel, Harit Pandya, Chao Zhang, and Roberto Cipolla. LanGWM: Language grounded world model. arXiv, November 2023. - Raffel et al. [2023] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv, September 2023. - Ren et al. [2023] Yi Ren, Samuel Lavoie, Mikhail Galkin, Danica J Sutherland, and Aaron Courville. Improving compositional generalization using iterated learning and simplicial embeddings. arXiv, October 2023. - Sharma et al. [2022] Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv, April 2022. - Tam et al. [2023] Allison C Tam, Neil C Rabinowitz, Andrew K Lampinen, Nicholas A Roy, Stephanie C Y Chan, D J Strouse, Jane X Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. arXiv, April 2023. - Tang et al. [2024] Hao Tang, Darren Key, and Kevin Ellis. WorldCoder, a model-based LLM agent: Building world models by writing code and interacting with the environment. arXiv, May 2024. - Vaswani et al. [2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv, August 2023. - Wang et al. [2021] Zhengyong Wang, Liquan Shen, Mei Yu, Kun Wang, Yufei Lin, and Mai Xu. Domain adaptation for underwater image enhancement. arXiv, August 2021. - Wilcoxon [1945] Frank Wilcoxon. Individual comparisons by ranking methods. Biom. Bull., 1(6):80–83, 1945. - Xie et al. [2023] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2Reward: Automated dense reward function generation for reinforcement learning. arXiv, September 2023. - Yu et al. [2023] Zhongwei Yu, Jingqing Ruan, and Dengpeng Xing. Explainable reinforcement learning via a causal world model. arXiv, May 2023. - Zhang et al. [2024a] Alex Zhang, Khanh Nguyen, Jens Tuyls, Albert Lin, and Karthik Narasimhan. Language-guided world models: A model-based approach to AI control. arXiv, July 2024a. - Zhang et al. [2024b] Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Zhen Wang, and Xuelong Li. Towards efficient LLM grounding for embodied multi-agent collaboration. arXiv, May 2024b. - Zheng et al. [2023] Zhaoheng Zheng, Haidong Zhu, and Ram Nevatia. CAILA: Concept-aware intra-layer adapters for compositional zero-shot learning. arXiv, May 2023. - Zhong et al. [2021] Victor Zhong, Austin W Hanjie, Sida Wang, Karthik Narasimhan, and Luke Zettlemoyer. SILG: The multi-domain symbolic interactive language grounding benchmark. In Advances in Neural Information Processing Systems, volume 34, page 21505–21519. Curran Associates, Inc., 2021. - Zhong et al. [2022a] Victor Zhong, Austin W Hanjie, Sida I Wang, Karthik Narasimhan, and Luke Zettlemoyer. SILG: The multi-environment symbolic interactive language grounding benchmark. arXiv, January 2022a. - Zhong et al. [2022b] Victor Zhong, Jesse Mu, Luke Zettlemoyer, Edward Grefenstette, and Tim Rocktäschel. Improving policy learning via language dynamics distillation. arXiv, September 2022b. - Zhou et al. [2025] Siyu Zhou, Tianyi Zhou, Yijun Yang, Guodong Long, Deheng Ye, Jing Jiang, and Chengqi Zhang. WALL-E 2.0: World alignment by NeuroSymbolic learning improves world model-based LLM agents. arXiv, April 2025. - Zhou et al. [2024] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. RoboDreamer: Learning compositional world models for robot imagination. arXiv, April 2024. ## Appendix A Training details | Batch size | 30 | | --- | --- | | Batch length | 300 | | Optimizer | Adam | | World model learning rate | 3e-4 | | Max. world model gradient norm | 30 | | Actor learning rate | 2e-4 | | Max. actor gradient norm | 100 | | Critic learning rate | 1e-4 | | Max. critic gradient norm | 100 | Table 5: Training hyperparameters. | Dynamics loss scale | $\beta_{dyn}$ | 1 | | --- | --- | --- | | Representation loss scale | $\beta_{rep}$ | 0.1 | | Latent unimix | — | 1% | | Free nats | — | 1 | | Sentence embedding dim | $d_{s}$ | 32 | | Symbol/Agent embedding dim | $d_{sb}$ | 32 | | MLP layers | — | 3 | | MLP hidden units | — | 512 | | Query/key dim | d | 128 | | Value dim | $d_{val}$ | 128 | | RSSM deterministic dim | — | 512 | Table 6: World model hyperparameters. Other hyperparameters are the same as in DreamerV3 [13]. | Hyperparameter | Symbol | S1 | S2 | S2-dev | S3 | MESSENGER-WM | | --- | --- | --- | --- | --- | --- | --- | | Number of entities | N | 3 | 3 | 3 | 5 | 3 | | Episode horizon | H | 4 | 32 | 32 | 32 | 32 | | Finetune threshold | $thres$ | - | 1.2 | 1.2 | 1.4 | - | | Training environment steps | - | 1M | 10M | 10M | 20M | 10M | | Training GPU hours | - | 6 | 24 | 24 | 72 | 24 | Table 7: Environment hyperparameters. Training GPU hours are estimated based on 1 NVIDIA H100 GPU. ## Appendix B Detailed related work How language is used in RL tasks? Language is often employed as step-by-step instructions or goal specification in domains such as 1) visual language navigation (VLN) [18] [19] [1] 2) grid-world games like BabyAI [5] and SILG benchmark [45], and 3) manipulation tasks [31] [42] [25]. Another research direction in language for RL explores how language can accelerate policy learning by providing richer feedback rather than just numerical rewards: language for plan correction [32] [21] [22] [4] [24] [23], providing more descriptions of the current state or current goal [24] [20], generating dense rewards [10] [39] [38], clarifying information [23], and speeding up exploration [33]. This study investigates an alternative use of language in RL problems: describing the dynamics of environments. Language-conditioned dynamics environments. In language-conditioned environments, while language can be used as step-by-step instructions [17] [2] [1], language can also be used to describe environment dynamnics—that is, how environments change over time. Formally, language describes the transition function $T(s^{\prime}|s,a)$ and reward function $r(s,a)$ of a MDP system defined in Section ˜ 2. Several environments have been proposed to provide language-conditioned dynamics [3] [43] but their settings do not require understanding entity interaction or grounding language to entities. To fill this gap, as discussed in Section ˜ 3 about the environment setup, Hanjie et al. [14] present MESSENGER, a more challenging game requiring language grounding to entities based on environment dynamics. Zhang et al. [40] later propose MESSENGER-WM built out of MESSENGER to test compositional generalization world model and policy. In this study, we focus on MESSENGER and MESSENGER-WM due to their requirements for language grounding to entities and a history of previous works on these datasets. How to solve language-conditioned dynamics environments? Generally speaking, there are two ways to understand language-conditioned dynamics: model-free and model-based learning. First, in model-free approach, language is used to build language-conditioned observation, which is then fed directly to policy in a model-free manner. Second, in model-based learning, language is used to build language-conditioned world model, which is then used to plan or learn policy. Our work focuses on the second category: we use dynamics-related language explicitly for dynamics learning, in model-based RL fashion, thus improving policy performance over model-free approaches. Language-conditioned dynamics in RL: model-free approach. In model-free approach, language is used to build language-conditioned observation, which is then fed directly to policy. Hanjie et al. [14] utilize an attention mechanism to ground language to individual entities, forming a language-aware representation for the policy. Zhong et al. [44] develop a model of environmental dynamics learned from language-conditioned and state-only (without action) demonstrations. This dynamics model is then used to initialize and distill to the representation of a policy learner, which helps sample-efficiency and generalization across language RL tasks. Wang et al. [36] proposes compact and invariant concept-like representations through extracting similarities across observations, which is then proved to be useful for policy learning. Wang et al. [36] proposes two-agent system where the manager agent reads the instructions and manuals to devise the plan with sub-goals and the worker agent fulfills the sub-goals in the plan one-by-one. The model, however, assumes access to the sub-goal text instructions to train the manager. Those works use language directly for policy learning while our work uses language for world model learning. Language-condition instruction-based world model (LWM). World models in model-based reinforcement learning (RL) involve learning the dynamics of the environment. In visual-understanding interaction domains like robotics and video games, world models have been widely studied and are empirically proven to be sample-efficient for policy learning [16] [13] [27]. However, in language-understanding interaction tasks, most language-conditioned world models have been developed to process task instructions or action descriptions, rather than to capture environmental dynamics. Poudel et al. [29] integrates human language into the world model, however language primarily describes observations rather than environment’s dynamics. For example, a description like "there is an apple on the left, 2 meters from here" describes the future observation of environment without addressing the transition function. Recent works [47] [9] [41] develop LWM with compositional generalizability. While these works use more visually realistic input and require more reasoning to solve their tasks, the language they study is task instruction that involves straightforward mapping from language to objects such as colors and object names, e.g. "Move A Red Block to A Brown Box." To bridge this gap, we focus on language-conditioned world models that incorporate dynamics-descriptive language—language that explains how entities interact and how the environment changes—rather than just providing direct task instructions. In contrast, the language used in our testbed environments MESSENGER and MESSENGER-WM describes environmental dynamics, which is the focus of our study. Large language model (LLM) for world model. Recent work [7] [34] [28] [46] use LLMs to build a world model for policy learning. However, their environments does not require language understanding or language grounding. In other words, they do not build a language-conditioned world model like our work. Further, they do not study the generalization behavior of LLM-based world model in a out-of-distribution (OOD) set up. In contrast, our work focuses on building a world model that requires language grounding to entities. We also study the generalization behavior of a language-conditioned world model and a policy learned from this world model. We achieve this by running experiments in a controlled OOD set up from MESSENGER and MESSENGER-WM. ## Appendix C Environment details ### C.1 MESSENGER #### C.1.1 Environment Dynamics and Action Reward and Game Ending. The agent loses the game and incurs -1 reward In the $S3$ setting of the game, there is an inconsistency in the environment implementation with the description provided in [14]: when the agent collides with two enemy entities, the environment returns a reward of -2 instead of -1. We observe that this rarely happens and thus has no significant impact on expected policy’s behavior. if either of two events occurs - the agent is in the same cell as the enemy or reaches the goal without first getting the message. Reaching the messenger gives the agent a reward of 0.5, and then reaching the goal provides a reward of 1. Observation change. The agent can be with or without a message, represented by two different symbols in the observation. The observation changes when the agent interacts with entities according to their roles, specifically: - When the agent without a message picks up messenger, the messenger disappears from the observation. The agent now has a message and is represented by a different symbol from when it was without a message. - When the agent loses the game, either by reaching the goal without first getting the message or by touching the enemy, the agent disappears from the observation. - When the agent wins the game by reaching the goal with the message, the goal disappears from the observation. Action. The agent can navigate the grid using five actions: left, right, up, down, and stay. The agent can only interact with entities when it is in the same grid cell as the entity. We observed an inconsistency in the implementation with the environment description outlined in [14]. The agent can collide with entities even when they are not in the same grid cell. This is however deemed acceptable to the policy as the agent is still able to try to either go back or stay away from the other entity. More details can be found in this discussion: https://github.com/ahjwang/messenger-emma/issues/6 #### C.1.2 Entities and Language Manuals Each game includes a language manual and an observation containing entities and a single agent. There are twelve different entities (e.g., airplane, researcher, etc.) denoted by a fixed set of corresponding symbols that are used consistently across game instances. For instance, symbol <details> <summary>figures/plane.png Details</summary>  ### Visual Description ## Icon: Stylized Airplane Symbol ### Overview The image depicts a minimalist, stylized airplane icon. It features a light blue fuselage with darker blue wings, outlined in black. The wings are positioned diagonally across the body, creating a dynamic, abstract representation. No textual elements, labels, or numerical data are present. ### Components/Axes - **Fuselage**: Central elongated shape in light blue (#ADD8E6), outlined in black. - **Wings**: Two triangular shapes in darker blue (#0000FF), outlined in black, intersecting the fuselage diagonally. - **Tail**: Small triangular extension at the rear of the fuselage, matching the light blue color. - **Background**: Plain white (#FFFFFF), providing high contrast with the black outlines. ### Detailed Analysis - **Color Scheme**: - Fuselage: Light blue (#ADD8E6) - Wings: Dark blue (#0000FF) - Outlines: Black (#000000) - **Design Elements**: - Wings are asymmetrically placed, crossing the fuselage at a 45-degree angle. - No shading, gradients, or additional details (e.g., windows, engines). - Simplified geometric shapes for clarity and recognizability. ### Key Observations - The icon prioritizes simplicity over realism, using bold outlines and flat colors. - The crossed-wing design may symbolize motion or flight dynamics. - No textual or numerical data is embedded in the image. ### Interpretation This icon likely serves as a universal symbol for aviation, travel, or flight-related applications (e.g., apps, websites, signage). The use of contrasting colors (light blue vs. dark blue) enhances visibility, while the black outline ensures scalability for small sizes. The absence of text aligns with standard iconography practices, relying on visual recognition rather than labels. The crossed wings could imply speed or agility, though this is speculative without contextual data. **Note**: The image contains no factual or numerical information. All descriptions are based on visual analysis of design elements and symbolic conventions. </details> for entity plane shown in Figure ˜ 1. Note that the observation does not have entity names (e.g. airplane) and the agent must observe the entity’s symbol and ground the entity’s name to its corresponding symbols from the manual. There are also three movement types for entities: moving, fleeing, and stationary, which describe movement trends relative to the agent’s position. For example, a manual "heading closer and closer to where you are" describes the movement type moving. For each game, the game engine assigns different roles (enemy, goal, messenger) and movement types (moving, fleeing, stationary) to a set of entities, along with the associated language manuals containing this information. For example, "the plane fleeing from you has the classified report". As a result, two games with the same set of entities and identical grid-world observations can have different language manuals and, consequently, different reward and transition functions. #### C.1.3 Evaluation settings MESSENGER provides three stages with different levels of language generalization assessment: Stage 1 (S1). This stage tests the agent’s ability to ground entity names in the manual to entity symbols in the observation. Test games offer two different levels of generalization evaluation. First, new languages describing the same entity name using synonyms, e.g. researcher-scholar. Second, new languages describing new combinations of known entities in a game, i.e. the agent has played with entities ferry, plane, researcher in train but not in the same game, and the agent is tasked to play with all of them in a test game. As shown in Figure ˜ 3, this stage includes three entities, each with one of the three roles: enemy, messenger, and goal, along with their corresponding descriptions. All entities are stationary and placed two steps away from the agent, which starts in the center of the grid. The language descriptions only specify the entities and their roles, with no mention of movement. The agent starts the game either with or without the message. <details> <summary>figures/stage_one.png Details</summary>  ### Visual Description ## Diagram: Grid-Based Scenario with Manual Instructions ### Overview The image depicts a 10x10 grid (100 cells) with three labeled icons and a text box titled "Manual." The grid contains: - A **person with a microscope** (researcher) in cell (3,5) - A **ferry** in cell (5,2) - An **airplane** in cell (5,7) The "Manual" box contains three bullet points describing relationships between these elements. ### Components/Axes - **Grid Structure**: - 10 rows (top to bottom) and 10 columns (left to right) - No axis labels or scales provided - **Icons**: - **Person with microscope**: Represents a researcher (cell 3,5) - **Ferry**: Labeled as "deadly adversary" (cell 5,2) - **Airplane**: Contains "classified report" (cell 5,7) - **Manual Text**: - Three bullet points: 1. "The ferry is a deadly adversary." 2. "The plane has the classified report." 3. "The researcher is a vital goal." ### Detailed Analysis - **Icon Placement**: - The researcher (microscope) is isolated in row 3, column 5. - The ferry (adversary) and airplane (classified report) occupy the same row (row 5) but different columns (2 and 7). - **Textual Relationships**: - The ferry’s adversarial role is explicitly stated. - The airplane’s "classified report" implies critical information. - The researcher is framed as a "vital goal," suggesting their importance to the scenario. ### Key Observations 1. **Spatial Proximity**: The ferry and airplane share row 5, potentially indicating a shared objective or conflict. 2. **Researcher’s Role**: Positioned separately, the researcher’s "vital goal" status contrasts with the adversarial ferry and informational airplane. 3. **Manual Directives**: The text box acts as a narrative guide, linking the icons to a story (e.g., a mission to retrieve classified data from the plane while evading the ferry). ### Interpretation The diagram likely represents a strategic scenario, such as a covert operation or investigative task. The researcher’s role as a "vital goal" suggests they are either the target of the adversarial ferry or the key to resolving the conflict involving the classified report. The ferry’s placement in row 5, column 2, and the airplane’s position in row 5, column 7, may imply a spatial relationship (e.g., the ferry blocking access to the airplane). The manual’s directives emphasize urgency and adversarial dynamics, framing the researcher as central to navigating these challenges. ### Notable Patterns - **Contrast**: The researcher’s isolation vs. the grouped ferry/airplane. - **Narrative Flow**: The manual’s bullet points create a cause-effect chain (adversary → classified data → researcher’s importance). - **Ambiguity**: No explicit instructions on movement or interaction between elements, leaving interpretation open to context. </details> Figure 3: An example game of MESSENGER S1. In this game, the entity does not have message at the beginning of the game. Therefore, it goes to the messenger to retrieve the message and ends the game. All entities except the agent are stationary, thus the manual only describes roles associated with entity names. <details> <summary>figures/stage_two.png Details</summary>  ### Visual Description ## Grid-Based Game Mechanics Diagram ### Overview The image depicts a 10x10 grid-based game environment with interactive elements and movement rules. The left side shows a spatial layout with icons representing game entities, while the right side provides textual instructions. The diagram combines visual and textual components to explain gameplay mechanics. ### Components/Axes - **Grid Layout**: - 10x10 grid with no explicit axis labels. - Coordinates implied by row/column positions (e.g., top-left = (1,1), bottom-right = (10,10)). - **Icons and Entities**: - **Researcher**: Icon of a person with a magnifying glass (top-right quadrant). - **Planes**: Three airplane icons (middle-left quadrant), with one plane highlighted in darker blue. - **Robot**: Simple robot icon (bottom-left quadrant). - **Arrows**: Dashed black lines indicating movement paths. - **Textual Instructions**: - Three bullet points on the right: 1. "The plane going away from you carries out a message." 2. "The researcher doesn’t move is final goal." 3. "The ferry chasing you is an enemy." ### Detailed Analysis - **Researcher Path**: - Dashed line originates from the researcher icon (top-right) and extends downward, suggesting a restricted movement path. - No movement allowed for the researcher, as per the second instruction. - **Plane Dynamics**: - Three planes are aligned horizontally in the middle-left quadrant. - One plane (darkest blue) is positioned farther from the researcher, implying it is "going away" to "carry out a message." - **Robot Interaction**: - Robot icon is connected via a dashed line to the planes, suggesting a relationship (e.g., control or dependency). - **Ferry Absence**: - No ferry icon is visible in the grid, but the third instruction warns of an enemy ferry, indicating it may appear dynamically or off-grid. ### Key Observations 1. **Immutable Researcher**: The researcher’s fixed position is critical to gameplay, as stated in the manual. 2. **Plane Movement**: The departing plane’s role as a messenger introduces a time-sensitive objective. 3. **Robot-Plane Link**: The dashed line between the robot and planes implies the robot may influence plane behavior. 4. **Enemy Ferry**: The ferry’s absence in the grid suggests it may spawn dynamically or operate outside the visible area. ### Interpretation The diagram outlines a strategic game where the player must: - **Protect the Researcher**: Ensure the researcher remains stationary despite external threats (e.g., the ferry). - **Coordinate Planes**: Use the departing plane to send messages, possibly to the robot or other entities. - **Anticipate Threats**: The ferry’s presence as an enemy implies dynamic obstacles that require evasion or countermeasures. The grid’s sparse layout emphasizes minimalism, focusing player attention on critical elements. The lack of a legend forces reliance on iconography and textual instructions, which may test the player’s ability to infer relationships (e.g., robot-plane interactions). The ferry’s implied threat adds urgency, contrasting with the researcher’s static role. </details> Figure 4: An example of a game play within a 10 × 10 grid-world from MESSENGER S2. The observation on the left includes three entities represented by their associated symbols: (ferry - <details> <summary>figures/ferry.png Details</summary>  ### Visual Description ## Icon: Stylized Boat on Water ### Overview The image depicts a minimalist, abstract representation of a boat floating on water. The design uses geometric shapes and a limited color palette to convey the subject. No textual elements, numerical data, or explicit labels are present. ### Components/Axes - **Boat Structure**: - **Hull**: A large, dark blue (hex: #00008B) trapezoidal shape representing the boat’s base. - **Cabin**: A smaller, light blue (hex: #ADD8E6) rectangular structure atop the hull, with a white (hex: #FFFFFF) rectangular superstructure. - **Mast**: A vertical white line extending from the cabin, suggesting a sail or antenna. - **Water**: A wavy, light blue (hex: #87CEEB) band at the bottom, simulating ocean waves. ### Detailed Analysis - **Color Usage**: - Dark blue dominates the hull, symbolizing depth or stability. - Light blue and white for the cabin and mast imply openness or functionality. - **Shape Simplification**: - The hull’s trapezoidal form abstracts the boat’s structure, prioritizing recognition over realism. - The cabin’s rectangular shape contrasts with the hull’s angularity, creating visual hierarchy. - **Water Representation**: - Wavy lines in light blue mimic motion, though no explicit wave frequency or amplitude is quantified. ### Key Observations - No textual labels, legends, or numerical values are present. - The design adheres to a flat, iconographic style, avoiding gradients or shading. - The boat’s orientation is horizontal, with the mast aligned vertically. ### Interpretation This icon likely serves as a universal symbol for maritime themes, navigation, or transportation. The absence of text or data suggests it is intended for broad, non-technical audiences. The color choices and geometric abstraction prioritize clarity and scalability, making it suitable for digital interfaces or signage. No outliers or anomalies exist, as the design is intentionally minimalist. **Note**: The image contains no factual or numerical data. The description above is based solely on visual analysis of design elements and symbolic representation. </details> ), (plane - <details> <summary>figures/plane.png Details</summary>  ### Visual Description ## Icon: Stylized Airplane Symbol ### Overview The image depicts a minimalist, stylized airplane icon. It features a light blue fuselage with darker blue wings, outlined in black. The wings are positioned diagonally across the body, creating a dynamic, abstract representation. No textual elements, labels, or numerical data are present. ### Components/Axes - **Fuselage**: Central elongated shape in light blue (#ADD8E6), outlined in black. - **Wings**: Two triangular shapes in darker blue (#0000FF), outlined in black, intersecting the fuselage diagonally. - **Tail**: Small triangular extension at the rear of the fuselage, matching the light blue color. - **Background**: Plain white (#FFFFFF), providing high contrast with the black outlines. ### Detailed Analysis - **Color Scheme**: - Fuselage: Light blue (#ADD8E6) - Wings: Dark blue (#0000FF) - Outlines: Black (#000000) - **Design Elements**: - Wings are asymmetrically placed, crossing the fuselage at a 45-degree angle. - No shading, gradients, or additional details (e.g., windows, engines). - Simplified geometric shapes for clarity and recognizability. ### Key Observations - The icon prioritizes simplicity over realism, using bold outlines and flat colors. - The crossed-wing design may symbolize motion or flight dynamics. - No textual or numerical data is embedded in the image. ### Interpretation This icon likely serves as a universal symbol for aviation, travel, or flight-related applications (e.g., apps, websites, signage). The use of contrasting colors (light blue vs. dark blue) enhances visibility, while the black outline ensures scalability for small sizes. The absence of text aligns with standard iconography practices, relying on visual recognition rather than labels. The crossed wings could imply speed or agility, though this is speculative without contextual data. **Note**: The image contains no factual or numerical information. All descriptions are based on visual analysis of design elements and symbolic conventions. </details> ), (researcher - <details> <summary>figures/scientist.png Details</summary>  ### Visual Description ## Icon/Symbol: Cartoon Scientist Using Microscope ### Overview The image depicts a stylized, cartoon-like illustration of a person engaged in scientific observation using a microscope. The figure is shown in profile, facing left, with a focus on their interaction with the microscope. No textual labels, numerical data, or explicit annotations are present. ### Components/Axes - **Primary Subject**: A humanoid figure with curly brown hair, wearing a light gray lab coat and blue safety goggles. - **Microscope**: A simplified, stylized microscope with a gray body, yellow arm, and black eyepiece. - **Background**: Plain white, with no additional contextual elements. - **Color Palette**: - Brown (hair) - Gray (lab coat, microscope body) - Blue (safety goggles) - Yellow (microscope arm) - Black (eyepiece, outlines) ### Detailed Analysis - **Figure Posture**: The figure leans forward, with their right hand positioned near the microscope’s stage, suggesting active observation. - **Microscope Design**: The microscope is rendered with minimal detail, emphasizing its functional components (eyepiece, arm, stage) rather than realism. - **Absence of Text**: No labels, legends, or textual annotations are visible in the image. ### Key Observations 1. The illustration prioritizes symbolic representation over realism, using bold outlines and flat colors. 2. The lack of textual elements suggests the image is intended for generic use (e.g., educational materials, icons) rather than data visualization. 3. The figure’s gender is ambiguous due to the absence of facial features beyond hair and goggles. ### Interpretation This icon likely represents a generic scientist or researcher in a laboratory setting. The absence of specific identifiers (e.g., race, gender, institutional branding) implies it is designed for broad applicability. The use of a microscope underscores themes of scientific inquiry, experimentation, or analysis. The simplicity of the design suggests it could serve as a visual metaphor for education, discovery, or technical work in fields like biology, chemistry, or materials science. **Note**: No factual or numerical data is present in the image. The description focuses on symbolic and design elements only. </details> ) and one agent (depicted by <details> <summary>figures/bot.png Details</summary>  ### Visual Description ## Icon: Robot Symbol ### Overview The image depicts a minimalist, black-and-white line-drawn robot icon. It features a square body, two circular eyes, a circular antenna atop the head, and two rounded handles on the sides. No textual elements, numerical data, or color gradients are present. ### Components/Axes - **Body**: A centrally positioned square with rounded corners. - **Eyes**: Two identical black circles with white centers, placed symmetrically in the upper half of the body. - **Antenna**: A single vertical line extending upward from the center of the body, terminating in a small black circle. - **Handles**: Two identical rounded rectangles attached to the left and right sides of the body, resembling arms. - **Color**: Pure black lines on a white background. ### Detailed Analysis - No numerical values, labels, or legends are visible. - The design adheres to a flat, simplistic style typical of modern UI icons. - The antenna’s position (top-center) and handles’ placement (symmetrical sides) suggest a humanoid robot archetype. ### Key Observations - The absence of text or data points indicates the icon is purely symbolic, not data-driven. - The use of circles for eyes and antenna aligns with common anthropomorphic robot representations. - Symmetry in design elements (eyes, handles) emphasizes balance and simplicity. ### Interpretation This icon likely serves as a visual shorthand for robotics, AI, or automation in user interfaces, educational materials, or branding. Its simplicity prioritizes recognizability over detail, making it versatile for small-scale applications (e.g., app icons, signage). The lack of complexity suggests it is not intended for technical schematics but rather for general symbolic use. </details> ). The game involves three roles: messenger, goal, and enemy. The agent’s task is to identify roles of all entities, locate the messenger, deliver it to the goal, and avoid the enemy. To achieve this objective, the agent must use the manual to infer entity roles based on their described dynamics and observed behavior. In the observation in the example, shaded icons indicate one possible scenario of entity locations over time. By observing entity movement patterns and grounding language to entities based on their according behaviors, the agent can infer the roles are assigned: (ferry-enemy), (plane-messenger), and (researcher-goal). After inferring all entity roles, the agent can execute an appropriate plan to complete the task. The dashed line in the observation shows such a possible plan. Stage 2 (S2) and S2-dev. As shown in Figure ˜ 4 This figure is the duplication of Figure 1 and is put here for the sake of reading flow., S2 uses the same set of entities as S1 but introduces movement dynamics: entities can now exhibit one of three movement types: moving, fleeing, or stationary. The agent always starts without the message. During training games, only one movement combination is used: one moving (chasing), one fleeing, and one stationary entity, all of which describe how entities are moving compared to the agent. In test games, the agent must handle scenarios where a movement type can appear multiple times, e.g. moving-moving-fleeing. To examine the impact of this single-movement constraint, MESSENGER provides a different stage S2-dev, a variation of S2 that also features unseen dynamics but maintains the same movement constraint observed during training for all test games. In addition to the capabilities demonstrated in S1, the objective of the agent in S2 is to generalize across new language featuring novel environmental dynamics. Specifically, the agent must understand the movement descriptions to make optimal actions, but does not need to ground movement descriptions to the entities based on their observed behaviors. This is because the agent can ground the sentences to the entities based on their names in the manual and their associated symbols in the observation. For example, given the game in Figure ˜ 1, the agent can ground the sentence "The plane fleeing from you has the classified report" to entity symbol <details> <summary>figures/plane.png Details</summary>  ### Visual Description ## Icon: Stylized Airplane Symbol ### Overview The image depicts a minimalist, stylized airplane icon. It features a light blue fuselage with darker blue wings, outlined in black. The wings are positioned diagonally across the body, creating a dynamic, abstract representation. No textual elements, labels, or numerical data are present. ### Components/Axes - **Fuselage**: Central elongated shape in light blue (#ADD8E6), outlined in black. - **Wings**: Two triangular shapes in darker blue (#0000FF), outlined in black, intersecting the fuselage diagonally. - **Tail**: Small triangular extension at the rear of the fuselage, matching the light blue color. - **Background**: Plain white (#FFFFFF), providing high contrast with the black outlines. ### Detailed Analysis - **Color Scheme**: - Fuselage: Light blue (#ADD8E6) - Wings: Dark blue (#0000FF) - Outlines: Black (#000000) - **Design Elements**: - Wings are asymmetrically placed, crossing the fuselage at a 45-degree angle. - No shading, gradients, or additional details (e.g., windows, engines). - Simplified geometric shapes for clarity and recognizability. ### Key Observations - The icon prioritizes simplicity over realism, using bold outlines and flat colors. - The crossed-wing design may symbolize motion or flight dynamics. - No textual or numerical data is embedded in the image. ### Interpretation This icon likely serves as a universal symbol for aviation, travel, or flight-related applications (e.g., apps, websites, signage). The use of contrasting colors (light blue vs. dark blue) enhances visibility, while the black outline ensures scalability for small sizes. The absence of text aligns with standard iconography practices, relying on visual recognition rather than labels. The crossed wings could imply speed or agility, though this is speculative without contextual data. **Note**: The image contains no factual or numerical information. All descriptions are based on visual analysis of design elements and symbolic conventions. </details> based on the entity name the plane -to-symbol <details> <summary>figures/plane.png Details</summary>  ### Visual Description ## Icon: Stylized Airplane Symbol ### Overview The image depicts a minimalist, stylized airplane icon. It features a light blue fuselage with darker blue wings, outlined in black. The wings are positioned diagonally across the body, creating a dynamic, abstract representation. No textual elements, labels, or numerical data are present. ### Components/Axes - **Fuselage**: Central elongated shape in light blue (#ADD8E6), outlined in black. - **Wings**: Two triangular shapes in darker blue (#0000FF), outlined in black, intersecting the fuselage diagonally. - **Tail**: Small triangular extension at the rear of the fuselage, matching the light blue color. - **Background**: Plain white (#FFFFFF), providing high contrast with the black outlines. ### Detailed Analysis - **Color Scheme**: - Fuselage: Light blue (#ADD8E6) - Wings: Dark blue (#0000FF) - Outlines: Black (#000000) - **Design Elements**: - Wings are asymmetrically placed, crossing the fuselage at a 45-degree angle. - No shading, gradients, or additional details (e.g., windows, engines). - Simplified geometric shapes for clarity and recognizability. ### Key Observations - The icon prioritizes simplicity over realism, using bold outlines and flat colors. - The crossed-wing design may symbolize motion or flight dynamics. - No textual or numerical data is embedded in the image. ### Interpretation This icon likely serves as a universal symbol for aviation, travel, or flight-related applications (e.g., apps, websites, signage). The use of contrasting colors (light blue vs. dark blue) enhances visibility, while the black outline ensures scalability for small sizes. The absence of text aligns with standard iconography practices, relying on visual recognition rather than labels. The crossed wings could imply speed or agility, though this is speculative without contextual data. **Note**: The image contains no factual or numerical information. All descriptions are based on visual analysis of design elements and symbolic conventions. </details> mapping. The agent must understand the entity’s behavior to move closer to it, and it can achieve this based on the description "The plane fleeing from you", even without directly observing the behavior. <details> <summary>figures/stage_three.png Details</summary>  ### Visual Description ## Diagram: Scenario Analysis Grid with Manual Instructions ### Overview The image contains two primary sections: a grid labeled "Observation" and a list labeled "Manual." The grid uses icons (e.g., people with binoculars, boats, airplanes, robots) and dashed lines to represent relationships or movements. The "Manual" section lists six bullet points describing scenarios involving boats, ferries, planes, and researchers. ### Components/Axes - **Grid (Observation Section)**: - **Structure**: 8 rows × 8 columns (total 64 cells). - **Icons**: - **People with binoculars**: 3 instances (top-left, middle-left, top-right). - **Boats**: 2 instances (middle-left, middle-right). - **Airplanes**: 4 instances (bottom-left, bottom-middle, bottom-right, middle-right). - **Robot**: 1 instance (middle-left). - **Dashed Lines**: - A vertical dashed line connects the top-right binocular icon to the middle-right airplane. - A horizontal dashed line connects the middle-left robot to the bottom-middle airplane. - **Labels**: - "Observation" (top-left corner). - **Manual Section**: - **Text**: Six bullet points describing scenarios: 1. *The boat going away from you and it is a goal.* 2. *The ferry which is approaching you is a deadly adversary.* 3. *The plane fleeing from you has the classified report.* 4. *The airplane chasing you is a deadly enemy.* 5. *The researcher won’t budge and it is a vital goal.* 6. *The researcher coming closer to you and it is an enemy.* ### Detailed Analysis - **Grid**: - The grid appears to map spatial relationships or movements of entities (e.g., boats, airplanes, researchers). - Dashed lines suggest directional or causal connections (e.g., a researcher’s movement influencing an airplane’s path). - No numerical data or axis labels are present; the grid is purely symbolic. - **Manual**: - The bullet points define rules or objectives for interpreting the grid. - Each bullet assigns a "goal," "adversary," or "enemy" status to specific entities based on their actions (e.g., "approaching," "fleeing," "chasing"). ### Key Observations - The grid and manual are interdependent: the grid visualizes entities, while the manual provides context for their roles. - The robot icon is isolated in the grid but not referenced in the manual, suggesting it may be a neutral or undefined element. - The dashed lines imply dynamic interactions (e.g., a researcher’s movement affecting an airplane’s trajectory). ### Interpretation This diagram likely represents a decision-making or scenario analysis framework. The grid maps entities and their spatial relationships, while the manual defines their roles (e.g., "goal," "adversary"). For example: - A boat moving away is a "goal," implying it is desirable. - A ferry approaching is a "deadly adversary," suggesting it poses a threat. - A plane fleeing with a "classified report" indicates it is a critical asset. - Researchers are framed as either "vital goals" (if stationary) or "enemies" (if approaching). The absence of numerical data or explicit labels for the grid’s axes suggests it is a conceptual or symbolic representation rather than a quantitative analysis. The manual’s rules may guide users in interpreting the grid’s symbolic elements in a specific context (e.g., strategy, risk assessment, or game mechanics). </details> Figure 5: An example game of MESSENGER S3. To win the game, the agent must infer the roles of entities given the manual. Specifically, the same entity names (e.g. airplane, plane with different roles (e.g. enemy, messenger) must be disambiguated by their movement dynamics (e.g. chasing, fleeing). Note that we have a italicized sentence describing an extraneous entity that is not available in the game observation. We also have synonyms for entity names and roles, e.g., airplane, plane; adversary, enemy. The shaded entities show possible entity locations over time and the dashed line shows a possible path for the agent to win the game. Stage 3 (S3). In addition to the capabilities demonstrated in Stage 1, the objective of the agent in Stage 3 is to generalize over new language featuring new combinations of known entity movement dynamics. Unlike in Stage 2, the agent in S3 must ground the sentences using both entity name-to-symbol mappings and observed entity behavior-to-movement description mappings. As shown in Figure ˜ 5, this stage includes five entities, three retain the roles of enemy, messenger, and goal. The manual has six sentences featuring these five entities and one extraneous entity, which is not available in the observation. Specifically, its referred sentence has the same name as the enemy entity, but is different in movement and is described as either goal or messenger. Two additional entities are duplicates that share the same entity symbols and names of the messenger and goal accordingly, but they are assigned the role of enemy. To differentiate these entities, their movement dynamics must be used. For example, descriptions like "the fleeing enemy is the dog" versus "the chasing goal is the dog" help the agent identify the correct entities based on their behaviors. In this case, the dog that is consistently going towards the agent can be inferred as the goal. ### C.2 Examples of different level of generalization evaluation in MESSENGER and MESSENGER-WM environment We illustrate different levels of generalizations, described in Table ˜ 1, by examples in Figure ˜ 6. For each type of generalization, for the sake of simplicity, we consider a hypothetical dataset where a training set consists of only two samples, and one test sample. The best view is in colors. <details> <summary>figures/surface.png Details</summary>  ### Visual Description ## Textual Analysis: Training Manuals and Unseen Manual ### Overview The image presents three distinct sections labeled **Training Manual 1**, **Training Manual 2**, and **Unseen manual**, each containing three bullet-pointed statements. Key terms are highlighted in gray, indicating potential emphasis or contextual differences. ### Components/Axes - **Sections**: 1. **Training Manual 1** 2. **Training Manual 2** 3. **Unseen manual** - **Bullet Points**: Each section contains three statements with varying terminology. - **Highlighted Terms**: - "deadly adversary" (Training Manual 1, Training Manual 2) - "enemy" (Unseen manual) ### Detailed Analysis #### Training Manual 1 1. **The ferry is a deadly adversary.** - Highlighted term: "deadly adversary" 2. **The plane has the classified report.** 3. **The researcher is a vital goal.** #### Training Manual 2 1. **The dog is a deadly adversary.** - Highlighted term: "deadly adversary" 2. **The wizard has the classified report.** 3. **The ferry is a goal.** #### Unseen manual 1. **The ferry is a messenger.** 2. **The dog is an enemy.** - Highlighted term: "enemy" 3. **The researcher is a vital goal.** ### Key Observations 1. **Terminology Shifts**: - "Deadly adversary" (Training Manuals 1 and 2) vs. "enemy" (Unseen manual) for the dog. - The ferry’s role changes from "deadly adversary" (Training Manual 1) to "goal" (Training Manual 2) to "messenger" (Unseen manual). 2. **Consistency**: - "The researcher is a vital goal" appears unchanged across all sections. 3. **Contextual Ambiguity**: - The "classified report" is attributed to different entities (plane in Training Manual 1, wizard in Training Manual 2). ### Interpretation The highlighted terms ("deadly adversary," "enemy") suggest a progression or variation in threat perception across manuals. The ferry’s evolving role (adversary → goal → messenger) implies shifting objectives or scenarios. The consistent labeling of the researcher as a "vital goal" underscores its central importance. The Unseen manual’s deviation in terminology ("enemy" instead of "deadly adversary") may indicate a distinct operational context or updated protocol. This analysis highlights how minor lexical changes can alter perceived relationships between entities, potentially affecting training outcomes or strategic decisions. Further investigation into the rationale behind these shifts is warranted. </details> (a) Novel language through synonyms and paraphrase <details> <summary>figures/novelcombo.png Details</summary>  ### Visual Description ## Textual Analysis: Training Manuals Comparison ### Overview The image presents three side-by-side textual comparisons labeled "Training Manual 1," "Training Manual 2," and "Unseen manual." Each section contains three bullet-point statements with specific words highlighted in distinct colors (gray, green, blue). The structure suggests a comparative analysis of entity classification across different contexts. ### Components/Axes - **Sections**: - Training Manual 1 (left) - Training Manual 2 (center) - Unseen manual (right) - **Highlight Colors**: - Gray: "ferry" (Training Manual 1, Unseen manual) - Green: "dog" (Training Manual 2, Unseen manual) - Blue: "researcher" (Training Manual 1, Unseen manual) ### Content Details #### Training Manual 1 1. "The ferry is a deadly adversary." - Highlight: "ferry" (gray) 2. "The plane has the classified report" - No highlights 3. "The researcher is a vital goal." - Highlight: "researcher" (blue) #### Training Manual 2 1. "The dog is a deadly adversary." - Highlight: "dog" (green) 2. "The wizard has the classified report" - No highlights 3. "The ferry is a goal." - Highlight: "ferry" (blue) #### Unseen manual 1. "The ferry is a messenger." - Highlight: "ferry" (gray) 2. "The dog is an enemy." - Highlight: "dog" (green) 3. "The researcher is a vital goal." - Highlight: "researcher" (blue) ### Key Observations 1. **Entity Consistency**: - "ferry" appears in all three manuals, with conflicting roles (adversary → goal → messenger). - "dog" appears in Training Manual 2 and Unseen manual, with roles shifting from "adversary" to "enemy." - "researcher" maintains consistent labeling as "vital goal" across Training Manual 1 and Unseen manual. 2. **Color Coding Patterns**: - Gray highlights ("ferry") appear in contexts of conflict ("adversary") and neutral roles ("messenger"). - Green highlights ("dog") denote adversarial entities in both Training Manual 2 and Unseen manual. - Blue highlights ("researcher") consistently associate with goal-oriented objectives. 3. **Contextual Shifts**: - Training Manual 1 and 2 establish baseline classifications. - Unseen manual introduces new roles for previously defined entities, testing contextual adaptability. ### Interpretation This structure resembles a natural language processing (NLP) training exercise for entity disambiguation or contextual classification. The manuals likely simulate scenarios where: - **Training Manuals 1-2** establish foundational entity roles (e.g., "ferry" as adversary, "dog" as enemy). - **Unseen manual** challenges the system to reconcile conflicting roles (e.g., "ferry" as both adversary and messenger) while maintaining consistency for stable entities like "researcher." The color-coded highlights suggest a focus on specific nouns as critical variables in context-dependent classification tasks. The repetition of "researcher" as a goal across manuals implies it serves as an anchor entity for stability in dynamic scenarios. </details> (b) Novel combinations of known entities. In test games, entities never appear in the same game during training. <details> <summary>figures/novel_entity_role.png Details</summary>  ### Visual Description ## Textual Analysis: Training Manuals Comparison ### Overview The image presents three side-by-side textual blocks labeled "Training Manual 1," "Training Manual 2," and "Unseen manual." Each block contains three bullet-point statements, with specific phrases highlighted in yellow to emphasize key terms. The content appears to describe adversarial relationships, goals, and roles of entities (e.g., ferry, dog, researcher) across different training scenarios. ### Components/Axes - **Training Manual 1**: - "The ferry is a deadly adversary." (highlighted) - "The plane has the classified report." - "The researcher is a vital goal." - **Training Manual 2**: - "The dog is a deadly adversary." - "The wizard has the classified report." - "The ferry is a goal." (highlighted) - **Unseen manual**: - "The ferry is a messenger." (highlighted) - "The dog is an enemy." - "The researcher is a vital goal." ### Detailed Analysis 1. **Training Manual 1**: - The ferry is explicitly labeled a "deadly adversary," contrasting with its role as a "goal" in Manual 2. - The plane is associated with a "classified report," while the researcher is a "vital goal." 2. **Training Manual 2**: - The dog replaces the ferry as the "deadly adversary," and the wizard holds the "classified report." - The ferry is redefined as a "goal," conflicting with its adversarial role in Manual 1. 3. **Unseen manual**: - The ferry is described as a "messenger," introducing a new role not seen in the training manuals. - The dog is labeled an "enemy," aligning thematically with the adversarial roles in the training manuals. - The researcher remains a "vital goal" across all manuals, suggesting consistency in this objective. ### Key Observations - **Ferry Role Conflict**: The ferry’s role shifts dramatically between manuals (adversary → goal → messenger), indicating potential contradictions or evolving scenarios. - **Adversarial Entities**: The "deadly adversary" label applies to the ferry (Manual 1) and dog (Manual 2), while the Unseen manual introduces the dog as an "enemy." - **Consistency in Researcher**: The researcher is universally a "vital goal," emphasizing its importance across all scenarios. - **Classified Reports**: The plane (Manual 1) and wizard (Manual 2) hold classified reports, but no such entity is mentioned in the Unseen manual. ### Interpretation The manuals appear to simulate adversarial environments with conflicting objectives: - **Training Manuals 1 and 2** establish a framework where entities (ferry, dog) are adversaries, and goals (researcher) are prioritized. - The **Unseen manual** introduces ambiguity by redefining the ferry as a "messenger" and the dog as an "enemy," suggesting a shift in context or a test of adaptability. - The recurring "vital goal" for the researcher implies it is a constant objective, while other roles (ferry, dog) are context-dependent. - The highlighted phrases ("deadly adversary," "goal," "messenger") may represent critical decision points or keywords for training algorithms to prioritize. This structure highlights inconsistencies in entity roles, potentially testing how systems resolve conflicting information or adapt to new scenarios (e.g., the Unseen manual’s "messenger" role for the ferry). The adversarial relationships and goal prioritization could reflect a simulation of dynamic threat assessment or resource allocation in a technical or operational context. </details> (c) Novel entity-role combinations. In test games, at least one entity-role combination is unseen during training. <details> <summary>figures/novel_entity_role_move.png Details</summary>  ### Visual Description ## Screenshot: Training Manuals Content ### Overview The image displays three training manuals with bullet points, each containing text with specific sections highlighted in blue, green, and yellow. The manuals are labeled as **Training Manual 1**, **Training Manual 2**, and **Unseen manual**. The content appears to define relationships between entities (plane, scientist, ship) and their roles (message, goal, enemy, messenger). ### Components/Axes - **Training Manual 1**: - **Bullet 1**: - Text: "The plane going away from you has a message" - Highlights: - Blue: "The plane going away from you has" - Green: "a message" - **Bullet 2**: - Text: "The scientist doesn't move is final goal you go to" - Highlights: - Green: "final goal you go to" - **Bullet 3**: - Text: "The ship chasing you is an enemy" - Highlights: - Yellow: "an enemy" - **Training Manual 2**: - **Bullet 1**: - Text: "The plane that doesn't move is the final goal" - Highlights: - Blue: "The plane" - Green: "final goal" - **Bullet 2**: - Text: "The scientist moving to you is a messenger" - Highlights: - Blue: "The scientist" - **Bullet 3**: - Text: "The ship chasing you is an enemy" - Highlights: - Yellow: "an enemy" - **Unseen manual**: - **Bullet 1**: - Text: "The plane going away from you is an enemy" - Highlights: - Blue: "The plane going away from you" - Yellow: "an enemy" - **Bullet 2**: - Text: "The scientist going away from you is final goal you go to" - Highlights: - Blue: "The scientist going away from you" - Green: "final goal you go to" - **Bullet 3**: - Text: "The ship chasing you is a messenger" - Highlights: - Blue: "The ship chasing you" ### Key Observations 1. **Highlighted Patterns**: - **Blue**: Often marks the subject/entity (e.g., "The plane," "The scientist"). - **Green**: Denotes goals or messages (e.g., "final goal," "a message"). - **Yellow**: Indicates adversarial roles (e.g., "an enemy"). 2. **Contradictions Across Manuals**: - The **ship chasing you** is labeled an **enemy** in Training Manuals 1 and 2 but a **messenger** in the Unseen manual. - The **plane** transitions from a "message" (Training Manual 1) to a "final goal" (Training Manual 2) and an **enemy** (Unseen manual). - The **scientist** shifts from a static goal (Training Manual 1) to a messenger (Training Manual 2) and a dynamic goal (Unseen manual). 3. **Structural Consistency**: - All manuals follow a three-bullet-point structure, but the roles of entities vary significantly between manuals. ### Interpretation The manuals likely represent scenarios for training in a dynamic environment where entities (plane, scientist, ship) have context-dependent roles. - **Training Manuals 1 and 2** establish baseline roles: - Planes deliver messages or represent goals. - Scientists are static objectives or messengers. - Ships are adversaries. - The **Unseen manual** introduces variability, suggesting scenarios where roles are inverted or redefined (e.g., ships as messengers, planes as enemies). This could simulate unpredictable conditions or adversarial adaptations. The use of color-coded highlights emphasizes critical elements for decision-making, possibly guiding users to prioritize entities (blue), objectives (green), and threats (yellow) in real-time scenarios. The contradictions in the Unseen manual imply a need for adaptive strategies when encountering unexpected role assignments. </details> (d) Novel entity - role - movement combinations. In test games, at least one entity-role-movement combination is unseen during training. <details> <summary>figures/novel_dynamic.png Details</summary>  ### Visual Description ## Textual Analysis: Training Manuals Comparison ### Overview The image presents three side-by-side sections labeled "Training Manual 1," "Training Manual 2," and "Unseen manual." Each section contains three bullet points with color-highlighted phrases (yellow, green, blue) embedded in sentences. The structure suggests a comparison of contextual interpretations or semantic roles across different scenarios. ### Components/Axes - **Sections**: - Training Manual 1 (left) - Training Manual 2 (center) - Unseen manual (right) - **Highlight Colors**: - **Yellow**: "going away from you" (appears in all three sections) - **Green**: "doesn't move" (Training Manual 1 and 2) and "going away from you" (Unseen manual) - **Blue**: "chasing you" (Training Manual 1 and Unseen manual) and "moving to you" (Training Manual 2) ### Detailed Analysis #### Training Manual 1 1. **Yellow Highlight**: "The plane **going away from you** has a message" - Context: Plane as a messenger. 2. **Green Highlight**: "The wizard **doesn't move** is final goal you go to" - Context: Wizard as a static endpoint. 3. **Blue Highlight**: "The ship **chasing you** is an enemy" - Context: Ship as an active threat. #### Training Manual 2 1. **Yellow Highlight**: "The ship **going away from you** is an enemy" - Context: Ship as a retreating threat. 2. **Green Highlight**: "The plane that **doesn't move** is the final goal" - Context: Plane as a static endpoint. 3. **Blue Highlight**: "The scientist **moving to you** is a messenger" - Context: Scientist as a purposeful communicator. #### Unseen manual 1. **Yellow Highlight**: "The plane **going away from you** is an enemy" - Context: Plane as a hostile entity. 2. **Green Highlight**: "The wizard **going away from you** is final goal you go to" - Context: Wizard as a dynamic endpoint. 3. **Blue Highlight**: "The ship **chasing you** is a messenger" - Context: Ship as a purposeful communicator. ### Key Observations 1. **Color Consistency**: - Yellow consistently highlights phrases involving spatial movement ("going away from you"). - Green highlights static or endpoint-related phrases ("doesn't move," "final goal"). - Blue highlights active movement toward or away from the subject ("chasing you," "moving to you"). 2. **Contextual Shifts**: - The same phrase ("going away from you") is labeled as "message," "enemy," or "final goal" depending on the manual. - "Chasing you" and "moving to you" are both blue but assigned opposing roles (enemy vs. messenger). ### Interpretation The color-coded highlights suggest a framework for categorizing entities based on their semantic roles (e.g., messenger, enemy, goal). However, the same phrase ("going away from you") is reinterpreted across manuals, indicating that context (e.g., "plane," "wizard," "ship") overrides literal meaning. This implies a training exercise in disambiguating ambiguous language or testing contextual awareness. The Unseen manual introduces contradictions (e.g., "chasing you" as a messenger), hinting at potential errors or deliberate ambiguity in the training data. ## Conclusion The image demonstrates how identical phrases can carry divergent meanings depending on contextual cues. The color-coding system aids in categorizing roles but reveals inconsistencies when applied across different scenarios, emphasizing the need for nuanced interpretation in technical documentation or natural language processing tasks. </details> (e) Novel combinations of known entity movements or novel game dynamics. In training games, there is only one movement combination chasing - fleeing - stationary. In test games, there are more than one movement combination. Figure 6: Examples of different levels of generalization evaluation in MESSENGER and MESSENGER-WM environments. ### C.3 MESSENGER-WM details #### C.3.1 Differences between MESSENGER and MESSENGER-WM Similar to MESSENGER S2, in MESSENGER-WM, there are three entities with three roles: messenger, enemy, and goal. However, unlike S2 where training games only have one movement combination chasing-fleeing-stationary, training games in MESSENGER-WM can have more than one entity having the same movement pattern, e.g. chasing-chasing-stationary. This makes generalization in S2 is more challenging than MESSENGER-WM because the agent must overcome this data bias to generalize over unseen combinations of movement patterns. #### C.3.2 Evaluation settings To help ground our descriptions of these settings, consider the following manual for a hypothetical test game: The hound is a deadly opponent. It is towards you. The whale comes towards you as the secret document. It also has the crucial goal, the queen, and is something that cannot be moved. This manual describes the entity combination: hound, whale, queen, with the following feature assignments: (hound-chasing-enemy), (whale-chasing-messenger), (queen-stationary-goal). Descriptions of each setting are as follows, based on whether the test game falls into each: 1. NewCombo: Each game represents an unseen combination of entities. However, any entity-role-movement combination in this set also presents in the training games. In this example, the agent never sees entity combination (hound, whale, queen) in the same game during training, although it can see each entity individually across different games. 1. NewAttr: Each game features seen combinations of entities, but at least one attribute (role, movement type, or both) for each entity is novel. In is example, the agent has seen entity combination (hound, whale, queen) during training but each entity-role-movement assignment is new: i.e. the assignment (hound-chasing-enemy) is unseen but (hound-chasing-goal) or (hound-fleeing-messenger) are seen during training. 1. NewAll: This setting combines the challenges of the first two. The combination of entities is novel, and each entity is assigned at least one new attribute. In this example, entity combinations hound, whale, queen and all entity-role-movement (i.e. (hound-chasing-enemy), (whale-chasing-messenger), (queen-stationary-goal)) are unseen. ## Appendix D LED-WM details Our world model LED-WM is built on the Recurrent State Space Model (RSSM) in [13]. However, we make the following modifications. First, we find that reconstruction decoder in DreamerV3 negatively impacts policy generalization. Therefore, we omit the decoder from DreamerV3 architecture. Second, to improve sample efficiency, we adopt multi-step prediction for reward and continue prediction [15] [27]. Specifically, we rollout the latent states in the future for $H$ steps to supervise reward and continue prediction. We replace the encoder of DreamerV3 with our LED encoder, and keep the rest of the architecture unchanged, resulting in the following components: $$ \displaystyle\raisebox{22.0pt}{ $\text{RSSM}~~\begin{cases}\hphantom{A}\\[-5.0pt] \hphantom{A}\\[-5.0pt] \hphantom{A}\end{cases}$}\begin{aligned} &\text{Sequence model:}&\quad h_{t}&=f_{\phi}(h_{t-1},z_{t-1},a_{t-1})\\ &\text{{LED} (\lx@cref{creftype~refnum}{sec:led}):}&\quad z_{t}&\sim q_{\phi}(z_{t}\mid h_{t},x_{t})\\ &\text{Dynamics predictor:}&\quad\hat{z}_{t}&\sim p_{\phi}(\hat{z}_{t}\mid h_{t})\\ &\text{Reward predictor:}&\quad\hat{r}_{t}&\sim p_{\phi}(\hat{r}_{t}\mid h_{t},z_{t})\\ &\text{Continue predictor:}&\quad\hat{c}_{t}&\sim p_{\phi}(\hat{c}_{t}\mid h_{t},z_{t})\\ \\ \end{aligned} \tag{8} $$ World model loss. Given a sequence of observations $o_{1:T}$ , actions $a_{1:T}$ , rewards $r_{1:T}$ and continuation flags $c_{1:T}$ where $T$ is the horizon of a training episode, we optimize the world model parameters $\phi$ to minimize the following loss: $$ \displaystyle\mathcal{L}(\phi)\doteq\mathbb{E}_{q_{\phi}}\left[\sum_{t=1}^{T}\left(\beta_{\text{pred}}\mathcal{L}_{\text{pred}}(\phi)+\beta_{\text{dyn}}\mathcal{L}_{\text{dyn}}(\phi)+\beta_{\text{rep}}\mathcal{L}_{\text{rep}}(\phi)\right)\right], \tag{9} $$ where $\mathcal{L}_{pred}$ , $\mathcal{L}_{dyn}$ , and $\mathcal{L}_{rep}$ are prediction loss, dynamics loss, and representation loss, along with their corresponding weights $\beta_{pred}$ , $\beta_{dyn}$ and $\beta_{rep}$ : 1. Prediction loss ( $\mathcal{L}_{pred}$ ): trains the reward predictor via symlog loss and the continue predictor via binary classification loss. To address the issue of slower training caused by removing the decoder and observational reconstruction loss, we adopt multi-step prediction for reward and continue [15] [27], to improve sample efficiency. We rollout the latent states in the future for $H$ steps to supervise reward and continue prediction. Specifically, given a state-action trajectory over $H+1$ step $(x_{t},a_{t},x_{t+1},\ldots,x_{t+H})$ associated with a sequence of rewards $r_{t:t+H}$ and continue flags $c_{t:t+H}$ , we first compute $z_{t}$ as the posterior state from $x_{t}$ . We then rollout this $z_{t}$ over $H$ steps to get prior states $\hat{z}_{t+1...t+H-1}$ and deterministic states $h_{t+1...t+H}$ to predict rewards $r_{t+1:t+H}$ and continue flags $c_{t:t+H}$ : $$ \displaystyle\mathcal{L}_{pred} \displaystyle=\underbrace{-\ln p_{\theta}(r_{t}|z_{t},h_{t})}_{\text{reward loss}}\underbrace{-\ln p_{\theta}(c_{t}|z_{t},h_{t})}_{\text{continue loss}} \displaystyle\quad\underbrace{-\sum_{k=t+1}^{H}\lambda^{k-t-1}\left[\ln p_{\theta}(r_{k}\mid\hat{z}_{k},h_{k})+\ln p_{\theta}(c_{k}\mid\hat{z}_{k},h_{k})\right],}_{\text{multi-step reward and continue loss}} \tag{10} $$ where $\lambda=0.9$ is a discount factor when the environment is stochastic and $\lambda=1$ when the environment is deterministic. Recall that $\hat{z}_{k}$ denotes the prior stochastic state generated by the world model, without access to observation at time step $k$ . 1. Dynamics and representation loss: We adopt the dynamics and representation loss unchanged from Dreamerv3. ## Appendix E Training procedure During world model training, we observe two challenges. First, at the beginning of training, successful episodes in which the agent wins the game and receives positive rewards are rare. As a result, the world model takes longer to learn from these rare instances, leading to reduced sample-efficiency in policy training. Second, as the policy converges and produces mostly successful episodes, the replay buffer becomes dominated by these episodes. This causes the world model to rarely encounter failed episodes where the agent loses the game and receives negative rewards, potentially harming its generalization performance. We therefore adapt the following strategies to improve sample efficiency: Prioritized Replay Buffer. We adopt the Prioritized Replay Buffer from [Kauvar2023-pk], where the authors propose the following prioritization strategies: - Count-based replay: Biases sampling towards recent experiences in the replay buffer. - Adversarial replay: Prioritizes experiences where the world model makes incorrect predictions. Balanced weights. We adopt a balanced weighting technique for handling class imbalance, inspired by methods used in classification tasks [He2009-bj], and apply it to world model training. This weighting ensures that underrepresented classes contribute proportionally more to the training loss, improving sample-efficiency of the policy. In our setting, for a given episode $e$ with $T$ is the episode horizon and the state-action trajectory: $\tau_{e}=(o_{1},a_{1},\ldots,o_{T},a_{T})$ , we define the "class" $c_{e}$ as the accumulated sum of rewards for the episode: $$ c_{e}=\sum_{t=1}^{T}r_{t}(o_{t},a_{t}), \tag{12} $$ representing different gameplay scenarios. For example, in the MESSENGER environment, episodes fall into three classes: 1.5, -0.5, and -1. To address the imbalance between training instances of negative classes and positive class in the replay buffer, we scale the world model loss $\mathcal{L}_{\text{pred}}$ in each class proportional to the inverse square root of its frequency in the replay buffer. The scaled loss is computed as: $$ \mathcal{L}_{\text{pred}}=\mathcal{L}_{\text{pred}}\times\sqrt{\frac{|RB|}{\text{count}(c)}}, \tag{13} $$ where $|RB|$ is the total number of episodes in the replay buffer $RB$ , and $\text{count}(c)$ is the number of instances of class $c$ in the replay buffer. Increase throughput for replay buffer. In the original Dreamerv3 implementation, one trajectory with $L$ time steps of observations in the replay buffer is duplicated $L$ times, making the training data throughput inefficient. We therefore remove this duplication to speed up the training throughput in the replay buffer, resulting in more sample-efficient. ## Appendix F Finetune a trained policy using a trained world model Input: The trained LED-WM, the trained policy $\pi$ , a test game $G$ with the first observation $obs_{o}$ and a language manual $L$ . Output: A finetuned policy $\hat{\pi}$ if needed 0.8em Function EstimateReturn ( $\pi$ , LED-WM, $obs_{o}$ , $L$ ): returns = [] for _ in range(60): // Generate synthetic test trajectories traj = LED-WM.GenerateTrajectory( $obs_{o}$ , $L$ ) returns.append(sum_rewards(traj)) $\hat{V}_{\pi}$ = mean(returns) return $\hat{V}_{\pi}$ Function FineTune ( $\pi$ , LED-WM, $obs_{o}$ , $L$ ): for gradient_step in range(2000): trajectories = [] for _ in range(60): trajectories.append(LED-WM.GenerateTrajectory( $obs_{o}$ , $L$ )) $\pi$ .train(trajectories) return $\pi$ 0.8em // Main function: Finetune the policy $\pi$ using LED-WM Function PolicyFinetune(LED-WM, $\pi$ , $obs_{0}$ , L): $\hat{V}_{\pi}$ = EstimateReturn(LED-WM, $\pi$ , $obs_{0}$ , $L$ ) if $\hat{V}_{\pi}>=thres$ : $\hat{\pi}=\pi$ else: $\hat{\pi}$ = FineTune( $\pi$ , LED-WM, $G$ , $obs_{0}$ , $L$ ) return $\hat{\pi}$ Algorithm 1 Policy Finetune with LED-WM ## NeurIPS Paper Checklist 1. Claims 1. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? 1. Answer: [Yes] 1. Justification: Our claim about policy generalization and world model generalization in Section ˜ 1 are reflected in our experiment results in Section ˜ 5. 1. Guidelines: - The answer NA means that the abstract and introduction do not include the claims made in the paper. - The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. - The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. - It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 1. Limitations 1. Question: Does the paper discuss the limitations of the work performed by the authors? 1. Answer: [Yes] 1. Justification: We discuss our limitations in policy generalization where CRL [26] outperforms LED-WM in S2 in Section ˜ 5. 1. Guidelines: - The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. - The authors are encouraged to create a separate "Limitations" section in their paper. - The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. - The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. - The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. - The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. - If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. - While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 1. Theory assumptions and proofs 1. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 1. Answer: [Yes] 1. Justification: In Section ˜ 1, we stated that our assumption is that our observation is symbolic and confied to a discrete grid-world. 1. Guidelines: - The answer NA means that the paper does not include theoretical results. - All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. - All assumptions should be clearly stated or referenced in the statement of any theorems. - The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. - Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. - Theorems and Lemmas that the proof relies upon should be properly referenced. 1. Experimental result reproducibility 1. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? 1. Answer: [Yes] 1. Justification: The paper provides the code and hyperparameters for training the world model and the policy in Section ˜ 5 and Appendix ˜ A. 1. Guidelines: - The answer NA means that the paper does not include experiments. - If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. - If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. - Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. - While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example 1. If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. 1. If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. 1. If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 1. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 1. Open access to data and code 1. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? 1. Answer: [Yes] 1. Justification: We provide the code and hyperparameters for training the world model and the policy in Section ˜ 5 and Appendix ˜ A. 1. Guidelines: - The answer NA means that paper does not include experiments requiring code. - Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. - While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). - The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. - The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. - The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. - At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). - Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 1. Experimental setting/details 1. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? 1. Answer: [Yes] 1. Justification: We adopt MESSENGER and MESSENGER-WM environments which follow their standard train/dev/test split. We provide our code and necessary hyperparameters in Appendix ˜ A. 1. Guidelines: - The answer NA means that the paper does not include experiments. - The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. - The full details can be provided either with the code, in appendix, or as supplemental material. 1. Experiment statistical significance 1. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? 1. Answer: [Yes] 1. Justification: We use Wilcoxon signed-rank [37], bootstrap sampling [Efron1979-wz], and hierarchical bootstrap sampling [8] to do statistical tests for policy finetune results in Table ˜ 4 in S2-dev. 1. Guidelines: - The answer NA means that the paper does not include experiments. - The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. - The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). - The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) - The assumptions made should be given (e.g., Normally distributed errors). - It should be clear whether the error bar is the standard deviation or the standard error of the mean. - It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. - For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). - If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 1. Experiments compute resources 1. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? 1. Answer: [Yes] 1. Justification: We provide training time and our used GPU information in Table ˜ 7. 1. Guidelines: - The answer NA means that the paper does not include experiments. - The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. - The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. - The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 1. Code of ethics 1. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 1. Answer: [Yes] 1. Justification: The authors have reviewed the NeurIPS Code of Ethics and the paper conforms to the code of ethics. 1. Guidelines: - The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. - If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. - The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 1. Broader impacts 1. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 1. Answer: [N/A] 1. Justification: We do not have any potential positive or negative societal impacts. 1. Guidelines: - The answer NA means that there is no societal impact of the work performed. - If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. - Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. - The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. - The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. - If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 1. Safeguards 1. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 1. Answer: [N/A] 1. Justification: We do not have any data or models that have a high risk for misuse. 1. Guidelines: - The answer NA means that the paper poses no such risks. - Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. - Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. - We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 1. Licenses for existing assets 1. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? 1. Answer: [Yes] 1. Justification: We cited the original papers for the environments and other baselines. 1. Guidelines: - The answer NA means that the paper does not use existing assets. - The authors should cite the original paper that produced the code package or dataset. - The authors should state which version of the asset is used and, if possible, include a URL. - The name of the license (e.g., CC-BY 4.0) should be included for each asset. - For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. - If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. - For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. - If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 1. New assets 1. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 1. Answer: [Yes] 1. Justification: We will release the code and instructionsfor our experiments. 1. Guidelines: - The answer NA means that the paper does not release new assets. - Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. - The paper should discuss whether and how consent was obtained from people whose asset is used. - At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 1. Crowdsourcing and research with human subjects 1. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 1. Answer: [N/A] 1. Justification: We do not involve crowdsourcing nor research with human subjects. 1. Guidelines: - The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. - Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. - According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 1. Institutional review board (IRB) approvals or equivalent for research with human subjects 1. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 1. Answer: [N/A] 1. Justification: We do not involve research with human subjects. 1. Guidelines: - The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. - Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. - We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. - For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 1. Declaration of LLM usage 1. Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required. 1. Answer: [N/A] 1. Justification: we only use LLM for editing purposes. 1. Guidelines: - The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. - Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.