# World-in-World: World Models in a Closed-Loop World
**Authors**:
- Jiahan Zhang
- Muqing Jiang
- Nanru Dai
- Taiming Lu
- Arda Uzunoglu
- Shunchi Zhang
- Yana Wei
- Jiahao Wang
- Vishal M. Patel
- Paul Pu Liang
- Daniel Khashabi
- Cheng Peng
- Rama Chellappa
- Tianmin Shu
- Alan Yuille
- Yilun Du
- Jieneng Chen (JHU PKU Princeton MIT Harvard)
Abstract
Generative world models (WMs) can now simulate worlds with striking visual realism, which naturally raises the question of whether they can endow embodied agents with predictive perception for decision making. Progress on this question has been limited by fragmented evaluation: most existing benchmarks adopt open-loop protocols that emphasize visual quality in isolation, leaving the core issue of embodied utility unresolved, i.e., do WMs actually help agents succeed at embodied tasks? To address this gap, we introduce World-in-World, the first open platform that benchmarks WMs in a closed-loop world that mirrors real agent-environment interactions. World-in-World provides a unified online planning strategy and a standardized action API, enabling heterogeneous WMs for decision making. We curate four closed-loop environments that rigorously evaluate diverse WMs, prioritize task success as the primary metric, and move beyond the common focus on visual quality; we also present the first data scaling law for world models in embodied settings. Our study uncovers three surprises: (1) visual quality alone does not guarantee task success—controllability matters more; (2) scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and (3) allocating more inference-time compute allows WMs to substantially improve closed-loop performance. Code will be available at github.com/World-In-World. footnotetext: Correspondence to “jienengchen01@gmail.com”. We warmly welcome contributions to the open benchmark.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Video Models and Embodied Success
### Overview
The image presents a diagram exploring the relationship between video models and embodied success. It outlines a system called "World-In-World," an open platform, and examines how different video models perform in various tasks. The diagram includes a leaderboard ranking models by embodied success, not visual fidelity, and correlation plots showing the relationship between task success and factors like seen examples, inference count, and visual quality.
### Components/Axes
* **Title:** "Do Better Video Models Lead to Higher Embodied Success?"
* **Central System:** "World-In-World" - described as "an open platform."
* **Embodied Policy:** A central component connecting "Unified Action" and "Closed-Loop Interaction."
* **SoTA video models:**
* Wan
* Cosmos
* SVD
* Runway
* LTX
* Hunyuan
* **Tasks:**
* Active Recognition (image of a room)
* Navigation (image of a floor plan)
* Question Answering (image of a kitchen)
* Manipulation (image of a robot arm manipulating colored blocks)
* **Leaderboard:** Ranks world models by embodied success.
* Columns: Rank, Model Name, Task Success, OSN Quality
* **Correlation Plots:** Three plots showing the relationship between Task Success and:
* Seen examples (Data Scaling)
* Inference count (Inference-time Scaling)
* Visual quality (Correlation)
### Detailed Analysis
**1. World-In-World System:**
* The diagram illustrates a flow where video models (SoTA video models) contribute to "Unified Action."
* "Unified Action" and "Closed-Loop Interaction" are connected via "Embodied Policy."
* The system supports tasks like Active Recognition, Navigation, Question Answering, and Manipulation.
**2. Leaderboard:**
The leaderboard ranks three video models based on "Task Success" and "OSN Quality."
| Rank | Model Name | Task Success | OSN Quality |
|------|------------------------------|--------------|-------------|
| 1 | Wan2.1 Alladin-Video Generation | 62.6% | 0.380 |
| 2 | SVD Stability AI - Video Generation | 61.0% | 0.363 |
| 3 | Cosmos Nvidia - Video Generation | 80.3% | 0.360 |
**3. Correlation Plots:**
* **Data Scaling (Seen examples):**
* The plot shows "Task success" vs. "Seen examples."
* The Wan model (represented by its logo) has a higher task success than the SVD model (represented by the letter "S"). Both show a positive correlation.
* **Inference-time Scaling (Inference count):**
* The plot shows "Task success" vs. "Inference count."
* The Wan model has a higher task success than the SVD model. Both show a positive correlation.
* **Correlation (Visual quality):**
* The plot shows "Task success" vs. "Visual quality."
* The plot shows the Wan model, SVD model, Cosmos model, Runway model, and LTX model.
* There is no clear linear correlation visible.
### Key Observations
* The "World-In-World" system aims to evaluate video models based on their embodied success in various tasks.
* The leaderboard emphasizes that the ranking is based on embodied success, not visual quality.
* The correlation plots suggest a positive relationship between task success and both "Seen examples" and "Inference count."
* The correlation between "Task success" and "Visual quality" is less clear.
### Interpretation
The diagram suggests that embodied success, as measured by performance in tasks like active recognition, navigation, question answering, and manipulation, is a key metric for evaluating video models. The "World-In-World" platform provides a framework for assessing these models. The positive correlation between task success and "Seen examples" and "Inference count" indicates that models benefit from more data and longer inference times. The lack of a clear correlation between task success and visual quality challenges the assumption that visually impressive models are necessarily more successful in embodied tasks. This implies that other factors, such as the model's ability to understand and interact with the environment, are more critical for embodied success. The leaderboard highlights specific models that perform well in this context, providing a benchmark for future research and development.
</details>
Figure 1: We introduce the first open benchmark to evaluate world models by closed-loop task success, analyze the link between task success and visual quality, and investigate scaling laws.
1 Introduction
Recent advances in visual generation have sparked interest in world generation, a field focused on the creation of diverse environments populated with varied scenes and entities, with applications in entertainment, gaming, simulation, and embodied AI. The rapid progress in video generation (Brooks et al., 2024b; Yang et al., 2024b; Wan et al., 2025), 3D scene generation (Fridman et al., 2023; Chung et al., 2023; Yu et al., 2024; Koh et al., 2023; Ling et al., 2025), and 4D scene generation (Bahmani et al., 2024b; Xu et al., 2024; Bahmani et al., 2024a) has demonstrated high-quality individual scene generation, highlighting the potential of these models as world generation systems.
Building on these developments, recent world generation systems (Yang et al., 2023b; Parker-Holder and Fruchter, 2025; Li et al., 2025b; Ye et al., 2025; Lu et al., 2025; He et al., 2025c) show promise as world models for embodied agents. Given an agent’s initial observation and a candidate action, such systems predict the resulting video, thereby estimating the future state of the environment. These action-conditioned simulators mirror human mental models by forecasting future states and can provide missing context under partial observability. As a result, they offer a pathway to improved decision-making for embodied tasks that rely on perception, planning, and control.
Despite this promise, the community lacks a unified benchmark that evaluates visual world models through the lens of embodied interaction. Existing suites emphasize video generation quality (e.g., VBench (Huang et al., 2024)) or visual plausibility (e.g., WorldModelBench (Li et al., 2025a)). The recent WorldScore (Duan et al., 2025) offers a unified assessment for models that take an image and a camera trajectory as input. However, no current benchmark tests whether generated worlds actually enhance embodied reasoning and task performance —for example, helping an agent perceive the environment, plan and execute actions, and replan based on new observations within such a closed loop. Establishing this evaluation framework is essential for tracking genuine progress across the rapidly expanding landscape of visual world models and embodied AI.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Scatter Plot: Task Success Rate vs. General Quality
### Overview
The image is a scatter plot comparing the Task Success Rate (in percentage) against the General Quality (Aesthetic + Image Quality) for various models. The plot categorizes the models into three groups: Zero-shot, Post-trained, and Others, represented by different colored markers.
### Components/Axes
* **Title:** None explicitly present in the image.
* **X-axis:** "Gen. Quality (Aesthetic+Image Quality) ↑". The axis ranges from approximately 0.325 to 0.475, with gridlines at intervals of 0.025.
* **Y-axis:** "Task Success Rate (%)". The axis ranges from 55% to 63%, with gridlines at intervals of 1%.
* **Legend:** Located in the top-right corner.
* Red circle: "Zero-shot"
* Blue circle: "Post-trained"
* Green square: "Others"
### Detailed Analysis
The scatter plot shows the distribution of different models based on their general quality and task success rate.
* **Zero-shot (Red circles):**
* "Wan2.1†": Located at approximately (0.375, 62.5).
* "Wan2.2 A14B": Located at approximately (0.43, 59).
* "Wan2.1": Located at approximately (0.47, 58.3).
* "Hunyuan": Located at approximately (0.40, 58).
* "Wan2.2 5B": Located at approximately (0.41, 55.3).
* "Cosmos-P2": Located at approximately (0.47, 55.2).
* **Post-trained (Blue circles):**
* "Wan2.1†": Located at approximately (0.35, 61).
* "SVD†": Located at approximately (0.35, 60.5).
* "Cosmos-P2†": Located at approximately (0.35, 60).
* "LTXVideo†": Located at approximately (0.35, 57.5).
* "Wan2.2 5B†": Located at approximately (0.37, 56.2).
* "LTXVideo": Located at approximately (0.40, 56).
* **Others (Green squares):**
* "NWM": Located at approximately (0.32, 57.4).
* "SE3DS": Located at approximately (0.38, 57.3).
* "Pathdreamer": Located at approximately (0.36, 57).
### Key Observations
* The "Wan2.1†" model (Zero-shot) has the highest task success rate among all models.
* The "Cosmos-P2" model (Zero-shot) has the lowest task success rate among all models.
* The "NWM" model (Others) has the lowest general quality among all models.
* There appears to be a positive correlation between general quality and task success rate, although it is not very strong.
### Interpretation
The scatter plot visualizes the performance of different models in terms of task success rate and general quality. The models are categorized into Zero-shot, Post-trained, and Others, allowing for a comparison of their performance within and across these categories. The plot suggests that, in general, models with higher general quality tend to have higher task success rates, but there are exceptions. For example, some Zero-shot models have high task success rates despite having relatively lower general quality. The plot can be used to identify models that perform well in both categories and to understand the trade-offs between general quality and task success rate for different types of models.
</details>
Figure 2: Task success rate vs. generation quality. $\dagger$ : post-trained with extra data. We defend that world models live and die by their closed-loop success, not flawless generated visuals.
In this work, we address this gap by proposing World-in-World, which wraps generative World models In a closed-loop World interface to measure their practical utility for embodied agents. Specifically, we present a unified strategy for closed-loop online planning and a standardized action API to seamlessly integrate diverse world models into closed-loop tasks. The online planning strategy allows the agent to look ahead by anticipating environmental changes and task rewards before committing to an action. The standardized action API harmonizes input modalities expected by different world models, so that each model can be controlled consistently within the same evaluation protocol. In addition, we introduce a post-training protocol that fine-tunes pretrained video generators using a modest amount of action–observation data drawn from the same action space as the downstream tasks, which allows us to examine their adaptation potential and to characterize a data scaling law.
World-in-World offers a fair, closed-loop world interface to evaluate diverse WMs. We benchmark leading video generators (Wan et al., 2025; HaCohen et al., 2024; Kong et al., 2024) alongside task-focused world models (Bar et al., 2025a; Koh et al., 2023, 2021a) in perception, navigation, and manipulation settings. Our findings reveal three consistent trends: (1) high visual quality does not necessarily translate into strong task success; (2) scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and (3) increasing inference-time compute via online planning substantially improves closed-loop performance. As shown in Figure ˜ 2, world models with strong visual scores do not necessarily bring high success rates, which underscores the need for closed-loop evaluation when judging WM practical value for embodied agents.
Our work makes three main contributions:
- We introduce World-in-World, the first comprehensive closed-loop benchmark that evaluates world models through the lens of embodied interaction, moving beyond the common focus on generation quality.
- We propose a unified closed-loop planning strategy with a unified action API, allowing diverse world models to be seamlessly integrated and evaluated within a single framework across four embodied tasks.
- We discover that high visual quality does not necessarily guarantee task success, and demonstrate how the performance of pretrained video generators can be substantially improved through training-time data scaling and inference-time scaling.
2 World-in-World: a Closed-Loop Interface for Visual World Models
Design overview. Our goal is to establish a benchmark that evaluates world-generation methods by their utility for embodied agents. Unlike prior work focused on generative quality, we develop a predictive-control framework to test how well a world model supports online decision-making. The evaluation setting mirrors practical scenarios in embodied AI, emphasizing the interaction between prediction, control, and reward under closed-loop operation.
We detail the unified strategy for closed-loop online planning (Section ˜ 2.1) and the unified action API (Section ˜ 2.2), which together provide a common interface across tasks and models. We then describe our task selection and evaluation protocol (Section ˜ 2.3). Finally, we present a post-training recipe that adapts a pretrained video generator into a more effective embodied world model (Section ˜ 2.4).
<details>
<summary>x3.png Details</summary>
### Visual Description
## Diagram: Closed-loop Online Planning
### Overview
The image presents a diagram illustrating a closed-loop online planning process, likely for a robotic system interacting with an environment. The diagram outlines the flow of information and actions between different components, including imagined interactions, real interactions, action proposal, action API, world model, and revision.
### Components/Axes
The diagram consists of the following key components:
* **Imagined Interactions → Real Interactions:** A sequence of images labeled O1, O2, O3, ..., Ot, representing visual inputs or states. A purple arrow indicates the flow from imagined to real interactions.
* **Embodied Task Env.:** A dashed box containing icons representing different aspects of the environment (e.g., map, drone, robotic arm, chatbot) and the Earth. Arrows indicate interaction between the environment and the robot.
* **Robot:** A simple robot icon.
* **Actions:** A sequence of arrows representing actions, labeled D*1, D*2, D*3, ..., D*t. The arrows indicate left, left, up, and up actions.
* **1. π proposal:** A neural network icon, labeled "π proposal".
* **Candidate Action Plans:** A list of candidate action plans: Candidate Action Plan 1 Ât(1), Candidate Action Plan 2 Ât(2), ..., Candidate Action Plan M Ât(M).
* **2. Unified Action API:** A rounded square containing an icon with arrows pointing in four directions and the letter "A" in the center.
* **Text/Camera traj/Actions:** A vertical list of icons representing text, camera trajectory, and actions.
* **3. World Model gθ:** A globe icon, labeled "World Model gθ".
* **Possible Future States:** A list of possible future states: Possible Future State 1 Ôt(1), Possible Future State 2 Ôt(2), ..., Possible Future State M Ôt(M).
* **4. π revision:** An icon representing a document with a gear, labeled "π revision".
### Detailed Analysis
1. **Imagined Interactions → Real Interactions:**
* A sequence of images, labeled O1, O2, O3, ..., Ot.
* The images are generic landscape icons.
* A purple arrow indicates the flow from imagined to real interactions.
2. **Embodied Task Env.:**
* A dashed box containing icons representing different aspects of the environment.
* Icons include a map, a drone, a robotic arm, a chatbot, and the Earth.
* Arrows indicate interaction between the environment and the robot.
3. **Robot:**
* A simple robot icon.
4. **Actions:**
* A sequence of arrows representing actions, labeled D*1, D*2, D*3, ..., D*t.
* The arrows indicate left, left, up, and up actions.
5. **1. π proposal:**
* A neural network icon, labeled "π proposal".
* The icon represents a multi-layered neural network.
6. **Candidate Action Plans:**
* A list of candidate action plans: Candidate Action Plan 1 Ât(1), Candidate Action Plan 2 Ât(2), ..., Candidate Action Plan M Ât(M).
* The action plans are generated by the "π proposal" neural network.
7. **2. Unified Action API:**
* A rounded square containing an icon with arrows pointing in four directions and the letter "A" in the center.
* The API likely provides a standardized interface for controlling the robot.
8. **Text/Camera traj/Actions:**
* A vertical list of icons representing text, camera trajectory, and actions.
* These are likely different modalities of actions or information that the robot can use.
9. **3. World Model gθ:**
* A globe icon, labeled "World Model gθ".
* The world model represents the robot's understanding of the environment.
10. **Possible Future States:**
* A list of possible future states: Possible Future State 1 Ôt(1), Possible Future State 2 Ôt(2), ..., Possible Future State M Ôt(M).
* These states are predicted by the world model based on the candidate action plans.
11. **4. π revision:**
* An icon representing a document with a gear, labeled "π revision".
* This component likely revises the action plans based on the predicted future states.
### Key Observations
* The diagram illustrates a closed-loop planning process, where the robot interacts with the environment, proposes actions, predicts future states, and revises its plans based on these predictions.
* The process involves multiple components, including a neural network for action proposal, a unified action API, a world model, and a revision mechanism.
* The diagram highlights the importance of both imagined interactions (simulations) and real interactions in the planning process.
### Interpretation
The diagram depicts a sophisticated approach to robot planning, where the robot uses a world model to predict the consequences of its actions and revises its plans accordingly. This closed-loop approach allows the robot to adapt to changing environments and achieve its goals more effectively. The use of a neural network for action proposal suggests that the robot can learn to generate effective action plans from experience. The unified action API provides a standardized interface for controlling the robot, which simplifies the integration of different components. The inclusion of both imagined and real interactions suggests that the robot can learn from both simulation and real-world experience.
</details>
Figure 3: Closed-loop online planning in World-in-World: At time step $t$ , the agent receives the world state, represented by observation $\mathbf{o}_{t}$ , and invokes a proposal policy $\pi_{\text{proposal}}$ (❶) to produce a total of $M$ candidate action plans. The unified action API (❷) transforms each plan into the control inputs required by the world model. The world model (❸) then predicts the corresponding future states as observations $\hat{\mathbf{O}}_{t}$ . The revision policy $\pi_{\text{revision}}$ (❹) evaluates all rollouts and commits to the best, yielding decision $\mathbf{D}^{\star}_{t}$ . This decision is applied in the environment, closing the interaction loop.
2.1 Unified Strategy for Closed-Loop Online Planning
In Figure ˜ 3, we present a unified closed-loop strategy that uses visual world models for decision-making. It cycles through proposal, simulation, and revision. In proposal, the agent generates candidate plans; in simulation, each plan is rolled out by the world model to predict counterfactual futures; in revision, the agent scores rollouts and refines its plan. Finally, the agent executes the top-scoring plan in the environment, coupling model-based planning with real execution.
Let $\mathbf{o}_{t}$ denote the agent’s egocentric observation at time step $t$ . The observation can be RGB, RGB-D, or other sensory modalities. For clarity, we use $\mathbf{o}$ as the generic notation throughout. Define the agent’s future potential action sequence of horizon $L$ starting at time step $t$ as $\hat{\mathbf{A}}_{t}\;=\;\bigl[\hat{a}_{t+1},\,\hat{a}_{t+2},\,...,\,\hat{a}_{t+L}\bigr],$ where each elementary action $\hat{a}$ is specified in either a continuous action space or a discrete action space, i.e., $\hat{a}∈\mathcal{V}$ , with $\mathcal{V}$ denoting the set of action primitives available to the agent.
Our unified strategy can be formalized as a policy-guided beam search. The beam width corresponds to the number of candidate plans $M$ drawn from the proposal policy $\pi_{\text{proposal}}$ . At time step $t$ , given the current observation $\mathbf{o}_{t}$ and the task goal $\mathrm{g}$ , the proposal policy $\pi_{\text{proposal}}$ samples $M$ candidate action sequences that serve as future candidate plans:
$$
\hat{\mathbf{A}}_{t}^{(m)}\;\sim\;\pi_{\text{proposal}}\bigl(\mathbf{A}\,\big|\,\mathbf{o}_{t},\,\mathrm{g}\bigr),\qquad m=1,\dots,M. \tag{1}
$$
Each candidate plan $\hat{\mathbf{A}}_{t}^{(m)}$ is subsequently transformed by the unified action API $C$ into the control inputs expected by the world model: $I_{t}^{(m)}\;=\;C\bigl(\hat{\mathbf{A}}_{t}^{(m)}\bigr),$ where $I_{t}^{(m)}$ may include textual prompts, camera trajectories, or low-level action sequences, depending on the required format of the chosen world model. The visual world model $g_{\boldsymbol{\theta}}$ then performs a counterfactual rollout based on these control inputs, predicting the future world states $\hat{\mathbf{O}}_{t}^{(m)}$ with horizon $L$ :
$$
\hat{\mathbf{O}}_{t}^{(m)}\;\sim\;g_{\boldsymbol{\theta}}\!\Bigl(\mathbf{O}\,\big|\,\mathbf{o}_{t},\,I_{t}^{(m)}\Bigr),\qquad\hat{\mathbf{O}}_{t}^{(m)}=\bigl[\hat{\mathbf{o}}_{t+1}^{(m)},\,\hat{\mathbf{o}}_{t+2}^{(m)},\,\dots,\,\hat{\mathbf{o}}_{t+L}^{(m)}\bigr]. \tag{2}
$$
Then, the candidate plans and their simulated rollouts $\bigl(\hat{\mathbf{A}}_{t}^{(m)},\hat{\mathbf{O}}_{t}^{(m)}\bigr)$ are evaluated and revised by the revision policy $\pi_{\text{revision}}$ , which assigns a score to each trajectory and selects the decision that maximizes the expected reward. In the most general form, we write
$$
\mathbf{D}^{\star}_{t}=\pi_{\text{revision}}\Bigl(\{\,(\hat{\mathbf{A}}_{t}^{(m)},\,\hat{\mathbf{O}}_{t}^{(m)})\,\}_{m=1}^{M},\,\mathbf{o}_{t},\,\mathrm{g}\Bigr). \tag{3}
$$
Here, $\mathbf{D}^{\star}_{t}$ denotes the best decision according to $\pi_{\text{revision}}$ at time step $t$ . Depending on the task, $\mathbf{D}^{\star}_{t}$ may represent a high-level answer, a recognition result, or a refined sequence of low-level actions, which renders the framework more general than classical Model Predictive Control (MPC) (Morari and H. Lee, 1999), where optimization is restricted to sequences of actions.
A common instantiation implements $\pi_{\text{revision}}$ as a score-and-select operator $S$ . When the decision is an action sequence, selection is performed over the $M$ candidate plans produced at time step $t$ :
$$
m^{\star}=\operatorname*{arg\,max}_{m\in\{1,\dots,M\}}\;S\!\left(\hat{\mathbf{A}}_{t}^{(m)},\,\hat{\mathbf{O}}_{t}^{(m)}\,\big|\,\mathbf{o}_{t},\,\mathrm{g}\right),\qquad\mathbf{D}^{\star}_{t}=\hat{\mathbf{A}}_{t}^{(m^{\star})}. \tag{4}
$$
Here, $S(·)$ denotes a task-specific scoring function that estimates the expected reward or utility of a candidate plan based on its simulated outcomes. Alternatively, $\pi_{\text{revision}}$ may synthesize or update a new decision by aggregating information across the candidate set and their predicted consequences, rather than selecting one candidate verbatim.
Once the best decision $\mathbf{D}^{\star}_{t}$ is executed in the environment, the agent acquires a new observation at time step $t{+}1$ . The unified strategy then re-enters the proposal-simulation-revision loop, using the newly observed state to initiate the next round of proposal, simulation, and revision. In our framework, both $\pi_{\text{proposal}}$ and $\pi_{\text{revision}}$ can be instantiated flexibly: they may be pretrained modules, such as large-scale vision-language models or diffusion policies, or simple rule-based heuristics. In our experiments, we explore multiple instantiations to systematically explore the flexibility and generality of our framework for different tasks.
2.2 Unified Action API
In this section, we present a unified action API that transforms an action sequence $\mathbf{A}$ into control inputs $I$ that guide the world model, i.e., $I\!=\!C(\mathbf{A})$ . The action API is designed to be flexible so that the same interface can serve a wide range of world models and tasks. It supports three principal types of control information: (1) text prompt, (2) camera trajectory/viewpoint, and (3) low-level actions, depending on the inputs expected by the chosen world model.
Text prompt. For image-and-text-to-video world models, the controller maps the intended action sequence into a descriptive text prompt. A predefined template converts each primitive action into a phrase, and concatenating these phrases yields the final prompt $I_{\text{text}}$ .
Camera trajectory / viewpoint. For models that consume explicit viewpoints, the controller translates $\mathbf{A}$ into a camera trajectory, e.g., each translation action moves the camera by $0.2\,\text{m}$ , and each rotation action changes the azimuth by $22.5^{\circ}$ . The resulting trajectory is represented as a sequence $\bigl[(x_{k},y_{k},\phi_{k})\bigr]_{k=1}^{K}$ with $(x_{k},y_{k})∈\mathbb{R}^{2}$ and azimuth $\phi_{k}∈\mathbb{R}$ .
Low-level actions. For world models that take discrete or continuous low-level actions as input, the controller maps the action sequence $\mathbf{A}$ to the world model’s action vocabulary, yielding $\mathbf{A}_{\text{world}}$ . This mapping $\mathbf{A}\mapsto\mathbf{A}_{\text{world}}$ applies the necessary transformations to maintain a unique and consistent correspondence between the agent’s actions and the inputs expected by the world model.
2.3 Comprehensive Embodied Tasks
To evaluate the practical utility of visual world models in embodied tasks, we select a diverse set of tasks that span multiple domains and stress distinct capabilities. We focus on four representative tasks: Active Recognition (AR), Active Embodied Question Answering (A-EQA), Image-Goal Navigation (ImageNav), and Robotic Manipulation, as illustrated in Figure ˜ 4. Taken together, these tasks emphasize complementary aspects of embodied intelligence, including perception, navigation, and object-level manipulation, and thus provide a comprehensive testbed for assessing how effectively a visual world model supports online planning and decision-making. Below, we describe the tasks included in our benchmark, and more detailed settings are provided in Appendix ˜ B.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Task-Oriented Robot Navigation and Manipulation
### Overview
The image presents four distinct tasks for robot navigation and manipulation within simulated environments. Each task is visually depicted with a top-down view of the environment, robot agent(s), and target objects or goals. The tasks are: Active Recognition, Image-Goal Navigation, Active Embodied QA, and Robotic Manipulation. Each task includes a brief description of the objective and visual examples of the environment and robot actions.
### Components/Axes
**1. Active Recognition:**
* **Title:** Active Recognition
* **Objective:** Navigate as needed and identify the object marked by the red bounding box (bbox).
* **Environment:** A bathroom scene with a toilet, sink, door, and other furniture.
* **Agent:** Represented by a humanoid icon.
* **Visual Cues:**
* Red bounding box highlighting a specific object (e.g., a door frame).
* Green shaded area indicating the navigable space.
* Purple arrows indicating the agent's movement.
* **Steps:**
* Step 1: <Front> view - A front view of a door with a red bounding box around the door frame.
* Step 2: <Front> view - A front view of a door with a red bounding box around the door frame.
* **Question:** "What is the target object bounded by the red box?"
**2. Image-Goal Navigation:**
* **Title:** Image-Goal Navigation
* **Objective:** Navigate to the location from which the <Goal Image> was captured.
* **Environment:** A bedroom scene with beds, furniture, and windows.
* **Agent:** Represented by a humanoid icon.
* **Visual Cues:**
* <Goal Image>: A photograph of the bedroom from a specific viewpoint.
* Green shaded area indicating the navigable space.
* Purple arrows indicating the agent's movement.
* **Steps:**
* Step 1: <Front> view - A front view of the bedroom matching the <Goal Image>.
* <Goal Image>: A photograph of the bedroom from a specific viewpoint.
**3. Active Embodied QA:**
* **Title:** Active Embodied QA
* **Objective:** Navigate as needed and answer the user's <Query>.
* **Environment:** A living room/kitchen scene with sofas, tables, chairs, and kitchen appliances.
* **Agent:** Represented by a humanoid icon.
* **Visual Cues:**
* Purple arrows indicating the agent's movement.
* **Steps:**
* Step 1: <Front> view - A front view of the kitchen area.
* **Question:** "How many cushions are on the red sofa?"
**4. Robotic Manipulation:**
* **Title:** Robotic Manipulation
* **Objective:** Use the robotic arm to slide the red block onto the blue target.
* **Environment:** A table with colored blocks (red, blue, yellow, green).
* **Agent:** Represented by a robotic arm.
* **Visual Cues:**
* Colored blocks (red, blue, yellow, green).
* **Steps:**
* Step 1: The robotic arm is positioned above the table with the colored blocks.
* Step 2: The robotic arm has moved the red block onto the blue target.
### Detailed Analysis or ### Content Details
* **Active Recognition:** The agent needs to identify the object within the red bounding box by navigating the environment. The example shows the agent identifying a door frame.
* **Image-Goal Navigation:** The agent needs to navigate to the location where the goal image was taken. The example shows the agent navigating to a specific viewpoint in a bedroom.
* **Active Embodied QA:** The agent needs to answer a question about the environment by navigating and observing. The example shows the agent answering a question about the number of cushions on a red sofa.
* **Robotic Manipulation:** The robotic arm needs to manipulate objects in the environment to achieve a specific goal. The example shows the arm sliding a red block onto a blue target.
### Key Observations
* The tasks involve different levels of complexity, ranging from object recognition to question answering and robotic manipulation.
* The environments are simulated and visually realistic.
* The agents are represented by humanoid icons or robotic arms.
* The tasks require both navigation and interaction with the environment.
### Interpretation
The image illustrates a range of tasks that robots can perform in simulated environments. These tasks demonstrate the capabilities of robots in areas such as object recognition, navigation, question answering, and manipulation. The tasks are designed to be challenging and require robots to reason about the environment and interact with it in a meaningful way. The image suggests that robots are becoming increasingly capable of performing complex tasks in real-world environments.
</details>
Figure 4: Top-left: Active Recognition (AR), the agent needs to identify a designated target under occlusions or extreme viewpoints while minimizing navigation cost. Top-right: Image-Goal Navigation (ImageNav), the agent reaches the viewpoint matching a goal image, emphasizing success rate and path efficiency. Bottom-left: Active Embodied Question Answering (A-EQA), the agent answers an open-ended question after active exploration. Bottom-right: Robotic Manipulation, the agent needs to control a robotic arm to complete tasks such as grasping and placement to specified targets.
Active Recognition (AR) is closely related to amodal recognition (Aydemir et al., 2013; Liu et al., 2018; Yang et al., 2019; Fan et al., 2024; Bhattacharjee et al., 2025), in which the agent must identify a designated target that may be observed from extreme viewpoints or be heavily occluded. In addition, AR allows the agent to acquire additional observations through active exploration. All AR experiments are conducted in the Habitat-Sim (Savva et al., 2019), encompassing 551 episodes across 29 scenes from the validation split of Matterport3D (Chang et al., 2017). Within AR, the visual world model assists two decision-making processes. For answering, synthetic views provide auxiliary evidence that helps the agent reason about occlusions and extreme viewpoints that impede recognition. For navigation, rollouts simulate the consequences of potential actions so that the agent can choose a path that is more likely to yield informative observations.
Image-Goal Navigation (ImageNav), also referred to as goal-conditioned visual navigation, requires an embodied agent to reach a target position in a scene given a single reference image that specifies the goal viewpoint. We construct 144 ImageNav episodes from 87 validation scenes of HM3D (Ramakrishnan et al., 2021). In this task, the visual world model exclusively supports navigation decisions. The agent simulates the outcomes of candidate action plans, selects the best option, executes the first segment of that plan, and then replans with the newly observed state in a closed-loop manner.
Active Embodied Question Answering (A-EQA) requires an agent to answer open-ended natural-language questions after actively exploring a 3D environment. Our evaluation set includes 184 questions across 54 indoor scenes from the official OpenEQA split (Majumdar et al., 2024) and the HM3D validation set (Ramakrishnan et al., 2021). As in AR, the visual world model supports both question answering and navigation. For answering, synthetic views generated by the world model provide complementary perspectives that help resolve references to occluded or distant objects. For navigation, the agent simulates high-level action plans using the world model’s predictions to choose exploration strategies likely to reveal question-relevant information.
Robotic Manipulations are fundamental capabilities for embodied agents that must operate in real-world interaction settings. We study how visual world models contribute to closed-loop manipulation planning, evaluating performance on four RLBench (James et al., 2020) tasks with 50 episodes per task. Here, the visual world model supports the agent in assessing candidate $7$ -DoF gripper actions by providing visual evidence about anticipated object motions and interactions, which enables a comparison of alternative plans before execution. The predicted outcomes then guide the selection of actions that are more likely to achieve the specified objective, thereby linking visual prediction accuracy to improvements in manipulation performance.
2.4 Exploiting World Models via Post-Training
To evaluate the feasibility of adapting pretrained video generators for embodied tasks, we introduce a post-training procedure that aligns a pretrained model with the domain distribution and action space of target environments. We perform fine-tuning separately on data from two simulators, Habitat-Sim and CoppeliaSim, to match the corresponding task domains. For Habitat-Sim tasks (AR, A-EQA, ImageNav), we post-train on a panoramic action-observation dataset collected from the HM3D (Ramakrishnan et al., 2021) training split. For CoppeliaSim tasks (Robotic Manipulation), we post-train on task demonstrations generated with RLBench (James et al., 2020). To assess generalization rather than memorization, all Habitat-Sim data used for post-training are sourced from scenes that are disjoint from our evaluation scenes, so the scenes in our evaluation tasks remain unseen by the world models after post-training. Additional details regarding the training objective, dataset construction, and training configuration are provided in Appendices ˜ C and D.
3 Evaluation Results and Analysis
In this section, we report quantitative results and key observations on the four embodied tasks in Section ˜ 3.1, followed by ablation studies in Section ˜ 3.2. We evaluate visual world models spanning image-based (PathDreamer (Koh et al., 2021b), SE3DS (Koh et al., 2023)) and video-based (SVD (Blattmann et al., 2023a), LTX-Video (HaCohen et al., 2024), Hunyuan (Kong et al., 2024), Wan2.1 (Wan et al., 2025), Wan2.2 (Wan et al., 2025), Cosmos-Predict2 (Agarwal et al., 2025), NWM (Bar et al., 2025a)) approaches, covering major control interfaces. For video-based models, we compare off-the-shelf versions with their post-trained variants.
3.1 Benchmark Results
Table 1: Active Recognition (AR) and Image-Goal Navigation (ImageNav) performance across various models and base policies. Higher success rate (SR %), success weighted by path length (SPL %), and lower mean trajectory length (Mean Traj.) are better. “ $\dagger$ ” denotes our post-trained video generators.
| Model Details | AR | ImageNav | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model Type | Method | Control Type | Input Type | #Param. | SR $\uparrow$ | Mean Traj. $\downarrow$ | SR $\uparrow$ | Mean Traj. $\downarrow$ | SPL $\uparrow$ |
| Base Policy | Heuristic (w/o WM) | – | RGB | – | 39.02 | 8.81 | 2.08 | 59.6 | 0.63 |
| + Video Gen. Post-Train | SVD $\dagger$ | Action | RGB; Pano | 1.5B | 60.62 | 5.17 | 20.83 | 58.5 | 11.86 |
| WAN2.1 $\dagger$ | Action | RGB; Pano | 14B | 62.98 | 4.71 | 22.92 | 58.7 | 11.63 | |
| Base Policy | VLM (w/o WM) | – | RGB | 72B | 50.27 | 6.24 | 35.42 | 47.5 | 25.88 |
| + Image Gen. | PathDreamer | Viewpoint | RGB - D; Pano | 0.69B | 56.99 | 5.28 | 36.80 | 47.3 | 26.85 |
| + Image Gen. | SE3DS | Viewpoint | RGB - D; Pano | 1.1B | 57.53 | 5.29 | 36.11 | 47.0 | 26.91 |
| + Video Gen. | NWM | Trajectory | RGB | 1B | 57.35 | 5.68 | 40.28 | 47.1 | 27.83 |
| + Video Gen. Zero-Shot | SVD | Image | RGB | 1.5B | 57.71 | 5.29 | 40.28 | 46.4 | 28.59 |
| LTX - Video | Text | RGB | 2B | 56.08 | 5.37 | 36.81 | 47.5 | 25.85 | |
| Hunyuan | Text | RGB | 13B | 57.71 | 5.21 | 36.11 | 46.8 | 26.89 | |
| Wan2.1 | Text | RGB | 14B | 58.26 | 5.24 | 38.19 | 48.2 | 25.92 | |
| Wan2.2 | Text | RGB | 5B | 55.35 | 5.73 | 38.88 | 46.5 | 28.87 | |
| Cosmos - P2 | Text | RGB | 2B | 55.35 | 5.71 | 36.81 | 47.6 | 25.89 | |
| Wan2.2 | Text | RGB | A14B | 59.53 | 4.91 | 43.05 | 45.8 | 31.46 | |
| Runway Gen4 (proprietary) | Text | RGB | – | 64.79 | 4.06 | - | - | - | |
| + Video Gen. Post-Train | SVD $\dagger$ | Action | RGB; Pano | 1.5B | 60.98 | 5.02 | 43.05 | 46.0 | 30.96 |
| LTX - Video $\dagger$ | Action | RGB; Pano | 2B | 57.53 | 5.49 | 38.89 | 47.4 | 27.47 | |
| WAN2.1 $\dagger$ | Action | RGB; Pano | 14B | 62.61 | 4.73 | 45.14 | 45.8 | 32.10 | |
| Cosmos - P2 $\dagger$ | Action | RGB; Pano | 2B | 60.25 | 5.08 | 41.67 | 45.5 | 30.29 | |
| Wan2.2 $\dagger$ | Action | RGB; Pano | 5B | 56.26 | 5.15 | 38.89 | 46.7 | 28.24 | |
| Wan2.2 $\dagger$ | Action | RGB; Pano | A14B | 62.43 | 4.67 | 46.53 | 44.6 | 34.61 | |
Table 2: Active Embodied Question Answering (A-EQA) performance.
| Model Details | A-EQA Performance | | | |
| --- | --- | --- | --- | --- |
| Model Type | Method | Ans. Score $\uparrow$ | Mean Traj. $\downarrow$ | SPL $\uparrow$ |
| Base Policy | VLM (w/o WM) | 45.7 | 20.4 | 29.6 |
| + Image Gen. | PathDreamer | 46.0 | 20.4 | 29.3 |
| + Image Gen. | SE3DS | 45.8 | 20.3 | 29.4 |
| + Video Gen. | NWM | 47.1 | 20.5 | 30.1 |
| + Video Gen. | Wan2.1 | 45.7 | 20.1 | 28.8 |
| Wan2.2 (5B) | 46.3 | 20.3 | 31.4 | |
| LTX - Video | 46.6 | 20.8 | 29.5 | |
| Cosmos - P2 | 46.6 | 21.0 | 31.3 | |
| Hunyuan | 46.8 | 20.4 | 29.9 | |
| SVD | 46.9 | 20.4 | 29.7 | |
| Wan2.2 (A14B) | 47.2 | 20.7 | 31.9 | |
| + Video Gen. Post-Train | SVD $\dagger$ | 46.4 | 21.1 | 30.1 |
| Cosmos - P2 $\dagger$ | 46.5 | 20.6 | 30.1 | |
| Wan2.2 $\dagger$ (5B) | 47.5 | 20.8 | 30.7 | |
| Wan2.1 $\dagger$ | 48.2 | 20.7 | 31.6 | |
| LTX - Video $\dagger$ | 48.6 | 20.7 | 31.8 | |
| Wan2.2 $\dagger$ (A14B) | 48.4 | 20.2 | 31.9 | |
Table 3: Robotic manipulation performance across various models and base policies.
| Model Details | Manipulation Performance | | |
| --- | --- | --- | --- |
| Model Type | Method | SR $\uparrow$ | Mean Traj. $\downarrow$ |
| Base Policy | VLM (w/o WM) | 44.5 | 2.52 |
| + Video Gen. | SVD | 44.0 | 2.47 |
| LTX - Video | 44.5 | 2.46 | |
| Hunyuan | 44.5 | 2.44 | |
| Wan2.1 | 44.0 | 2.51 | |
| Cosmos - P2 | 44.0 | 2.50 | |
| + Video Gen. Post-Train | SVD $\dagger$ | 46.5 | 2.38 |
| Cosmos - P2 $\dagger$ | 45.0 | 2.40 | |
| Base Policy | 3D - DP (w/o WM) | 24.0 | 5.21 |
| + Video Gen. Post-Train | SVD $\dagger$ | 44.7 | 4.41 |
| Cosmos - P2 $\dagger$ | 38.0 | 4.79 | |
World models can enhance the performance of the base proposal policy. Across AR, A-EQA, ImageNav, and Manipulation, adding a visual world model consistently improves the performance of the base proposal policy (e.g., a VLM policy, a heuristic policy, or a 3D diffusion policy), as shown in Tables ˜ 1, 3 and 3. For example, in AR, the best proprietary model (Runway Gen4) attains an accuracy of $64.79\%$ while reducing the mean steps per episode to $4.06$ , compared to the VLM base policy with an accuracy of $50.27\%$ and mean steps $6.24$ . Similarly, in ImageNav, the best open-source model Wan2.1 $\dagger$ achieves a success rate of $45.14\%$ with an average path length of $45.8$ , outperforming the VLM base policy at $35.42\%$ SR and $47.5$ average length. These results support the effectiveness of our World-in-World online planning framework with world models, in which the world model provides simulated future states that inform better decisions.
World models struggle to simulate precise motion and dynamics in manipulation. The gains are less pronounced for Robotic Manipulations (Table ˜ 3), likely because accurately modeling contact-rich interactions and robot kinematics is significantly more challenging than predicting purely view changes. For instance, the best post-trained model on manipulation (SVD $\dagger$ ) reaches an SR of $46.5\%$ with a mean trajectory length of $2.38$ , only modestly above the VLM baseline at $44.5\%$ SR and $2.52$ mean length. This gap suggests that while current visual world models can effectively guide perception and navigation, capturing fine-grained physical dynamics and action-conditioned object motion remains an open challenge.
Post-training substantially boosts world-model utility. Our post-training adaptation yields consistent improvements. Relative to off-the-shelf Wan2.1, Wan2.1 $\dagger$ raises AR accuracy from $58.26\%$ to $62.61\%$ and ImageNav SR from $38.19\%$ to $45.14\%$ (Table ˜ 1). Likewise, SVD $\dagger$ improves AR accuracy from $57.71\%$ to $60.98\%$ and ImageNav SR from $40.28\%$ to $43.05\%$ . These gains show that aligning the generative model to the target domain and action space of the specific embodied tasks improves downstream decision-making.
3.2 Ablation and Findings
<details>
<summary>x5.png Details</summary>
### Visual Description
## Scatter Plot: Task Success Rate vs. Quality/Controllability
### Overview
The image contains two scatter plots comparing the Task Success Rate (%) of different models against two different metrics: General Quality (Aesthetic+Image Quality) and Controllability (1 - LPIPS). The plots show the performance of Zero-shot, Post-trained, and Other models.
### Components/Axes
**Left Scatter Plot:**
* **Title:** Implicitly "Task Success Rate vs. General Quality"
* **Y-axis:** Task Success Rate (%), ranging from 55% to 65% with gridlines at each percentage point.
* **X-axis:** Gen. Quality (Aesthetic+Image Quality) ↑, ranging from approximately 0.325 to 0.475, with gridlines at intervals of 0.025. The upward arrow indicates that higher values are better.
* **Legend (Top-Left):**
* Red circle: Zero-shot
* Blue circle: Post-trained
* Green square: Others
**Right Scatter Plot:**
* **Title:** Implicitly "Task Success Rate vs. Controllability"
* **Y-axis:** Task Success Rate (%), ranging from 55% to 65% with gridlines at each percentage point.
* **X-axis:** Controllability (1 - LPIPS) ↑, ranging from approximately 0.15 to 0.50, with gridlines at intervals of 0.05. The upward arrow indicates that higher values are better.
* **Legend (Top-Left):**
* Red circle: Zero-shot
* Blue circle: Post-trained
* Green square: Others
### Detailed Analysis
**Left Scatter Plot (Task Success Rate vs. General Quality):**
* **Zero-shot (Red):**
* Runway Gen4: Task Success Rate ~64.8%, Gen. Quality ~0.47
* Wan2.1: Task Success Rate ~58.2%, Gen. Quality ~0.46
* Wan2.2 A14B: Task Success Rate ~59.5%, Gen. Quality ~0.43
* Hunyuan: Task Success Rate ~58%, Gen. Quality ~0.40
* SVD: Task Success Rate ~58%, Gen. Quality ~0.37
* LTXVideo: Task Success Rate ~56%, Gen. Quality ~0.36
* Wan2.2 5B: Task Success Rate ~55.8%, Gen. Quality ~0.41
* Cosmos-P2: Task Success Rate ~55.5%, Gen. Quality ~0.47
* **Post-trained (Blue):**
* Wan2.1†: Task Success Rate ~62.5%, Gen. Quality ~0.39
* SVD†: Task Success Rate ~60.8%, Gen. Quality ~0.36
* Cosmos-P2†: Task Success Rate ~60.2%, Gen. Quality ~0.36
* LTXVideot: Task Success Rate ~57.5%, Gen. Quality ~0.35
* Wan2.2 5B†: Task Success Rate ~56.2%, Gen. Quality ~0.39
* **Others (Green):**
* NWM: Task Success Rate ~57.3%, Gen. Quality ~0.32
* SE3DS: Task Success Rate ~57.2%, Gen. Quality ~0.36
* Pathdreamer: Task Success Rate ~57%, Gen. Quality ~0.35
**Right Scatter Plot (Task Success Rate vs. Controllability):**
* **Zero-shot (Red):**
* Runway Gen4: Task Success Rate ~64.8%, Controllability ~0.32
* Wan2.2 A14B: Task Success Rate ~59.5%, Controllability ~0.28
* Wan2.1: Task Success Rate ~58.2%, Controllability ~0.23
* Hunyuan: Task Success Rate ~58%, Controllability ~0.34
* SVD: Task Success Rate ~58%, Controllability ~0.32
* Cosmos-P2: Task Success Rate ~55.5%, Controllability ~0.17
* Wan2.2 5B: Task Success Rate ~55.8%, Controllability ~0.37
* **Post-trained (Blue):**
* Wan2.1†: Task Success Rate ~62.5%, Controllability ~0.45
* SVD†: Task Success Rate ~60.8%, Controllability ~0.48
* Cosmos-P2†: Task Success Rate ~60.2%, Controllability ~0.46
* Wan2.2 5B†: Task Success Rate ~56.2%, Controllability ~0.40
* LTXVideot: Task Success Rate ~57.5%, Controllability ~0.38
* **Others (Green):**
* NWM: Task Success Rate ~57.3%, Controllability ~0.35
* SE3DS: Task Success Rate ~57.2%, Controllability ~0.33
* Pathdreamer: Task Success Rate ~57%, Controllability ~0.28
### Key Observations
* Runway Gen4 (Zero-shot) achieves the highest Task Success Rate in both plots, but its Controllability is relatively low.
* Post-trained models (Wan2.1†, SVD†, Cosmos-P2†) generally show a good balance between Task Success Rate and both General Quality and Controllability.
* The "Others" category models (NWM, SE3DS, Pathdreamer) tend to cluster in the lower-left region of both plots, indicating lower performance in both Task Success Rate and the respective quality metrics.
* There is a positive correlation between Task Success Rate and both General Quality and Controllability, although the relationship is not strictly linear.
### Interpretation
The scatter plots provide a comparative analysis of different models based on their Task Success Rate, General Quality, and Controllability. The data suggests that post-training can improve the performance of models in terms of both Task Success Rate and the quality metrics. However, there is a trade-off between General Quality and Controllability for some models. For example, Runway Gen4 excels in Task Success Rate but has relatively low Controllability. The plots highlight the importance of considering multiple metrics when evaluating the performance of models, as optimizing for one metric may come at the expense of others. The "Others" category models appear to be less effective compared to the Zero-shot and Post-trained models based on these metrics.
</details>
Figure 5: (a) SR vs. generation quality in AR; generation quality is scored as the average of an aesthetic predictor (Akio Kodaira, 2024) and an image-quality predictor (Ke et al., 2021), both trained to match human preferences. (b) SR vs. controllability in AR; controllability is quantified as $1-\mathrm{LPIPS}$ between ground-truth and predicted observations.
Fine-grained controllability matters more than visuals for task success. Although recent off-the-shelf video generators like Wan2.1 produce visually appealing clips, they are driven by text prompts with limited fine-grained low-level controls. Without adaptation, these models yield only small gains on downstream embodied tasks. We further study the relation between controllability and the success rate on AR. Here, controllability is defined as alignment between intended actions and the motions in the model’s predictions. After action-conditioned post-training, alignment improves substantially and SR rises accordingly. Figure ˜ 5 (b) shows a positive correlation: models that respond reliably to low-level controls achieve higher SR. These results indicate that precise control, not just visual quality, is critical for embodied world models to support effective decision-making.
<details>
<summary>x6.png Details</summary>
### Visual Description
## Line Chart: Success Rate vs. Seen Examples During Training
### Overview
The image is a line chart comparing the success rates of three different models (Wan2.2, Wan2.1, and SVD) as a function of the number of seen examples during training. The x-axis represents the number of seen examples during training, while the y-axis represents the success rate in percentage.
### Components/Axes
* **X-axis:** "Seen Examples During Training" with markers at 400, 4K, 40K, and 80K.
* **Y-axis:** "Success Rate (%)" with a scale from 52 to 64, incrementing by 1.
* **Legend:** Located in the bottom-right corner.
* Wan2.2† (represented by a yellow star)
* Wan2.1† (represented by a green circle)
* SVD† (represented by a blue square)
### Detailed Analysis
* **Wan2.1† (Green Line):** The line slopes upward.
* At 400 seen examples, the success rate is 60.25%.
* At 4K seen examples, the success rate is 61.52%.
* At 40K seen examples, the success rate is 62.61%.
* At 80K seen examples, the success rate is 63.34%.
* **SVD† (Blue Line):** The line slopes upward until 40K, then plateaus.
* At 400 seen examples, the success rate is 56.26%.
* At 4K seen examples, the success rate is 56.44%.
* At 40K seen examples, the success rate is 60.98%.
* At 80K seen examples, the success rate is 60.98%.
* **Wan2.2† (Yellow Star):** Only one data point is shown.
* At 40K seen examples, the success rate is 62.61%.
### Key Observations
* Wan2.1 consistently outperforms SVD across all tested numbers of seen examples.
* SVD's performance plateaus after 40K seen examples.
* Wan2.2 has a single data point at 40K, matching the performance of Wan2.1 at that point.
### Interpretation
The chart suggests that Wan2.1 is a more effective model than SVD, as it achieves higher success rates with the same amount of training data. The plateau in SVD's performance indicates that it may have reached its learning capacity with the given dataset or architecture. The single data point for Wan2.2 at 40K suggests it may be a variant or configuration of Wan2.1, achieving similar performance at that specific training level. The dagger symbol "†" next to each model name may indicate a specific version or configuration.
</details>
Figure 6: SR vs. seen examples during post-training. SR increases consistently with more downstream data, revealing a clear data-scaling trend for adaptation.
<details>
<summary>x7.png Details</summary>
### Visual Description
## Line Chart: Success Rate vs. Avg Inference Count
### Overview
The image is a line chart comparing the success rate (%) of two methods, Wan2.1 and SVD, against the average inference count per episode. The x-axis represents the average inference count per episode, ranging from 3.0 to 11.0. The y-axis represents the success rate in percentage, ranging from 52% to 64%.
### Components/Axes
* **X-axis:** Avg Inference Count per Episode (ranging from 3.0 to 11.0 in increments of 1.0)
* **Y-axis:** Success Rate (%) (ranging from 52 to 64 in increments of 1)
* **Legend:** Located in the bottom-right corner.
* Green line with circle markers: Wan2.1†
* Blue line with square markers: SVD†
### Detailed Analysis
* **Wan2.1† (Green Line):** The success rate generally increases with the average inference count per episode.
* At 3.0 inference count, the success rate is 56.62%.
* At 6.0 inference count, the success rate is 58.26%.
* At 8.0 inference count, the success rate is 59.71%.
* At 11.0 inference count, the success rate is 62.61%.
* **SVD† (Blue Line):** The success rate also increases with the average inference count per episode.
* At 3.0 inference count, the success rate is 53.36%.
* At 6.0 inference count, the success rate is 56.44%.
* At 8.0 inference count, the success rate is 57.17%.
* At 11.0 inference count, the success rate is 60.98%.
### Key Observations
* Both Wan2.1 and SVD show an increase in success rate as the average inference count per episode increases.
* Wan2.1 consistently outperforms SVD across all measured inference counts.
* The SVD line shows a steeper increase in success rate between inference counts of 8.0 and 11.0 compared to the segment between 3.0 and 6.0.
### Interpretation
The data suggests that increasing the average inference count per episode improves the success rate for both Wan2.1 and SVD methods. Wan2.1 appears to be a more effective method overall, achieving higher success rates than SVD at each inference count level. The steeper increase in SVD's success rate at higher inference counts may indicate that SVD benefits more from increased inference counts than Wan2.1, although it still remains below Wan2.1's performance.
</details>
Figure 7: SR vs. average number of world-model inferences per episode. Increasing the inference-time computation allocated to each decision step leads to higher SR.
Data-size scaling for post-trained models. We study how post-training data size affects WM performance (Wan2.2 $\dagger$ , Wan2.1 $\dagger$ , SVD $\dagger$ ). Each WM is post-trained for one epoch on datasets from $400$ to $80\text{K}$ instances. As shown in Figure ˜ 7, more post-training data consistently improves AR performance: Wan2.1 $\dagger$ rises from $60.25\%$ to $63.34\%$ , and SVD $\dagger$ from $56.80\%$ to $60.98\%$ . Wan2.2 $\dagger$ (A14B), despite substantially larger web-video pretraining, reaches nearly the same performance as Wan2.1 $\dagger$ after $40\text{K}$ post-training instances, suggesting that scaling action-conditioned post-training is more effective for embodied utility than upgrading the pretrained generator. Moreover, larger models (Wan2.1 $\dagger$ , 14B) benefit more and saturate less than smaller ones (SVD $\dagger$ , 1.5B), indicating greater capacity to absorb action-conditioned supervision.
Inference-time scaling for online planning with world models. Within our online planning framework, the number of world-model inferences (simulated potential futures per episode) directly affects task performance. As shown in Figure ˜ 7, increasing the average inferences per episode for AR yields a clear positive correlation with SR. For example, increasing the average inference count from 3 to 11 improves SR from $53.36\%$ to $60.98\%$ for SVD $\dagger$ . This suggests that allocating more inference-time computation to simulate potential futures lets the planner make more informed decisions, thereby improving overall performance.
Table 4: Post-training with different input contexts: front view vs. panorama.
| Task | Model | Front View | Panorama | | |
| --- | --- | --- | --- | --- | --- |
| SR $\uparrow$ | Mean Traj. $\downarrow$ | SR $\uparrow$ | Mean Traj. $\downarrow$ | | |
| AR | SVD $\dagger$ | 57.89 | 5.04 | 60.98 | 5.02 |
| Wan2.1 $\dagger$ | 62.25 | 4.82 | 62.61 | 4.73 | |
| Wan2.2 $\dagger$ (5B) | 57.16 | 5.08 | 56.26 | 5.15 | |
| Cosmos-P2 $\dagger$ | 58.98 | 4.94 | 60.25 | 5.08 | |
| ImageNav | SVD $\dagger$ | 38.19 | 47.0 | 43.05 | 46.0 |
| Wan2.1 $\dagger$ | 48.61 | 43.8 | 45.14 | 45.8 | |
| Wan2.2 $\dagger$ (5B) | 40.97 | 45.8 | 38.89 | 46.7 | |
| Cosmos-P2 $\dagger$ | 40.97 | 47.0 | 41.67 | 45.5 | |
Global vs. local context for generation. We study the effect of input context format. Specifically, we compare post-trained models conditioned on panoramic versus front-view input (Table ˜ 4). Panoramic input provides a $360^{\circ}$ field of view, whereas front view offers a focused but limited perspective. For fairness, generated panoramas are converted to perspective views with the same horizontal field of view during evaluation. Although panoramic input offers richer global context, it does not consistently yield large gains across all settings. Likely, panorama-to-perspective conversion introduces resolution loss, degrading downstream perception and planning.
4 Discussion and Future Directions
Generalization capacity of world models is critical for practical use. Most video generators are pretrained on web videos. In unseen embodied environments, they may revert to training priors or ignore action controls, yielding plausible but physically or semantically inconsistent rollouts (see Figures ˜ 13 and 14). These deviations mislead planning and reduce success. Larger models or more pretraining data can partly help, but robust generalization remains central. Future work should prioritize strategies and action representations to improve transfer to novel environments, such as unified action representations (Wang et al., 2025b; Zhi et al., 2025; Wang et al., 2025a) and curriculum or domain-specific data collection (Zhao et al., 2025).
Long-horizon planning with world models remains challenging. In our experiments, visual world models simulate short-term changes but struggle on long horizons due to limited mechanisms for accumulating spatiotemporal history. We attempted to alleviate this issue by replacing front-view inputs with panoramas to provide global context, but gains were inconsistent across models and tasks. Future work should better encode and retrieve long-term dependencies, e.g., spatial memory (Zhou et al., 2025b; Xiao et al., 2025; Li et al., 2025c; Yu et al., 2025a; Ren et al., 2025) and episode-level memory (Cai et al., 2025; Guo et al., 2025), to maintain scene-level context and enable coherent planning over extended horizons.
Precise modeling of interactions and dynamics remains difficult. For manipulation, capturing contact-rich interactions, compliance, friction, and state changes of articulated or deformable objects is essential. Current visual world models often miss these details, producing rollouts that violate physics and degrade planning and control—consistent with our observations and prior analyses (Kang et al., 2024). Promising directions include physics-guided motion generation (Chefer et al., 2025; Zhang et al., 2025b; Akkerman et al., 2025) and inferring or generating physical properties to inform action-conditioned predictions (Cao et al., 2025; Gillman et al., 2025; Zhang et al., 2024). Integrating such signals into conditioning pathways may improve fidelity when precise dynamics are required.
Stronger proposal and revision policies set the performance floor. The agent’s overall performance depends on both world-model fidelity and the strength of the proposal and revision policies that select and refine decisions. While simulated rollouts improve decision-making, base policies must be effective to provide a reliable starting point, and strengthening them raises the ceiling. Future work could explore stronger policies (Geng et al., 2025; Kim et al., 2025), and integration strategies that deepen synergy between world models and decision-making (Neary et al., 2025), such as more human-aligned reward models (Wang et al., 2024; Seneviratne et al., 2025; Rocamonde et al., 2023).
5 Conclusion
We introduce World-in-World, a closed-loop world interface and benchmark that evaluates generative world models via embodied interaction rather than isolated visual metrics. By unifying heterogeneous controls, our action API enables any world model to serve as perception and planning utilities for an embodied agent. Coupled with a unified closed-loop planning strategy that proposes, simulates, and revises action plans, the benchmark measures agent performance on four demanding tasks. Our experiments reveal large gaps between visual metrics and task success, underscoring the need for closed-loop evaluation, and show that pretrained video generators improve with post-training data scaling and inference-time scaling. We expect World-in-World to guide world models toward not only striking visual realism but also reliable perception, planning, and action in embodied scenarios.
References
- Agarwal et al. (2025) Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.
- Akio Kodaira (2024) Sayan Goswami Akio Kodaira. Aesthetic predictor v2.5, May 2024. URL https://github.com/discus0434/aesthetic-predictor-v2-5/.
- Akkerman et al. (2025) Rick Akkerman, Haiwen Feng, Michael J. Black, Dimitrios Tzionas, and Victoria Fernández Abrevaya. Interdyn: Controllable interactive dynamics with video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12467–12479, 2025.
- Alonso et al. (2024) Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
- Aydemir et al. (2013) Alper Aydemir, Andrzej Pronobis, Moritz Göbelbecker, and Patric Jensfelt. Active visual object search in unknown environments using uncertain semantics. IEEE Transactions on Robotics, 29(4):986–1002, August 2013. ISSN 1941-0468.
- Bahmani et al. (2024a) Sherwin Bahmani, Xian Liu, Wang Yifan, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. Tc4d: Trajectory-conditioned text-to-4d generation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 53–72. Springer, 2024a.
- Bahmani et al. (2024b) Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7996–8006, 2024b.
- Bai et al. (2025) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
- Bar et al. (2025a) Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025a.
- Bar et al. (2025b) Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
- Bhattacharjee et al. (2025) Subhransu S. Bhattacharjee, Dylan Campbell, and Rahul Shome. Believing is seeing: Unobserved object detection using generative models, March 2025.
- Blattmann et al. (2023a) Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023a.
- Blattmann et al. (2023b) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
- Brooks et al. (2024a) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAI Blog, 1:8, 2024a.
- Brooks et al. (2024b) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Sora: Video generation models as world simulators. OpenAI Blog, 1:8, 2024b.
- Cai et al. (2025) Shengqu Cai, Ceyuan Yang, Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang, Yinghao Xu, Zhenheng Yang, Alan L. Yuille, Leonidas J. Guibas, Maneesh Agrawala, Lu Jiang, and Gordon Wetzstein. Mixture of contexts for long video generation. ArXiv, 2508.21058, 2025.
- Cao et al. (2025) Ziang Cao, Zhaoxi Chen, Liang Pan, and Ziwei Liu. Physx-3d: Physical-grounded 3d asset generation. ArXiv, 2507.12465, 2025.
- Chang et al. (2017) Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In Proceedings of the International Conference on 3D Vision (3DV), pages 667–676, October 2017.
- Chefer et al. (2025) Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. ArXiv, 2502.02492, 2025.
- Cheng et al. (2024) Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16901–16911, 2024.
- Chung et al. (2023) Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384, 2023.
- Du et al. (2023) Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156–9172, 2023.
- Du et al. (2024) Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video language planning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
- Duan et al. (2025) Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, and Jiajun Wu. Worldscore: A unified evaluation benchmark for world generation. arXiv preprint arXiv:2504.00983, 2025.
- Fan et al. (2024) Lei Fan, Mingfu Liang, Yunxuan Li, Gang Hua, and Ying Wu. Evidential active recognition: Intelligent and prudent open-world embodied perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16351–16361, 2024.
- Fridman et al. (2023) Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. Advances in Neural Information Processing Systems (NeurIPS), 36:39897–39914, 2023.
- Gao et al. (2024) Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In Advances in Neural Information Processing Systems (NeurIPS), November 2024.
- Geng et al. (2025) Haoran Geng, Feishi Wang, Songlin Wei, Yuyang Li, Bangjun Wang, Boshi An, Charlie Tianyue Cheng, Haozhe Lou, Peihao Li, Yen-Jen Wang, Yutong Liang, Dylan Goetting, Chaoyi Xu, Haozhe Chen, Yuxi Qian, Yiran Geng, Jiageng Mao, Weikang Wan, Mingtong Zhang, Jiangran Lyu, Siheng Zhao, Jiazhao Zhang, Jialiang Zhang, Chengyang Zhao, Haoran Lu, Yufei Ding, Ran Gong, Yuran Wang, Yuxuan Kuang, Ruihai Wu, Baoxiong Jia, Carlo Sferrazza, Hao Dong, Siyuan Huang, Yue Wang, Jitendra Malik, and Pieter Abbeel. Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning. ArXiv, 2504.18904, 2025.
- Gillman et al. (2025) Nate Gillman, Charles Herrmann, Michael Freeman, Daksh Aggarwal, Evan Luo, Deqing Sun, and Chen Sun. Force prompting: Video generation models can learn and generalize physics-based control signals. ArXiv, 2505.19386, 2025.
- Guo et al. (2025) Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, and Lu Jiang. Long context tuning for video generation. ArXiv, 2503.10589, 2025.
- HaCohen et al. (2024) Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103, 2024.
- He et al. (2025a) Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2025a.
- He et al. (2025b) Hao He, Ceyuan Yang, Shanchuan Lin, Yinghao Xu, Meng Wei, Liangke Gui, Qi Zhao, Gordon Wetzstein, Lu Jiang, and Hongsheng Li. Cameractrl ii: Dynamic scene exploration via camera-controlled video diffusion models. arXiv preprint arXiv:2503.10592, 2025b.
- He et al. (2025c) Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Cyrus Wu, Wei Li, Xuchen Song, Yang Liu, Eric Li, and Yahui Zhou. Matrix-game 2.0: An open-source, real-time, and streaming interactive world model. arXiv preprint arXiv:2508.13009, 2025c.
- Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- Hu et al. (2023) Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, September 2023.
- Huang et al. (2024) Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818, 2024.
- James et al. (2020) Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
- Jiang et al. (2018) Jindong Jiang, Lunan Zheng, Fei Luo, and Zhijun Zhang. Rednet: Residual encoder-decoder network for indoor rgb-d semantic segmentation. arXiv preprint arXiv:1806.01054, 2018.
- Kang et al. (2024) Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. ArXiv, 2411.02385, 2024.
- Ke et al. (2021) Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5128–5137, 2021.
- Ke et al. (2024) Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. Arxiv, 2024.
- Kim et al. (2025) Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. ArXiv, 2502.19645, 2025.
- Ko et al. (2023) Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. arXiv preprint arXiv:2310.08576, 2023.
- Koh et al. (2021a) Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021a.
- Koh et al. (2021b) Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14738–14748, 2021b.
- Koh et al. (2023) Jing Yu Koh, Harsh Agrawal, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Simple and effective synthesis of indoor 3d scenes. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 37(1):1169–1178, June 2023. ISSN 2374-3468.
- Kong et al. (2024) Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024.
- Li et al. (2025a) Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph E. Gonzalez, Ion Stoica, Song Han, and Yao Lu. Worldmodelbench: Judging video generation models as world models. ArXiv, 2502.20694, 2025a.
- Li et al. (2025b) Jiaqi Li, Junshu Tang, Zhi-Ting Xu, Longhuang Wu, Yuan Zhou, Shuai Shao, Tianbao Yu, Zhiguo Cao, and Qinglin Lu. Hunyuan-gamecraft: High-dynamic interactive game video generation with hybrid history condition. ArXiv, 2506.17201, 2025b.
- Li et al. (2025c) Runjia Li, Philip H. S. Torr, Andrea Vedaldi, and Tomas Jakab. Vmem: Consistent interactive video scene generation with surfel-indexed view memory. ArXiv, 2506.18903, 2025c.
- Ling et al. (2025) Lu Ling, Chen-Hsuan Lin, Tsung-Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint arXiv:2505.02836, 2025.
- Liu et al. (2018) Huaping Liu, Yupei Wu, and Fuchun Sun. Extreme trust region policy optimization for active object recognition. IEEE Transactions on Neural Networks and Learning Systems, 29(6):2253–2258, June 2018. ISSN 2162-2388.
- Long et al. (2025) Xiaoxiao Long, Qingrui Zhao, Kaiwen Zhang, Zihao Zhang, Dingrui Wang, Yumeng Liu, Zhengjie Shu, Yi Lu, Shouzheng Wang, Xinzhe Wei, Wei Li, Wei Yin, Yao Yao, Jiangtian Pan, Qiu Shen, Ruigang Yang, Xun Cao, and Qionghai Dai. A survey: Learning embodied intelligence from physical simulators and world models. ArXiv, 2507.00917, 2025.
- Lu et al. (2025) TaiMing Lu, Tianmin Shu, Alan Yuille, Daniel Khashabi, and Jieneng Chen. Generative world explorer. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
- Majumdar et al. (2024) Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16488–16498, 2024.
- Morari and H. Lee (1999) Manfred Morari and Jay H. Lee. Model predictive control: Past, present and future. Computers & Chemical Engineering, 23(4):667–682, May 1999. ISSN 0098-1354.
- Neary et al. (2025) Cyrus Neary, Omar G. Younis, Artur Kuramshin, Ozgur Aslan, and Glen Berseth. Improving pre-trained vision-language-action policies with model-based search. ArXiv, 2508.12211, 2025.
- Parker-Holder and Fruchter (2025) Jack Parker-Holder and Shlomi Fruchter. Genie 3: A new frontier for world models, August 2025. URL https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/. Google DeepMind Blog.
- Ramakrishnan et al. (2021) Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M. Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X. Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In Advances in Neural Information Processing Systems (NeurIPS), August 2021.
- Ravi et al. (2024) Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.
- Ren et al. (2025) Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Muller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed world-consistent video generation with precise camera control. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6121–6132, 2025.
- Rocamonde et al. (2023) Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. ArXiv, 2310.12921, 2023.
- Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
- Runway Research (2025) Runway Research. Introducing runway gen-4. https://runwayml.com/research/introducing-runway-gen-4, March 2025. Research announcement, Runway AI, Inc. Accessed: 2025-09-21.
- Sargent et al. (2024) Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. Zeronvs: Zero-shot 360-degree view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9420–9429, 2024.
- Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9339–9347, 2019.
- Seneviratne et al. (2025) Gershom Seneviratne, Jianyu An, Sahire Ellahy, Kasun Weerakoon, Mohamed Bashir Elnoor, Jonathan Deepak Kannan, Amogha Thalihalla Sunil, and Dinesh Manocha. Halo: Human preference aligned offline reward learning for robot navigation. ArXiv, 2508.01539, 2025.
- Seo et al. (2024) Junyoung Seo, Kazumi Fukuda, Takashi Shibuya, Takuya Narihira, Naoki Murata, Shoukang Hu, Chieh-Hsin Lai, Seungryong Kim, and Yuki Mitsufuji. Genwarp: Single image to novel views with semantic-preserving generative warping. In Advances in Neural Information Processing Systems (NeurIPS), November 2024.
- Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
- Voleti et al. (2024) Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitrii Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from a single image using latent video diffusion. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
- Wan et al. (2025) Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.
- Wang et al. (2023) Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
- Wang et al. (2025a) Yiqi Wang, Mrinal Verghese, and Jeff Schneider. Latent policy steering with embodiment-agnostic pretrained world models. ArXiv, 2507.13340, 2025a.
- Wang et al. (2025b) Yuang Wang, Chao Wen, Haoyu Guo, Sida Peng, Minghan Qin, Hujun Bao, Xiaowei Zhou, and Ruizhen Hu. Precise action-to-video generation through visual action prompts. ArXiv, 2508.13104, 2025b.
- Wang et al. (2024) Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. ArXiv, 2402.03681, 2024.
- Xiao et al. (2025) Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan. Worldmem: Long-term consistent world simulation with memory, April 2025.
- Xie et al. (2024) Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency, July 2024.
- Xu et al. (2024) Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: Llm-guided compositional 4d scene generation. arXiv preprint arXiv:2403.16993, 2024.
- Yang et al. (2019) Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David J Crandall, Devi Parikh, and Dhruv Batra. Embodied amodal recognition: Learning to move to perceive objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2040–2050, 2019.
- Yang et al. (2023a) Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023a.
- Yang et al. (2023b) Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2):6, 2023b.
- Yang et al. (2025) Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents, February 2025.
- Yang et al. (2024a) Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a.
- Yang et al. (2024b) Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024b.
- Ye et al. (2025) Deheng Ye, Fangyun Zhou, Jiacheng Lv, Jianqi Ma, Jun Zhang, Junyan Lv, Junyou Li, Minwen Deng, Mingyu Yang, Qiang Fu, Wei Yang, Wenkai Lv, Yangbin Yu, Yewen Wang, Yonghang Guan, Zhihao Hu, Zhongbin Fang, and Zhongqian Sun. Yan: Foundational interactive video generation. ArXiv, 2508.08601, 2025.
- Yin et al. (2023) Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089, 2023.
- Yu et al. (2024) Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. arXiv preprint arXiv:2406.09394, 2024.
- Yu et al. (2023) Jason J. Yu, Fereshteh Forghani, Konstantinos G. Derpanis, and Marcus A. Brubaker. Long-term photometric consistent novel view synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7094–7104, 2023.
- Yu et al. (2025a) Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Context as memory: Scene-consistent interactive long video generation with memory retrieval. ArXiv, 2506.03141, 2025a.
- Yu et al. (2025b) Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos, January 2025b.
- Zhang et al. (2025a) Hongxin Zhang, Zeyuan Wang, Qiushi Lyu, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Behzad Dariush, Kwonjoon Lee, Yilun Du, and Chuang Gan. COMBO: Compositional world models for embodied multi-agent cooperation. In Proceedings of the International Conference on Learning Representations (ICLR), 2025a.
- Zhang et al. (2025b) Ke Zhang, Cihan Xiao, Yiqun Mei, Jiacong Xu, and Vishal M. Patel. Think before you diffuse: Llms-guided physics-aware video generation. ArXiv, 2505.21653, 2025b.
- Zhang et al. (2023) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836–3847, 2023.
- Zhang et al. (2024) Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T. Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. ArXiv, 2404.13026, 2024.
- Zhao et al. (2025) Qi Zhao, Xingyu Ni, Ziyu Wang, Feng Cheng, Ziyan Yang, Lu Jiang, and Bohan Wang. Synthetic video enhances physical fidelity in video synthesis. ArXiv, 2503.20822, 2025.
- Zhen et al. (2025) Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprint arXiv:2504.20995, 2025.
- Zhi et al. (2025) Hongyan Zhi, Peihao Chen, Siyuan Zhou, Dong Yu, Quanxi Wu, Lei Han, and Mingkui Tan. 3dflowaction: Learning cross-embodiment manipulation from 3d flow world model. ArXiv, 2506.06199, 2025.
- Zhou et al. (2025a) Jensen Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthesis with diffusion models. arXiv preprint arXiv:2503.14489, April 2025a.
- Zhou et al. (2025b) Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Peihao Chen, Dit-Yan Yeung, and Chuang Gan. Learning 3d persistent embodied world models. ArXiv, 2505.05495, 2025b.
- Zhu et al. (2025) Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Cong He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Ying Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Lijun Wu, Kai Zhang, Hui Deng, Jiaye Ge, Kaiming Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. ArXiv, 2504.10479, 2025.
World-in-World: World Models in a Closed-Loop World Appendix Contents
1. 1 Introduction
1. 2 World-in-World: a Closed-Loop Interface for Visual World Models
1. 2.1 Unified Strategy for Closed-Loop Online Planning
1. 2.2 Unified Action API
1. 2.3 Comprehensive Embodied Tasks
1. 2.4 Exploiting World Models via Post-Training
1. 3 Evaluation Results and Analysis
1. 3.1 Benchmark Results
1. 3.2 Ablation and Findings
1. 4 Discussion and Future Directions
1. 5 Conclusion
1. A Related Work
1. B Embodied Task Details
1. B.1 Active Recognition (AR)
1. B.2 Image-Goal Navigation (ImageNav)
1. B.3 Active Embodied Question Answering (A-EQA)
1. B.4 Robotic Manipulation
1. B.5 Policies in Embodied Tasks
1. B.6 World Models in Embodied Tasks
1. C Post-Training Recipe for Embodied World Models
1. C.1 Problem Formulation
1. C.2 Post-Training Configuration
1. D Post-Training Dataset Construction
1. D.1 Trajectory Sampling
1. E Visualizing World Model Predictions
1. F Prompt Templates used in World-in-World
1. F.1 Active Recognition (AR) Prompt
1. F.2 Image-Goal Navigation (ImageNav) Prompt
1. F.3 Active Embedded Question Answering (A-EQA) Prompt
1. F.4 Robotic Manipulation Prompt
Appendix A Related Work
Visual generation. Recent advances in diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Rombach et al., 2022; Brooks et al., 2024a) have significantly improved the quality of image generation (Rombach et al., 2022; Zhang et al., 2023) and video generation (Blattmann et al., 2023b, a; Voleti et al., 2024; Xie et al., 2024), enabling temporally coherent and visually rich content synthesis from text prompts or a single image. Image generators (Koh et al., 2021a, 2023; Yu et al., 2023; Sargent et al., 2024; Seo et al., 2024) allow us to synthesize novel views with conditions on targeted viewpoints. Text-to-video generators such as Sora (Brooks et al., 2024a) can generate minutes-long videos from text. Extensions incorporating camera trajectories as conditioning signals (Yin et al., 2023; Bar et al., 2025b; He et al., 2025a, b; Zhou et al., 2025a; Bahmani et al., 2024a) push video generation toward dynamic scenes. However, the absence of a unified conditioning framework hinders integration into downstream applications (e.g., embodied decision making) and prevents fair cross‐method comparisons. Moreover, these generative methods remain passive: generated worlds are treated as static backdrops and evaluated in an open-loop fashion using visual quality score (Huang et al., 2024) or controllability score (Duan et al., 2025). In contrast, our work assesses not only generation quality but also closed-loop task success within a physical simulation.
World models.
Video-based generative models used as world models have demonstrated effectiveness in various settings, including games (Alonso et al., 2024; Yu et al., 2025b; Li et al., 2025b; Ye et al., 2025; He et al., 2025c), manipulation (Du et al., 2023; Ko et al., 2023; Du et al., 2024; Yang et al., 2024a; Zhen et al., 2025), autonomous driving (Gao et al., 2024; Hu et al., 2023), and navigation (Bar et al., 2025b; Wang et al., 2023; Koh et al., 2021a), with extensions to broader embodied tasks (Lu et al., 2025; Zhang et al., 2025a; Long et al., 2025). However, most of these works concentrate on a single task or a narrow domain, and systematic comparisons across multiple embodied tasks under practical closed-loop conditions remain limited. In contrast, our work provides a comprehensive evaluation across four closed-loop embodied tasks, benchmarking the practical utility of diverse world models.
Appendix B Embodied Task Details
This section details the setups for the four embodied tasks evaluated in World-in-World: Active Recognition (AR) in Section ˜ B.1, Image-Goal Navigation (ImageNav) in Section ˜ B.2, Active Embodied Question Answering (A-EQA) in Section ˜ B.3, and Robotic Manipulation in Section ˜ B.4. We also describe the policies used across these tasks in Section ˜ B.5 and summarize the world model details in Section ˜ B.6.
B.1 Active Recognition (AR)
All AR experiments are performed in Habitat-Sim using scenes from the validation split of Matterport3D (Chang et al., 2017). We focus on 29 scenes and curate a subset of 551 challenging episodes adapted from the dataset released by prior work (Fan et al., 2024). Each episode is manually inspected to ensure that it presents either an extreme viewpoint or a heavily occluded target object. These conditions force the agent to actively explore the environment and to rely on its world model for informed decision-making.
Task setup. In the AR setting, the agent is allowed at most $K=10$ decision steps. At each step $t$ , the agent receives an RGB observation $\mathbf{o}_{t}$ that includes a panoramic view and a front view with a horizontal field of view of $90^{\circ}$ . The agent’s output at each step consists of answers to two multiple-choice queries: (i) which object category $\hat{y}_{t}$ matches the target. (ii) which navigation primitive $a_{t}\!∈\!\mathcal{V}$ to execute next. For each query, the VLM selects the token with the highest likelihood, and the associated probability is interpreted as the model’s confidence. After choosing $a_{t}$ , the agent executes the action, acquires the next observation, and proceeds to step $t{+}1$ . The episode terminates when either the step budget $K$ is reached or the confidence of the predicted category $\hat{y}_{t}$ exceeds $95\%$ .
Integrating a world model. Within the AR pipeline, the world model supports decision-making in two complementary ways that mirror the two queries above. For query (i), the model generates synthetic future views that act as auxiliary evidence in addition to the real observation $\mathbf{o}_{t}$ . These additional cues help the agent reason about occlusions, extreme viewpoints, and other distribution shifts that hinder recognition, as illustrated in Figure ˜ 8. For query (ii), agent will first generate $M$ candidate action sequences $\{\mathbf{A}_{t}^{m}\}_{m=1}^{M}$ , each of length $L$ . Given each candidate plan and its corresponding predicted observations, the agent estimates the value of alternative low-level control sequences before committing to an action in the real environment. Unlike a baseline policy that greedily chooses $a_{t+1}$ from $\mathbf{o}_{t}$ alone, the agent equipped with a world model compares simulated outcomes for all candidates and executes the sequence that is expected to yield the most informative next view. When a world model is used, the planner proposes $M=2$ candidate action sequences per step, each with horizon $L=4$ .
<details>
<summary>x8.png Details</summary>
### Visual Description
## Diagram: World Model Enhancements
### Overview
The image illustrates the concept of enhancing a "World Model" for a robot navigating an indoor environment. It compares the robot's planned actions with enhancements in planning and perception. The diagram shows a 3D representation of a room, a robot, and a sequence of images representing the robot's actions with and without enhancements.
### Components/Axes
* **Left Side:** A 3D rendering of a room with a robot. A speech bubble points to a doorway and asks "What is the target object?". An arrow indicates a path for the robot.
* **Center:** A brain icon labeled "World Model" with an arrow pointing to the right.
* **Top Right:** A sequence of four images labeled "Act. 1: Turn Left", "Act. 2: Forward", "Act. 3: Forward", and "Act. 4: Forward". These images are associated with "Planning Enhancement".
* **Bottom Right:** A sequence of four images, mirroring the top sequence, associated with "Perception Enhancement".
* **Arrows:** Two horizontal arrows, one above and one below the image sequences, indicating the progression of actions. The top arrow is associated with "Planning Enhancement" and the bottom arrow with "Perception Enhancement".
### Detailed Analysis
**Left Side (3D Rendering):**
* The 3D rendering shows a room with wooden floors, white walls, and a window.
* A doorway is highlighted in red/pink.
* A blue robot is positioned in the room.
* A speech bubble originating from the robot displays the question "What is the target object?".
* A curved purple arrow indicates a possible path for the robot.
* The floor is colored green along the path.
**Center (World Model):**
* A brain icon with digital circuit patterns represents the "World Model".
* A grey arrow points from the "World Model" to the right, indicating the flow of information or processing.
**Top Right (Planning Enhancement):**
* The sequence of images represents the robot's actions with planning enhancement.
* Each image corresponds to a specific action:
* Act. 1: Turn Left - The doorway is highlighted in red.
* Act. 2: Forward - The doorway is highlighted in red.
* Act. 3: Forward - The doorway is highlighted in red.
* Act. 4: Forward - The doorway is highlighted in red.
* A purple arrow indicates the progression of actions.
**Bottom Right (Perception Enhancement):**
* The sequence of images represents the robot's actions with perception enhancement.
* Each image corresponds to a specific action:
* Act. 1: Turn Left - The doorway is highlighted in red.
* Act. 2: Forward - The doorway is highlighted in red.
* Act. 3: Forward - The doorway is highlighted in red.
* Act. 4: Forward - The doorway is highlighted in red.
* A brown arrow indicates the progression of actions.
### Key Observations
* The diagram contrasts "Planning Enhancement" and "Perception Enhancement" in the context of robot navigation.
* The red highlights in the images likely indicate the robot's focus or target object.
* The "World Model" is central to both planning and perception enhancements.
### Interpretation
The diagram illustrates how a "World Model" can be enhanced to improve a robot's ability to navigate and interact with its environment. The comparison between "Planning Enhancement" and "Perception Enhancement" suggests that both aspects are crucial for effective robot navigation. The question "What is the target object?" highlights the importance of object recognition and goal-oriented behavior in robotics. The diagram implies that by improving the robot's understanding of the world (through the "World Model"), its planning and perception capabilities can be significantly enhanced, leading to more efficient and accurate navigation.
</details>
Figure 8: In AR, the world model supports both queries (perception and planning). In this example, the agent must identify a wooden door that is initially visible only from an extreme viewpoint. For each candidate action sequence, the world model predicts future observations; these forecasts augment the agent’s perception and inform the choice of the next action.
Bounding box annotation. The target object is marked by a red bounding box overlaid on the image. For the current real observation $\mathbf{o}_{t}$ , the box is obtained from Habitat ground-truth annotations. For the predicted frames $\{\hat{\mathbf{o}}_{i}\}_{i=t+1}^{t+L}$ produced by the world model, we apply SAM2 (Ravi et al., 2024) to segment the target, seeding the segmenter with the ground-truth box from the current real observation $\mathbf{o}_{t}$ to maintain correspondence across time.
Metrics. AR performance is reported using two metrics: (1) Success Rate (SR), defined as the fraction of episodes in which the final predicted label $\hat{y}$ matches the ground-truth label $y$ ; and (2) Mean Trajectory Length, defined as the average number of executed actions before the agent either issues its final prediction or exhausts the step budget $K$ .
B.2 Image-Goal Navigation (ImageNav)
Image-Goal Navigation (ImageNav), also known as goal-conditioned visual navigation, requires an embodied agent to reach the target location depicted by a single reference image of the goal. The environment is unknown, so the navigation policy must determine how to explore in order to locate the goal efficiently. To examine how world models can assist, we create 144 ImageNav episodes taken from 87 validation scenes of HM3D (Ramakrishnan et al., 2021).
Task setup. Each episode permits at most $K=20$ decision steps. As in the AR setting, at step $t$ the agent receives an RGB observation $\mathbf{o}_{t}$ comprising a panoramic view and a front view with a horizontal field of view of $90^{\circ}$ . The agent then proposes a sequence of low-level navigation primitives $\mathbf{A}_{t}=[a_{t+1},\,a_{t+2},\,...,\,a_{t+L}]$ with a maximum horizon of $L=5$ . The first $L-2$ primitives from the selected plan are executed in the real environment, after which the agent replans based on the newly acquired observation. An episode is successful if, within the budget of $K$ steps, the agent’s position enters a sphere of radius $R_{g}=0.5,\text{m}$ centered at the location specified by the goal image $\mathbf{g}$ .
Integrating a world model. In ImageNav, the agent answers only the navigation query of which action sequence to execute next; therefore, the world model is used exclusively for planning enhancement. The agent first enumerates several candidate action sequences. For each candidate, the world model predicts the future observations that would follow if the sequence were executed from the current state. The agent then scores each sequence by assessing how informative its predictions are for locating the goal, and selects the sequence with the highest expected utility. When a world model is used, the planner proposes $M=3$ candidate action sequences at each decision step, with horizon $L=5$ . The first $L-2$ actions from the chosen sequence are carried out before the next cycle begins.
Metrics. We report three standard metrics for ImageNav: (1) Success Rate (SR), the fraction of episodes in which the agent reaches the goal within the decision budget; (2) Mean Trajectory Length, the average number of executed actions across all episodes; and (3) Success weighted by Path Length (SPL), which accounts for both success and path efficiency. Formally, for a set of $N$ episodes,
$$
\mathrm{SPL}=\frac{1}{N}\sum_{i=1}^{N}S_{i}\,\frac{L_{i}^{*}}{\max\!\bigl(L_{i},\,L_{i}^{*}\bigr)}\times 100\%,
$$
where $S_{i}∈\{0,1\}$ indicates whether episode $i$ is successful, $L_{i}^{*}$ is the shortest path length from the start position to the goal for episode $i$ , and $L_{i}$ is the actual path length executed by the agent in that episode.
B.3 Active Embodied Question Answering (A-EQA)
Active Embodied Question Answering (A-EQA) tasks an embodied agent with answering open-ended, natural-language questions after actively exploring an environment. The questions span six broad categories that are common in embodied QA: recognizing objects, recognizing object attributes, recognizing object states, localizing objects, performing spatial reasoning, and performing functional reasoning. Our evaluation set contains 184 questions distributed across 54 indoor scenes drawn from the official OpenEQA split (Majumdar et al., 2024) and the validation set of HM3D (Ramakrishnan et al., 2021).
Task setup. In A-EQA, there is no predefined navigation goal, so the agent must design its own exploration strategy to gather sufficient visual evidence for answering the question. At every decision step $t$ , the agent receives a panoramic RGB observation that we decompose into four perspective views, each with a horizontal field of view of $105^{\circ}$ (see Figure ˜ 10). The exploration budget is limited to $250$ low-level actions; a single decision step can comprise multiple low-level actions, depending on the high-level intent. An episode terminates when the budget is exhausted or when the agent outputs a final answer $\hat{y}$ .
For A-EQA, we implement a two-level policy that separates deliberation and control. The high-level planner periodically issues one of two types of commands: (i) a textual instruction (for example, “move to the hallway visible in the front view”), or (ii) the index of a landmark object detected in the current panorama. Once a high-level command is produced, execution is delegated to the low-level controller. If the command specifies a landmark, the controller uses depth data together with a custom pathfinder to plan and follow a route to that landmark. If the command is a textual instruction, the controller generates a sequence of low-level actions to carry out the instruction. This planner-controller loop continues until either the $250$ atomic actions are consumed or the high-level planner decides to emit the final answer $\hat{y}$ .
<details>
<summary>x9.png Details</summary>
### Visual Description
## Diagram: Robot Navigation Plan
### Overview
The image depicts a top-down view of a house interior, illustrating a robot's navigation plan. The robot's potential paths ("Candidate plans from WMs") and the actual path taken ("Executed plan") are overlaid on the image of the house. The robot is tasked with answering a question about the number of cushions on a red sofa.
### Components/Axes
* **Legend (Top-Right)**:
* Purple Arrow: Candidate plans from WMs (World Models)
* Blue Chevron: Executed plan
* **Environment**: The image shows a house interior including a kitchen, dining area, living room, and hallway.
* **Robot**: A light blue robot is shown at multiple locations, representing its potential and actual positions.
* **Question Bubble**: A speech bubble near the robot states: "How many cushions are on the red sofa?"
* **Red Sofa**: A sofa is outlined in orange in the living room.
### Detailed Analysis
* **Candidate Plans**: Purple arrows indicate the robot's potential paths. These paths originate from different locations in the house.
* One set of candidate plans originates from the kitchen area.
* Another set originates from the dining area.
* A third set originates from the hallway.
* **Executed Plan**: Blue chevrons show the actual path taken by the robot. The robot moves from the kitchen, through the dining area and hallway, and into the living room.
* **Robot Positions**: The robot is shown at the starting point in the kitchen, in the dining area, in the hallway, and near the sofa in the living room.
* **Sofa**: The sofa in the living room is outlined in orange. It appears to have multiple cushions on it.
### Key Observations
* The robot explores multiple potential paths before executing a specific plan.
* The executed plan takes the robot from the kitchen to the living room.
* The robot's task is to answer a question about the sofa in the living room.
### Interpretation
The diagram illustrates a robot's navigation and planning process within a home environment. The robot uses world models (WMs) to generate candidate plans, and then executes one of these plans to reach a specific location (the sofa) to answer a question. The multiple candidate plans suggest that the robot is capable of considering different routes and making decisions based on its understanding of the environment. The orange outline around the sofa highlights the object of interest for the robot's task.
</details>
Figure 9: Overview of our embodied closed-loop evaluation for A-EQA. For each question, the high-level planner proposes multiple candidate action plans and queries the world model to generate the corresponding future observations. The agent then evaluates each plan together with its predicted observations and selects the plan that maximizes the expected reward before executing it in the environment.
Integrating a world model. In A-EQA, the world model is primarily used to strengthen the high-level planner. At each high-level decision point, the planner samples $M$ candidate action plans and queries the world model to produce the corresponding predicted observations, as illustrated in Figure ˜ 9. The agent then evaluates each plan-observation pair $(\hat{\mathbf{A}}_{t}^{(m)},\,\hat{\mathbf{O}}_{t}^{(m)})$ and chooses the plan that maximizes the estimated reward under the current question context. This differs from the AR setting, where perception and planning are evaluated through two separate queries. In A-EQA, the high-level planner must both design a long-horizon exploration sequence and decide when to stop exploring to output a final answer $\hat{y}$ . Consequently, the world model supports a single unified query: the predicted observations simultaneously refine the agent’s understanding of the scene and provide forecasts for scoring alternative exploration plans. When a world model is enabled, the planner proposes $M=3$ candidate sequences per step, each with horizon $L=14$ . Unlike AR or ImageNav, only the terminal predicted observation at step $L$ is returned to the high-level planner for scoring, rather than the full rollout over all $L$ steps.
<details>
<summary>x10.png Details</summary>
### Visual Description
## Image: Interior Views
### Overview
The image presents four different views (Front, Left, Right, Back) of an interior space, likely a room or a section of a house. Each view provides a perspective of the same area from a different angle.
### Components/Axes
* **Titles:**
* Curr. View: Front (top-left)
* Curr. View: Left (top-right)
* Curr. View: Right (bottom-left)
* Curr. View: Back (bottom-right)
### Detailed Analysis
* **Curr. View: Front:** Shows an entryway with a console table, a mirror, and a view into another room. The flooring appears to be dark wood with a light-colored rug.
* **Curr. View: Left:** Shows a hallway with a closed double door. The wall color is light.
* **Curr. View: Right:** Shows a corner of a room with a small table and a decorative item on the wall. The flooring matches the "Front" view.
* **Curr. View: Back:** Shows a room with a window, a desk, and various decorative items. There is a chair and a small table visible.
### Key Observations
* The four views are interconnected, showing different perspectives of the same space.
* The room appears to be decorated in a modern style with neutral colors.
* The flooring is consistent across the "Front" and "Right" views.
### Interpretation
The image provides a 360-degree view of an interior space by presenting four different perspectives. This allows the viewer to understand the layout and design of the room more comprehensively than a single image would. The consistent design elements across the views suggest a cohesive aesthetic throughout the space.
</details>
Figure 10: Illustration of the Set-of-Marks (SoM) representation that encodes candidate navigable directions. The high-level planner chooses among these discrete landmarks when constructing candidate action plans.
Landmark detection and labeling. Landmark objects are detected by first running YOLO-World to obtain bounding boxes and then applying SAM2 to derive instance masks (Ravi et al., 2024; Cheng et al., 2024). This detection pipeline follows the Set-of-Marks (SoM) strategy (Yang et al., 2023a) shown in Figure ˜ 10 and provides a discrete set of navigable targets for high-level planning.
Metrics. A-EQA performance is evaluated with three metrics. (1) Answering Score: a large language model (e.g., GPT-4o) compares the agent’s final answer $\hat{y}$ to the ground-truth answer $y$ and assigns a raw score in $[1,5]$ , where $5$ indicates a perfect match. We average the raw score across episodes and then linearly map it to $[0,100]$ . (2) Mean Trajectory Length. This is the average travel distance the agent covers before either producing its final answer or exhausting the step budget $K$ , lower is better. (3) Success weighted by Path Length (SPL): this metric rewards both answer quality and navigation efficiency. For episodes in which the agent fails to return an answer, we fall back to its blind LLM variant and set the SPL contribution to zero. Formally,
$$
\text{SPL}_{\text{A-EQA}}=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{\sigma_{i}-1}{4}\right)\frac{L_{i}^{*}}{\max\!\bigl(L_{i},\,L_{i}^{*}\bigr)}\times 100\%,
$$
where $N$ is the number of evaluation episodes, $\sigma_{i}∈[1,5]$ denotes the raw Answering Score for episode $i$ , $L_{i}^{*}$ denotes the shortest-path length from the start to a viewpoint that affords a correct answer, and $L_{i}$ denotes the actual path length executed by the agent in episode $i$ . A higher value indicates both more accurate answering and more efficient exploration.
B.4 Robotic Manipulation
We study whether world models can improve low-level manipulation, which is a core capability for embodied agents. Our evaluation covers four robotic manipulation tasks in RLBench (James et al., 2020): Push Buttons, Slide Block to Color Target, Insert onto Square Peg, and Stack Cups. RLBench is a widely used benchmark for robot learning. Each episode provides a natural-language instruction that specifies the task objective, and the agent must control a 7-DoF robotic arm to satisfy that objective. We prepare a total of 200 evaluation episodes, with 50 episodes for each task.
Task setup. At each decision step $t$ , the agent receives an observation $\mathbf{o}_{t}$ and proposes an action sequence $\mathbf{A}_{t}=\bigl[\mathbf{a}_{t+1},\,\mathbf{a}_{t+2},\,...,\,\mathbf{a}_{t+L}\bigr]$ , where each low-level action is parameterized as $\mathbf{a}_{t}=[x,\,y,\,z,\,\text{roll},\,\text{pitch},\,\text{yaw},\,\text{gripper}]$ . We consider two base policy settings with different horizons: $L=5$ for a VLM base policy that emits discrete actions, and $L=50$ for a 3D diffusion base policy that emits continuous actions. An episode is counted as a success if the specified goal $\mathbf{g}$ is achieved within the step budget $K$ .
<details>
<summary>x11.png Details</summary>
### Visual Description
## Image Analysis: Robotic Task Simulation
### Overview
The image depicts a simulated robotic task environment. It includes a robotic arm interacting with colored blocks on a surface. The image provides the task objective and auxiliary information about the object positions.
### Components/Axes
* **Task Objective (Top-Left)**: A text box stating the objective: "Slide the red block to magenta target."
* **Auxiliary Information (Left)**: A text box providing object positions in a dictionary format.
* **Simulation Environment (Right)**: A 3D simulation showing a robotic arm, colored blocks (red, blue, green, yellow, magenta), and a wooden surface. A coordinate system (red, green, blue axes) is visible.
### Detailed Analysis
* **Task Objective**: The robot's goal is to manipulate the red block and move it to the magenta target.
* **Object Positions**: The auxiliary information lists the positions of five objects. The positions are represented as 3D coordinates (x, y, z).
* Object 1 (Green): [45, 13, 18]
* Object 2 (Blue): [72, 20, 18]
* Object 3 (Red): [50, 42, 17]
* Object 4 (Magenta): [36, 42, 18]
* Object 5 (Yellow): [69, 39, 15]
* **Simulation Environment**:
* The robotic arm is positioned above the blocks.
* The blocks are placed on a wooden surface.
* The coordinate system indicates the orientation of the environment.
### Key Observations
* The task is clearly defined: moving the red block to the magenta target.
* The object positions are provided as numerical data, which could be used for planning or analysis.
* The simulation environment provides a visual representation of the task.
### Interpretation
The image presents a simulated robotic task, providing both a visual representation and numerical data about the environment. The task objective is straightforward, and the object positions are given, allowing for the development and testing of robot control algorithms. The simulation could be used for training robots to perform manipulation tasks in a controlled environment.
</details>
Figure 11: Illustration of the auxiliary information provided to the VLM policy. The objects are marked with indices, and their positions are given to the VLM to facilitate decision-making.
When a VLM is the base policy, directly producing precise low-level controls is challenging for current VLMs. Following (Yang et al., 2025), we therefore introduce two enhancements. First, we discretize the action space by dividing the position components $(x,y,z)$ into 100 bins and the orientation components $(\text{roll},\text{pitch},\text{yaw})$ into 120 bins. Second, we augment the observations with object index markers and provide precise object poses for indexed objects so that the VLM can directly access spatial information during planning (shown in Figure ˜ 11). Under this configuration, the manipulation policy is allowed at most $K=15$ low-level action steps per episode. In contrast, when using a 3D diffusion policy (Ke et al., 2024) as the base policy, the controller naturally generates continuous low-level actions, so we do not apply the discretization or the additional indexing enhancements. In this configuration, the manipulation policy is permitted at most $K=8$ macro decision steps per episode.
Integrating a world model. As in ImageNav, we use the world model exclusively for planning enhancement. The agent executes a propose, simulate, and revise loop so that it can reason about the consequences of alternative plans before applying any action in the real environment. At each decision step, the planner proposes $M=5$ candidate action sequences. When the length of a candidate sequence is shorter than the world model’s required action-conditioning length, the unified action API linearly interpolates the sequence to the required length. Conversely, when the candidate sequence is longer than required, the unified action API uniformly samples actions along the sequence to match the world model’s input length. The planner then evaluates the simulated outcomes and selects the sequence with the highest expected reward, and the loop repeats with updated observations.
Metrics. We report two standard metrics for manipulation tasks: (1) Success Rate (SR), the fraction of episodes in which the agent reaches the goal within the decision budget; and (2) Mean Trajectory Length, the average number of decision steps across all episodes.
B.5 Policies in Embodied Tasks
There are three types of policies in paper: the base policy, the proposal policy, and the revision policy. The base policy is an independent policy that interacts with the environment without using a world model, and when a world model is enabled, it is always the same as the corresponding proposal policy. When a world model is integrated, the proposal policy generates multiple candidate action sequences at each decision step, and the revision policy evaluates these candidates and selects one based on the predicted rollouts produced by the world model.
In our experiments, we employ two types of base policies for AR and ImageNav: a VLM policy and a heuristic policy. For the VLM policy, we use Qwen2.5-VL-72B-Instruct-AWQ (Bai et al., 2025) as the default base policy and as the proposal policy when integrated with a world model to answer queries. For the heuristic policy, we implement a primitive action sampling mechanism that draws actions from the action space according to the previously executed actions and a set of handcrafted rules. Concretely, if there exists a previous action, then the next action must not be its inverse (for example, a turn_left cannot be immediately followed by a turn_right). In addition, we prevent excessively long subsequences of turns in the same direction by capping the maximum number of consecutive turns to four. These rules help the heuristic policy to avoid redundant back-and-forth movements and to explore the environment effectively.
For manipulation tasks, we likewise consider two base policies: a VLM policy and a 3D diffusion policy. The VLM policy remains Qwen2.5-VL-72B-Instruct-AWQ by default. The 3D diffusion policy follows 3D Diffuser Actor (Ke et al., 2024); we train it using the authors’ official code. To encourage diverse action trajectory proposals, we drop its text input and modify the task-definition scripts so that task variants occur with equal frequency during training. For each manipulation task, the diffusion policy is trained on 120 demonstrations and used as the proposal policy to generate short-horizon 7-DoF gripper action sequences within the planning loop.
For the revision policy in our closed-loop online planning, we use the same VLM as the proposal policy by default to score candidate plans and to select the decision that maximizes the expected task reward. For ablations, we also replace Qwen2.5-VL-72B-Instruct-AWQ with InternVL3-78B-AWQ (Zhu et al., 2025) as the VLM policy; results in Table ˜ 5 show that world model integration consistently improves performance regardless of the specific VLM used.
Table 5: Task performance for InternVL3 variants with and without a world model. Higher SR %, SPL %, and Ans. Score are better; lower Mean Traj. is better.
| Model Details | AR | ImageNav | A-EQA | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model Type | Method | SR $\uparrow$ | Mean Traj. $\downarrow$ | SR $\uparrow$ | Mean Traj. $\downarrow$ | SPL $\uparrow$ | Ans. Score $\uparrow$ | Mean Traj. $\downarrow$ | SPL $\uparrow$ |
| Base Policy | InternVL3 (w/o WM) | 49.91 | 7.06 | 13.19 | 60.30 | 7.46 | 47.28 | 20.45 | 31.22 |
| + Image Gen. | SVD $\dagger$ | 55.72 | 5.37 | 40.97 | 52.50 | 26.26 | 47.13 | 16.78 | 34.54 |
B.6 World Models in Embodied Tasks
Output format. The world models evaluated in our framework fall into two categories according to their native output format: perspective models and panoramic models. Perspective models, such as NWM (Bar et al., 2025a), LTX-Video (HaCohen et al., 2024), and Wan2.1 (Wan et al., 2025), generate frames in a perspective view. Panoramic models, including PathDreamer (Koh et al., 2021b), SE3DS (Koh et al., 2023), and our post-trained variants, produce equirectangular panoramas. For integration into our closed-loop pipeline, panoramic outputs are decomposed into perspective views, which are then supplied to the agent. In A-EQA, the agent consumes four principal perspective views (front, left, right, back) when they are available. In AR, the agent uses the view that contains the target bounding box; if the box is not visible, we discard the generated frames until the predicted box (from SAM2) enters the field of view. Unless otherwise specified, each perspective view image is resized to $384× 384$ pixels before being passed to the agent.
Input format. Panoramic models are conditioned on an equirectangular panorama at a resolution of $576× 1024$ pixels. Perspective models, when possible, take the current front-view observation with resolution $480× 480$ as input. Some models require additional modalities. SE3DS expects a depth map, while PathDreamer requires both depth and a per-pixel semantic label map. For all depth-aware models, we provide ground-truth depth from Habitat. For PathDreamer, the initial semantic map is obtained by running a pretrained RedNet (Jiang et al., 2018) on the initial RGB-D frame to produce per-pixel labels that match the required input specification.
Appendix C Post-Training Recipe for Embodied World Models
In this section, we describe how an off-the-shelf video generation model is adapted, via post-training, into an action-controllable world model suitable for embodied tasks. We first formalize the learning objective and the action-observation alignment (Section ˜ C.1), and then detail the concrete post-training setup used for tasks in Habitat-Sim and for Robotic Manipulations (Section ˜ C.2).
C.1 Problem Formulation
Let $\mathbf{x}_{1}\!∈\!\mathbb{R}^{3× H× W}$ denotes the initial RGB frame that conditions the generation process. Our goal is to synthesize an $N$ -frame video $\mathbf{X}=\bigl[\mathbf{x}_{1},\,\mathbf{x}_{2},\,...,\,\mathbf{x}_{N}\bigr]∈\mathbb{R}^{3× H× W× N},$ where $\mathbf{X}$ represents a plausible sequence of future observations after executing a sequence of actions $\mathbf{A}=\bigl[a_{1},\,a_{2},\,...,\,a_{N}\bigr].$
For tasks in Habitat-Sim, we adopt a discrete action space with $a_{i}∈\mathcal{V}$ , where $\mathcal{V}$ is a finite set of navigation primitives (e.g., Forward, Turn-Left, Turn-Right, Stop). For manipulation, we use a continuous action space with $a_{i}∈\mathbb{R}^{7}$ , corresponding to 7-DoF end-effector poses. Actions in Habitat-Sim specify relative transformations between consecutive observations. Since $a_{i}$ maps $\mathbf{x}_{i-1}$ to $\mathbf{x}_{i}$ , no action precedes the first frame. To maintain a one-to-one alignment between frames and actions, we prepend a special token and set $a_{1}=a_{\text{Null}}$ . In contrast, for manipulation tasks during post-training, actions are absolute end-effector poses expressed in the world frame, so there is naturally a one-to-one correspondence between actions and frames.
We formulate future-observation synthesis with the world model $g_{\boldsymbol{\theta}}$ by learning the conditional distribution $p_{\boldsymbol{\theta}}\bigl(\mathbf{X}\,\big|\,\mathbf{x}_{1},\,C(\mathbf{A})\bigr),$ where $C(\mathbf{A})$ denotes the control signal emitted by the unified action API. This API converts the native action sequence $\mathbf{A}$ into the conditioning interface expected by the pretrained video generator (for example, a text prompt, a camera trajectory, or a sequence of low-level controls). This formulation yields action-conditioned rollouts that evolve from the initial frame $\mathbf{x}_{1}$ according to the specified action sequence, thereby aligning the pretrained model with the domain distribution and action space of the target embodied tasks.
C.2 Post-Training Configuration
For tasks in Habitat-Sim, we use panoramic observations as both the input and the output of the video generators. We fine-tune the pretrained video generation models at a resolution of $576× 1024$ and train them to predict $N$ future frames on our self-collected panoramic action-observation corpus from Habitat-Sim. In these tasks, the action space is discrete and comprises four navigation primitives: Forward 0.2 m, Turn_Left 22.5 ∘, Turn_Right 22.5 ∘, and Stop. For manipulation tasks, we use front-view observations as both the input and the output of the video generators. We fine-tune the pretrained video generation models at a resolution of $480× 480$ (Cosmos-Predict2) or $448× 448$ (SVD) and train them to predict $N$ future frames with continuous 7-DoF end-effector poses as conditioning.
Unless otherwise stated, post-training uses 40K sampled instances for the Habitat-Sim tasks and for the manipulation tasks. All models are initialized from their official pretrained weights and adapted on the corresponding dataset for one epoch. We rely on the official implementations and the recommended hyperparameters for fine-tuning whenever available; specific post-training details of various world models are summarized below in Tables ˜ 6 and 7.
Table 6: Post-trained (action-conditioned) world models used in our experiments, with repositories and training configurations.
| World Model | Domain | Repository | Frames ( $N$ ) | Train Res. | Notes |
| --- | --- | --- | --- | --- | --- |
| Post-training on Habitat-Sim data | | | | | |
| Cosmos-Predict2 $\dagger$ (Agarwal et al., 2025) | Habitat-Sim | github.com/nvidia-cosmos/cosmos-predict2 | 13 | $576× 1024$ | Official repo |
| LTX-Video $\dagger$ (HaCohen et al., 2024) | Habitat-Sim | github.com/Lightricks/LTX-Video-Trainer | 17 | $576× 1024$ | Official repo |
| Wan2.1 $\dagger$ (Wan et al., 2025) | Habitat-Sim | github.com/modelscope/DiffSynth-Studio | 13 | $576× 1024$ | Official repo |
| Wan2.2 (5B) $\dagger$ (Wan et al., 2025) | Habitat-Sim | github.com/modelscope/DiffSynth-Studio | 13 | $576× 1024$ | Official repo |
| Wan2.2 (A14B) $\dagger$ (Wan et al., 2025) | Habitat-Sim | github.com/modelscope/DiffSynth-Studio | 13 | $576× 1024$ | Official repo |
| SVD $\dagger$ (Blattmann et al., 2023a) | Habitat-Sim | github.com/pixeli99/SVD_Xtend | 14 | $576× 1024$ | Self-adapted based on repo |
| Post-training on manipulation data | | | | | |
| Cosmos-Predict2 $\dagger$ (Agarwal et al., 2025) | Manipulation | github.com/nvidia-cosmos/cosmos-predict2 | 13 | $480× 480$ | Official repo |
| SVD $\dagger$ (Blattmann et al., 2023a) | Manipulation | github.com/pixeli99/SVD_Xtend | 14 | $448× 448$ | Self-adapted based on repo |
Table 7: All the world models and their details in World-in-World. “ $\dagger$ ” denotes post-trained (action-conditioned) variants.
| World Model | Model Type | Control Type | Input Type | #Param. |
| --- | --- | --- | --- | --- |
| Zero-shot (no post-training) | | | | |
| PathDreamer (Koh et al., 2021b) | Image Gen. | Viewpoint | RGB-D; Pano | 0.69B |
| SE3DS (Koh et al., 2023) | Image Gen. | Viewpoint | RGB-D; Pano | 1.1B |
| NWM (Bar et al., 2025a) | Video Gen. | Trajectory | RGB | 1B |
| SVD (Blattmann et al., 2023a) | Video Gen. | Image | RGB | 1.5B |
| LTX-Video (HaCohen et al., 2024) | Video Gen. | Text | RGB | 2B |
| Hunyuan (Kong et al., 2024) | Video Gen. | Text | RGB | 13B |
| Wan2.1 (Wan et al., 2025) | Video Gen. | Text | RGB | 14B |
| Wan2.2 (Wan et al., 2025) | Video Gen. | Text | RGB | 5B |
| Wan2.2 (Wan et al., 2025) | Video Gen. | Text | RGB | A14B |
| Cosmos-Predict2 (Agarwal et al., 2025) | Video Gen. | Text | RGB | 2B |
| Runway Gen4 (Runway Research, 2025) | Video Gen. | Text | RGB | – |
| Post-trained (action-conditioned) | | | | |
| SVD $\dagger$ (Blattmann et al., 2023a) | Video Gen. | Action | RGB; Pano | 1.5B |
| LTX-Video $\dagger$ (HaCohen et al., 2024) | Video Gen. | Action | RGB; Pano | 2B |
| Wan2.1 $\dagger$ (Wan et al., 2025) | Video Gen. | Action | RGB; Pano | 14B |
| Wan2.2 $\dagger$ (Wan et al., 2025) | Video Gen. | Action | RGB; Pano | 5B |
| Wan2.2 $\dagger$ (Wan et al., 2025) | Video Gen. | Action | RGB; Pano | A14B |
| Cosmos-Predict2 $\dagger$ (Agarwal et al., 2025) | Video Gen. | Action | RGB; Pano | 2B |
In Table ˜ 8, we summarize the computational resources required to post-train each world model on $\sim$ 40k domain-specific clips collected from Habitat-Sim. This post-training stage is intentionally lightweight and is several orders of magnitude less expensive than full pretraining. For 14B-parameter variants, we adopt LoRA fine-tuning to reduce GPU memory usage, while all other models are fine-tuned with full weights.
Table 8: Post-training resources for $\sim$ 40k domain clips per model. The procedure is lightweight and substantially cheaper than full retraining.
| Model | Model Size | GPU Memory (peak) | H100 GPU-hours |
| --- | --- | --- | --- |
| SVD | 1.5B | 84 GB | 29 |
| LTX-Video | 2B | 61 GB | 5 |
| Wan2.1 | 14B | 57 GB | 74 |
| Cosmos-Predict2 | 2B | 71 GB | 15 |
Appendix D Post-Training Dataset Construction
For the post-training dataset used in manipulation tasks, we rely on the official RLBench codebase (James et al., 2020) to generate data. Specifically, we produce 200 demonstrations for each manipulation task. Each demonstration includes approximately 150 front-view RGB observations together with the corresponding sequence of 7-DoF end-effector poses. These pose sequences are aligned with the image observations and serve as the action labels during post-training. For the tasks evaluated in Habitat-Sim (Savva et al., 2019), there is no existing pipeline for constructing a large-scale dataset of panoramic action trajectories. To address this gap, we build a comprehensive post-training dataset by sampling action trajectories from the training splits of indoor scenes in HM3D (Ramakrishnan et al., 2021) and Matterport3D (Chang et al., 2017). Our trajectory sampling procedure is described in Section ˜ D.1. A summary of the resulting dataset statistics is provided in Table ˜ 9.
D.1 Trajectory Sampling
| Statistic | Value |
| --- | --- |
| Number of scenes | 858 |
| Panorama RGB frames | 763,724 |
| Action trajectories | 439,213 |
| Depth recorded | ✓ |
| Camera poses recorded | ✓ |
| Low-level actions recorded | ✓ |
Table 9: Statistics of the post-training panoramic dataset.
Our aim is to record physically reasonable trajectories that resemble the exploration behavior of real agents in indoor spaces. We follow three guiding principles: (i) Diversity. The trajectories should cover many viewpoints and actions so that the model sees the scene from different perspectives and motion patterns. (ii) Plausibility. The paths must respect physical constraints; the agent must not move through walls or other solid objects. (iii) Manageability. The data should be free of excessive redundancy so that training remains balanced and efficient.
Algorithm 1 Three-stage construction of the post-training panoramic dataset
Input: scene mesh $\mathcal{S}$ , waypoint density $\rho$ , weight $\alpha$ , filter radius $r_{\mathrm{f}}$ , leaf ratio $\eta$ Output: set of panoramic trajectories $\mathcal{T}$
1: // Stage 1: waypoint selection
2: $S←\mathrm{Area}(\mathcal{S})$
3: $N_{\mathrm{wp}}←\max\!\bigl(1400,\lfloor\rho S\rfloor\bigr)$ $\triangleright$ target number of points
4: $\mathcal{P}←\textsc{UniformSampleNavigable}(\mathcal{S},N_{\mathrm{wp}})$
5: build geodesic distance matrix $D$ on $\mathcal{P}$
6: for all $p_{i}∈\mathcal{P}$ do $\triangleright$ leaf score $s(i)$
7: $\text{ecc}(i)←\max_{j}D_{ij}$
8: $\bar{d}(i)←\frac{1}{|\mathcal{P}|-1}\sum_{j}D_{ij}$
9: $s(i)←\text{ecc}(i)+\alpha\,\bar{d}(i)$
10: sort $\mathcal{P}$ by $s(i)$ in descending order $\triangleright$ higher $s(i)$ = more peripheral
11: $\mathcal{W}←\varnothing$
12: for all $p_{i}$ in sorted $\mathcal{P}$ do $\triangleright$ radius-based greedy pruning
13: if $∀ w∈\mathcal{W}:D_{iw}≥ r_{\mathrm{f}}$ then
14: $\mathcal{W}←\mathcal{W}\cup\{p_{i}\}$
15: // Stage 2: path generation
16: $\mathcal{T}←\varnothing$
17: $N_{\text{leaf}}←\lceil\eta N_{\mathrm{wp}}\rceil$
18: $\mathcal{U}←\mathcal{W}[{:}N_{\text{leaf}}]$ $\triangleright$ unvisited waypoints
19: $c←\textsc{RandomSample}(\mathcal{U})$ $\triangleright$ random start
20: while $\mathcal{U}≠\varnothing$ do
21: $n←\arg\min_{w∈\mathcal{U}\setminus\{c\}}\textsc{GeodesicDist}(c,w)$
22: $\tau←\textsc{ShortestPath}(c,n)$ $\triangleright$ Habitat planner
23: record panoramic RGB-D frames along $\tau$ and append to $\mathcal{T}$
24: // Stage 3: waypoint dynamic update
25: for all $w∈\mathcal{W}$ do
26: if $∃ m∈\tau:\textsc{GeodesicDist}(m,w)<r_{\mathrm{f}}$ then
27: $\mathcal{W}←\mathcal{W}\setminus\{w\}$ $\triangleright$ mark as visited
28: recompute $s(·)$ on updated $\mathcal{W}$ , then sort in descending order
29: $\mathcal{U}←\mathcal{W}[{:}N_{\text{leaf}}]$ $\triangleright$ refresh unvisited set
30: $c← n$
31: return $\mathcal{T}$
We implement these principles with a sampling procedure shown in Algorithm ˜ 1 and described below.
1. Waypoint selection. For a scene of floor area $S$ we set the waypoint density to $\rho=4\ \text{m}^{-2}$ and draw
$$
N_{\mathrm{wp}}=\max\!\bigl(1400,\lfloor\rho S\rfloor\bigr)
$$
navigable points $\mathcal{P}$ uniformly across the scene. We construct a complete graph whose edge weights $D_{ij}$ are the geodesic distances between points $p_{i}$ and $p_{j}$ . Each vertex $i$ is assigned a leaf score
$$
s(i)=\operatorname{ecc}(i)+\alpha\,\bar{d}(i),
$$
where $\operatorname{ecc}(i)=\max_{j}D_{ij}$ is the eccentricity, $\bar{d}(i)=(|\mathcal{P}|-1)^{-1}\sum_{j}D_{ij}$ is the mean geodesic distance to all other vertices, and $\alpha=1.7$ . Sorting vertices by $s(i)$ in descending order, we greedily build a waypoint set $\mathcal{W}$ that respects a minimum spacing of $r_{\mathrm{f}}=3$ m: a candidate $v$ is accepted only if $D_{vj}≥ r_{\mathrm{f}}$ for every waypoint $j$ already chosen.
1. Path generation. We maintain a list $\mathcal{U}$ of unvisited waypoints, initialized with the top $N_{\text{leaf}}$ vertices of $\mathcal{W}$ . Starting from a random waypoint $c∈\mathcal{U}$ , we repeatedly move to the nearest unvisited waypoint
$$
n=\arg\min_{w\in\mathcal{U}\setminus\{c\}}\textsc{GeodesicDist}(c,w),
$$
and use the Habitat path-finder to compute the shortest collision-free path $\tau$ from $c$ to $n$ . Panoramic RGB-D frames are recorded at every step along $\tau$ and appended to the trajectory set $\mathcal{T}$ .
1. Waypoint dynamic update. After each segment $\tau$ we label any waypoint $w$ with GeodesicDist(m,w) $<r_{\mathrm{f}}$ for some path point $m∈\tau$ as visited and remove it from $\mathcal{W}$ . We then recompute $s(·)$ on the remaining vertices, resort $\mathcal{W}$ , and refresh the unvisited list
$$
\mathcal{U}\leftarrow\mathcal{W}[{:}N_{\text{leaf}}].
$$
The next segment starts from $c← n$ , and the loop continues until $\mathcal{U}$ is empty. This dynamic reselection guarantees that peripheral regions are covered while avoiding redundant sampling in interior corridors.
<details>
<summary>x12.png Details</summary>
### Visual Description
## Diagram: Two Maze Layouts with Waypoints
### Overview
The image presents two distinct maze layouts, each marked with waypoints. The left maze features red waypoints, while the right maze uses orange waypoints. Both mazes are rendered in gray against a light gray background, and each has a blue arrow indicating a starting point. The text "waypoint_idx: 0, step: 0" appears above each maze.
### Components/Axes
* **Maze Layouts:** Two different maze configurations, each with a unique arrangement of walls and open spaces.
* **Waypoints:** Discrete points marked within each maze, represented by colored dots (red on the left, orange on the right).
* **Starting Point:** A blue arrow indicates the initial position within each maze.
* **Text Labels:** "waypoint_idx: 0, step: 0" is displayed above each maze, indicating the initial waypoint index and step.
### Detailed Analysis
* **Left Maze:**
* Waypoint Color: Red
* Number of Waypoints: Approximately 15
* Starting Point: Located near the bottom-left corner.
* General Trend: Waypoints are distributed throughout the maze, appearing at various intersections and dead ends.
* **Right Maze:**
* Waypoint Color: Orange
* Number of Waypoints: Approximately 10
* Starting Point: Located near the center of the maze.
* General Trend: Waypoints are distributed throughout the maze, appearing at various intersections and dead ends.
* **Text Labels:**
* "waypoint_idx: 0, step: 0" appears above both mazes. This suggests that both mazes are at the initial state of a waypoint-based navigation or pathfinding process.
### Key Observations
* The two mazes have different layouts and a different number of waypoints.
* The starting point is in a different location in each maze.
* The waypoint colors are distinct for each maze (red vs. orange).
* The "waypoint_idx" and "step" are both initialized to 0 in both mazes.
### Interpretation
The image likely represents the initial state of a pathfinding or navigation algorithm being applied to two different maze environments. The waypoints could represent key locations or decision points within each maze. The "waypoint_idx" and "step" parameters suggest that the algorithm will iterate through the waypoints to find a path from the starting point to a goal (which is not explicitly shown in the image). The different maze layouts and waypoint configurations indicate that the algorithm is being tested or demonstrated in diverse environments.
</details>
Figure 12: Top-down visualization of sampled waypoints in a scene. Red (left) and yellow (right) dots are the final waypoints after radius-based pruning. The proposed strategy places waypoints throughout peripheral regions while avoiding redundant interior points, yielding diverse and spatially balanced trajectories.
Compared with random sampling of start and end waypoints, the above strategy distributes waypoints across peripheral areas such as bedrooms while avoiding redundant paths through interior corridors. The resulting dataset therefore offers a balanced and diverse set of viewpoints for post-training (see Figure ˜ 12).
Appendix E Visualizing World Model Predictions
We illustrate the behavior of several world models under identical action sequences generated by the planner. Figure ˜ 13 and Figure ˜ 14 show example rollouts in which the action sequence consists solely of Forward actions; a well-behaved model should yield pure forward motion. The figures contrast models that follow the commands with those that drift or hallucinate, underscoring the importance of precise action control for downstream embodied tasks. For further examples of good and bad predictions, see Figures ˜ 15, 16, 17 and 18.
<details>
<summary>x13.png Details</summary>
### Visual Description
## Image: Action Control Examples
### Overview
The image presents examples of "Good" and "Bad" action control, specifically for the "Forward" action. It shows a sequence of images depicting a hallway, with the "Good Example" row showing a consistent and clear progression forward, while the "Bad Examples" rows show distorted or inconsistent movement.
### Components/Axes
* **Title:** Action Control: Forward
* **Rows:**
* Good Example: (Top row)
* Bad Examples: (Middle and Bottom rows)
* **Content:** Each row contains a sequence of images depicting a hallway.
### Detailed Analysis
* **Good Example:** The top row shows a sequence of images that smoothly transition forward through a hallway. The perspective and lighting are consistent across the images, creating a clear sense of movement.
* **Bad Examples:** The middle and bottom rows show sequences of images that are distorted or inconsistent. The perspective shifts abruptly, the lighting changes erratically, and the overall sense of forward movement is disrupted. The images in these rows appear blurry or warped.
### Key Observations
* The "Good Example" demonstrates a smooth and consistent forward movement.
* The "Bad Examples" demonstrate distorted, inconsistent, or erratic movement.
### Interpretation
The image illustrates the difference between successful and unsuccessful action control in a simulated environment. The "Good Example" represents a desired outcome where the agent moves forward smoothly and predictably. The "Bad Examples" highlight potential problems with the action control system, such as jerky movements, disorientation, or failure to maintain a consistent trajectory. The image suggests that the action control system needs to be refined to avoid the issues demonstrated in the "Bad Examples". The consistent perspective and lighting in the "Good Example" are key indicators of successful action control, while the distortions and inconsistencies in the "Bad Examples" indicate failures in the system.
</details>
Figure 13: Examples of good and bad predictions. The action sequence contains only Forward actions. Models that violate this requirement yield observations that can mislead the planner.
<details>
<summary>x14.png Details</summary>
### Visual Description
## Image: Action Control Examples
### Overview
The image presents a visual comparison of "Good" and "Bad" examples of an "Action Control: Forward" task. It consists of a series of images arranged in rows, where each image represents a step in the forward action. The top row shows successful examples, while the bottom two rows display unsuccessful examples. The scene depicted is primarily a bathroom with a door.
### Components/Axes
* **Title:** "Action Control: Forward" (located at the top center of the image)
* **Row Labels:**
* "Good Example:" (top-left)
* "Bad Examples:" (middle-left)
* **Image Sequences:** Each row contains a sequence of images showing the progression of the action.
### Detailed Analysis
**Good Example Row:**
The top row shows a sequence of images where a door gradually opens, revealing a bathroom. The view progresses forward into the bathroom.
* Image 1: Door is mostly closed.
* Image 2: Door is slightly more open.
* Image 3: Door is approximately halfway open.
* Image 4: Door is significantly open, revealing more of the bathroom.
* Image 5: Door is almost fully open.
* Image 6: Door is fully open, showing the entire bathroom.
**Bad Examples Rows:**
The bottom two rows show sequences where the door either doesn't open correctly or the view doesn't progress forward into the bathroom as expected.
* Row 1: The door opens, but the view remains relatively static, not progressing forward.
* Row 2: The door opens, but the view changes drastically, showing a different part of the room or a different room altogether.
### Key Observations
* The "Good Example" demonstrates a smooth and consistent forward progression as the door opens.
* The "Bad Examples" show deviations from this expected progression, either through static views or abrupt changes in the scene.
### Interpretation
The image illustrates the concept of "Action Control" in the context of a "Forward" action. The "Good Example" serves as a benchmark for successful execution of this action, while the "Bad Examples" highlight potential failure modes. The comparison suggests that the system being evaluated should not only open the door but also maintain a consistent forward-moving perspective. The "Bad Examples" indicate scenarios where the system fails to maintain this forward progression, resulting in either a static view or a jump to a different scene.
</details>
Figure 14: Examples of good and bad predictions. The action sequence contains only Forward actions. Models that violate this requirement yield observations that can mislead the planner.
<details>
<summary>x15.png Details</summary>
### Visual Description
## Image Comparison: Good vs. Bad Examples of Simulated Actions
### Overview
The image presents a comparison of "Good Examples" and "Bad Examples" of a simulated agent performing actions in an indoor environment. Each example consists of a sequence of images showing the agent's current observation and the imagined outcomes of subsequent actions. The "Good Examples" demonstrate a coherent and realistic progression of actions, while the "Bad Examples" show unrealistic or flawed simulations.
### Components/Axes
The image is divided into two main sections: "Good Examples" at the top and "Bad Examples" at the bottom. Each section contains multiple rows of images. Each row represents a different scenario. Within each row, there are five images:
1. "It is the current observation before acting" - The initial view of the environment.
2. "Imagined action <1>:" - The result of the first imagined action.
3. "Imagined action <2>:" - The result of the second imagined action.
4. "Imagined action <3>:" - The result of the third imagined action.
5. "Imagined action <4>:" - The result of the fourth imagined action.
Each image contains a red rectangular bounding box highlighting a specific area of interest.
### Detailed Analysis or ### Content Details
**Good Examples:**
* **Row 1:**
* Image 1: "It is the current observation before acting." - Shows a living room with a sofa, dining table, and chairs. The red box is on the sofa.
* Image 2: "Imagined action <1>: turn right 22.5 degrees." - The view has shifted slightly to the right. The red box is on the sofa.
* Image 3: "Imagined action <2>: go straight for 0.20m." - The view has moved closer to the dining table. The red box is on the dining table.
* Image 4: "Imagined action <3>: go straight for 0.20m." - The view has moved closer to the dining table. The red box is on the dining table.
* Image 5: "Imagined action <4>: go straight for 0.20m." - The view has moved closer to the dining table. The red box is on the dining table.
* **Row 2:**
* Image 1: "It is the current observation before acting." - Shows a kitchen counter with objects on it. The red box is on the counter.
* Image 2: "Imagined action <1>: turn right 22.5 degrees." - The view has shifted slightly to the right. The red box is on the counter.
* Image 3: "Imagined action <2>: go straight for 0.20m." - The view has moved slightly forward. The red box is on the counter.
* Image 4: "Imagined action <3>: go straight for 0.20m." - The view has moved slightly forward. The red box is on the counter.
* Image 5: "Imagined action <4>: go straight for 0.20m." - The view has moved slightly forward. The red box is on the counter.
**Bad Examples:**
* **Row 1:**
* Image 1: "It is the current observation before acting." - Shows a living room similar to the first "Good Examples" row. The red box is on the sofa.
* Image 2: "Imagined action <1>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the sofa.
* Image 3: "Imagined action <2>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the sofa.
* Image 4: "Imagined action <3>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the sofa.
* Image 5: "Imagined action <4>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the sofa.
* **Row 2:**
* Image 1: "It is the current observation before acting." - Shows a kitchen counter similar to the second "Good Examples" row. The red box is on the counter.
* Image 2: "Imagined action <1>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the counter.
* Image 3: "Imagined action <2>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the counter.
* Image 4: "Imagined action <3>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the counter.
* Image 5: "Imagined action <4>: go straight for 0.20m." - The view is distorted and blurry. The red box is on the counter.
### Key Observations
* The "Good Examples" demonstrate a realistic simulation of actions, with smooth transitions and clear views.
* The "Bad Examples" show distorted and blurry views, indicating a failure in the simulation process.
* The red bounding boxes highlight areas of interest, but their specific purpose is not explicitly stated.
### Interpretation
The image illustrates the importance of accurate simulation in an agent's ability to understand and interact with its environment. The "Good Examples" suggest that the agent can successfully predict the outcome of its actions, while the "Bad Examples" indicate a breakdown in the simulation process, potentially leading to incorrect decisions. The distortion in the "Bad Examples" could be due to errors in the agent's perception, action execution, or environment modeling. The red bounding boxes likely represent the agent's focus of attention or the objects it is interacting with.
</details>
Figure 15: Additional examples of good and bad predictions.
<details>
<summary>x16.png Details</summary>
### Visual Description
## Image Comparison: Good vs. Bad Examples of Object Detection in Simulated Environments
### Overview
The image presents a comparison of "Good Examples" and "Bad Examples" of an object detection task within a simulated environment. Each example consists of a series of images depicting a virtual bathroom scene from a first-person perspective. The task involves identifying and localizing a specific object (likely a picture frame) as the viewpoint moves through the environment. Red bounding boxes indicate the detected object. The "Good Examples" show accurate and consistent detection, while the "Bad Examples" show inaccurate or inconsistent detection.
### Components/Axes
* **Title:** "Good Examples:" (top-left)
* **Title:** "Bad Examples:" (mid-left)
* **Image Rows:** Two rows, one for "Good Examples" and one for "Bad Examples."
* **Image Columns:** Five columns in each row, representing different "Imagined actions."
* **Image Labels (Good Examples):**
* Column 1: "It is the current observation before acting"
* Column 2: "Imagined action <1>: go straight for 0.20m"
* Column 3: "Imagined action <2>: go straight for 0.20m"
* Column 4: "Imagined action <3>: go straight for 0.20m"
* Column 5: "Imagined action <4>: go straight for 0.20m"
* **Image Labels (Bad Examples):**
* Column 1: "It is the current observation before acting"
* Column 2: "Imagined action <1>: go straight for 0.20m"
* Column 3: "Imagined action <2>: go straight for 0.20m"
* Column 4: "Imagined action <3>: go straight for 0.20m"
* Column 5: "Imagined action <4>: go straight for 0.20m"
* **Objects:** The primary object of interest is a picture frame on the wall.
* **Bounding Boxes:** Red rectangles indicating the detected object location.
### Detailed Analysis or ### Content Details
**Good Examples:**
* **Initial Observation:** A tall, narrow object (likely a door frame or narrow picture) is detected with a red bounding box.
* **Action 1-4:** The picture frame on the wall is consistently detected with a red bounding box. The bounding box appears to accurately enclose the picture frame in each subsequent image.
**Bad Examples:**
* **Initial Observation:** A tall, narrow object (likely a door frame or narrow picture) is detected with a red bounding box.
* **Action 1-4:** The red bounding box appears to detect the door frame or a similar vertical structure in the environment, rather than the picture frame on the wall. The bounding box is consistently misplaced and does not accurately enclose the intended object.
### Key Observations
* The "Good Examples" demonstrate successful object detection, with the bounding box accurately identifying the picture frame across different viewpoints.
* The "Bad Examples" demonstrate a failure in object detection, with the bounding box consistently misidentifying a different object (likely the door frame).
* The simulated environment includes a bathroom setting with a toilet, door, and wall-mounted picture frame.
* The "Imagined action" involves moving straight for 0.20 meters in each step.
### Interpretation
The image illustrates the performance of an object detection algorithm in a simulated environment. The "Good Examples" indicate that the algorithm can accurately identify and localize the target object (picture frame) under ideal conditions. The "Bad Examples" highlight potential challenges, such as misidentification of similar objects or sensitivity to changes in viewpoint. The comparison suggests that the algorithm's performance is not robust and can be easily misled by the environment's features. This could be due to factors like insufficient training data, poor feature extraction, or inadequate handling of occlusions or variations in lighting. The experiment demonstrates the importance of evaluating object detection algorithms under diverse and challenging conditions to ensure reliable performance in real-world applications.
</details>
Figure 16: Additional examples of good and bad predictions.
<details>
<summary>x17.png Details</summary>
### Visual Description
## Image Grid: Good vs. Bad Action Examples
### Overview
The image presents a grid of visual examples, categorized as "Good Examples" and "Bad Examples," demonstrating the impact of different imagined actions on a simulated environment. Each example consists of a series of images showing the current observation before acting and the subsequent imagined actions. The actions involve turning and moving straight, with the goal of navigating a hallway.
### Components/Axes
* **Title:** "Good Examples:" (top-left)
* **Title:** "Bad Examples:" (mid-left)
* **Image Sets:** Each set consists of a starting image labeled "It is the current observation before acting" followed by images representing imagined actions.
* **Action Labels:** Each imagined action image is labeled with "Imagined action <N>:" followed by a description of the action.
* Actions include: "turn right 22.5 degrees" and "go straight for 0.20m".
* **Bounding Boxes:** Red bounding boxes highlight specific regions of interest within each image.
### Detailed Analysis or ### Content Details
**Good Examples:**
* **Row 1:**
* Image 1: "It is the current observation before acting" - Shows a hallway with a red bounding box around a distant object.
* Image 2: "Imagined action <1>: turn right 22.5 degrees" - Shows the hallway after a slight right turn, with the red bounding box now focused on a different object.
* Image 3: "Imagined action <2>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
* Image 4: "Imagined action <3>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
* Image 5: "Imagined action <4>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
* **Row 2:**
* Image 1: "It is the current observation before acting" - Shows a hallway with a red bounding box around a table and chair.
* Image 2: "Imagined action <1>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
* Image 3: "Imagined action <2>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
* Image 4: "Imagined action <3>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
* Image 5: "Imagined action <4>: go straight for 0.20m" - Shows the hallway after moving straight, with the red bounding box around a table and chair.
**Bad Examples:**
* **Row 1:**
* Image 1: "It is the current observation before acting" - Shows a hallway with a red bounding box around a distant object.
* Image 2: "Imagined action <1>: go straight for 0.20m" - Shows a distorted hallway after moving straight, with the red bounding box around a distant object.
* Image 3: "Imagined action <2>: go straight for 0.20m" - Shows a distorted hallway after moving straight, with the red bounding box around a distant object.
* Image 4: "Imagined action <3>: go straight for 0.20m" - Shows a distorted hallway after moving straight, with the red bounding box around a distant object.
* Image 5: "Imagined action <4>: go straight for 0.20m" - Shows a distorted hallway after moving straight, with the red bounding box around a distant object.
* **Row 2:**
* Image 1: "It is the current observation before acting" - Shows a hallway with a red bounding box around a distant object.
* Image 2: "Imagined action <1>: turn right 22.5 degrees" - Shows a distorted hallway after a slight right turn, with the red bounding box now focused on a different object.
* Image 3: "Imagined action <2>: go straight for 0.20m" - Shows a distorted hallway after moving straight, with the red bounding box around a distant object.
* **Row 3:**
* Images 1-3: Blank images with red diagonal lines forming an "X" shape.
### Key Observations
* The "Good Examples" show a clear progression of actions and a relatively stable environment.
* The "Bad Examples" exhibit distortions and inconsistencies in the imagined actions, suggesting errors in the simulation or prediction.
* The red bounding boxes highlight the focus of the agent's attention in each image.
* The last row of "Bad Examples" is completely blank with red "X" marks, indicating a failure case.
### Interpretation
The image illustrates the difference between successful and unsuccessful imagined actions in a simulated environment. The "Good Examples" demonstrate a coherent and predictable outcome of the actions, while the "Bad Examples" highlight scenarios where the simulation fails to accurately predict the consequences of the actions. This comparison is crucial for training and evaluating AI agents that rely on imagined actions for decision-making. The distortions in the "Bad Examples" suggest potential issues with the agent's perception, planning, or the simulation environment itself. The blank images with red "X" marks indicate a complete failure of the simulation or prediction process.
</details>
Figure 17: Additional examples of good and bad predictions.
<details>
<summary>x18.png Details</summary>
### Visual Description
## Image Analysis: Robot Simulation Examples
### Overview
The image presents a series of simulated robotic arm actions, categorized as "Good Examples" and "Bad Examples." Each example shows the robotic arm at different stages of an action sequence, labeled as "Simulation after Action <N>," where N is a number indicating the sequence step. The simulations involve the robot interacting with colored squares and rectangular objects on a wooden surface.
### Components/Axes
* **Title:** "Good Examples:" and "Bad Examples:"
* **Subtitles:** "Simulation after Action <1>", "Simulation after Action <2>", "Simulation after Action <3>", "Simulation after Action <4>", "Simulation after Action <5>"
* **Objects:** Robotic arm (white), colored squares (red, green, yellow, pink), colored rectangular objects (blue, black, brown).
* **Environment:** Wooden surface, background with a gradient from light to dark.
### Detailed Analysis
**Good Examples:**
* **Row 1:**
* Simulation after Action <1>: The robotic arm is positioned above a red square.
* Simulation after Action <2>: The robotic arm is positioned above a red square.
* Simulation after Action <3>: The robotic arm is positioned above a yellow square.
* Simulation after Action <4>: The robotic arm is positioned above a yellow square.
* **Row 2:**
* Simulation after Action <1>: The robotic arm is positioned above a blue rectangular object.
* Simulation after Action <2>: The robotic arm is positioned above a blue rectangular object.
* Simulation after Action <4>: The robotic arm is positioned above a blue rectangular object.
* Simulation after Action <5>: The robotic arm is positioned above a blue rectangular object.
**Bad Examples:**
* **Row 1:**
* Simulation after Action <1>: The robotic arm is positioned above a red square.
* Simulation after Action <2>: The robotic arm is positioned above a pink square.
* Simulation after Action <3>: The robotic arm is positioned above a green square.
* Simulation after Action <4>: The robotic arm is positioned above a green square.
* **Row 2:**
* Simulation after Action <1>: The robotic arm is positioned above a brown rectangular object.
* Simulation after Action <2>: The robotic arm is positioned above a brown rectangular object.
* Simulation after Action <3>: The robotic arm is positioned above a brown rectangular object.
* Simulation after Action <4>: The robotic arm is positioned above a brown rectangular object.
### Key Observations
* The "Good Examples" appear to show the robot consistently interacting with the same colored object across multiple actions, or with a set of similar objects.
* The "Bad Examples" show the robot interacting with different colored objects in sequence.
* The number of actions varies between the examples. The first row of "Good Examples" has 4 actions, while the second row has 5. The "Bad Examples" both have 4 actions.
### Interpretation
The image likely illustrates a reinforcement learning or robotic control scenario where the robot is being trained to perform specific tasks. The "Good Examples" likely represent successful execution of a task, such as consistently picking up or interacting with a specific object. The "Bad Examples" likely represent failures, such as picking up the wrong object or deviating from the intended sequence of actions. The simulation actions are snapshots of the robot's state at different time steps during the task execution. The image suggests that the robot's performance is being evaluated based on its ability to consistently interact with the correct objects.
</details>
Figure 18: Additional examples of good and bad predictions.
Appendix F Prompt Templates used in World-in-World
In this section, we provide the exact prompt templates used in our experiments for four tasks in World-in-World: (i) Active Recognition (AR), (ii) Image-Goal Navigation (ImageNav), (iii) Active Embedded Question Answering (A-EQA), and (iv) Robotic Manipulation.
F.1 Active Recognition (AR) Prompt
AR Answerer Prompt
Please recognize the object in the image bounded by the red box.
AR Planner Prompt
You are an AI agent tasked with identifying a target object within an image—specifically, the object enclosed by a red bounding box. Your objective is to navigate toward a viewpoint that maximizes the target’s visibility and recognition accuracy. Instructions: 1.
Based on the current {obs_key} observation, plan the next <look_ahead_action_num> action(s) to take in sequence. 2.
Use the following heuristics to guide planning: •
If the red-boxed object appears on the left side of the image, turning left often improves visibility. •
If it appears on the right side, turning right is usually beneficial. •
If the object is partially occluded or obstructed, consider repositioning to bypass the obstacle and refine your viewpoint. 3.
Choose a sequence of actions that leads to a clear, centered, and unobstructed view of the red-boxed object.
AR Answerer Additional Prompt (with WM Rollouts)
You now have a composite visualization formed by stitching imagined views from multiple perspectives around your current position. These perspectives are centered on the target object (enclosed within the red bounding box). Use these synthesized views to: •
Improve object identification accuracy. •
Make more informed recognition decisions.
AR Planner Additional Prompt (with WM Rollouts)
You are now simulating imagined future trajectories by generating hypothetical actions and their corresponding observations. Use these imagined observations to: •
Evaluate the potential outcomes of different action sequences. •
Make informed navigation decisions by selecting the next best action based on predicted future states and your current state. Note: •
Each imagined frame is annotated with the specific action taken and its index at the top of the image. •
Pay attention to the presence of red bounding boxes indicating the target object. If the target is not visible in a frame, this indicates a poor action. •
You should adjust your action selection strategy to avoid such failure states.
F.2 Image-Goal Navigation (ImageNav) Prompt
ImageNav Planner Prompt
You are an AI navigation agent tasked with locating the position from which the goal image was captured. Your objective is to plan a sequence of actions that leads to a position where the goal image is clearly visible, centered in the front view, and appears to have been taken within at your current position. Inputs: You are provided with a sequence of images: 1.
First, the current egocentric observation: {obs_key}. 2.
Last, the goal image: a reference image that represents the target viewpoint you are trying to reach. Task: 1.
Based on the input images, plan the next <{look_ahead_action_num}> action(s) in order. 2.
Optimize for: •
Alignment: The goal image should be centered in the front view. •
Proximity: Your position should match the goal image’s capture point. •
Visibility: The goal image should appear clear and unobstructed in your current front view.
ImageNav Planner Prompt (with WM Rollouts)
You are an AI navigation agent tasked with locating the position from which the goal image was captured. Your objective is to plan a sequence of actions that leads to a position where the goal image is clearly visible, centered in the front view, and appears to have been taken within at your current position. Inputs: You are provided with: 1.
The goal image: a reference image that represents the target viewpoint you are trying to reach. Task: 1.
Based on the input images, plan the next <{look_ahead_action_num}> action(s) in order. 2.
Optimize for: •
Alignment: The goal image should be centered in the front view. •
Proximity: Your position should match the goal image’s capture point. •
Visibility: The goal image should appear clear and unobstructed in your current front view.
F.3 Active Embedded Question Answering (A-EQA) Prompt
A-EQA High-Level Planner Prompt
You are an embodied navigation and question-answering agent specialized in indoor scene understanding. Your goal is to either answer the user’s question directly from the current observation or propose a high-level navigation planning to gather more information. User Query: {question} Inputs: You are provided with the following: 1.
A stitched panoramic image with annotations — composed of multiple directional images captured from your current position (the name of each view is labeled on the top of the image). Each detected object is annotated with its contour and a unique object index. 2.
A stitched panoramic image without annotations — visually identical but without overlays, serving as a clean reference. 3.
A dictionary mapping detected objects to their corresponding perspective views and object indices in the annotated image: Format: {{view_id: {{object_index: object_name}}}} Current mapping: {detected_objs} Note all the provided images are in the formt of {obs_key}. Task Description: Your task is to: 1.
Analyze the visual information from each perspective direction. 2.
Identify all possible exits and doorways in the environment. 3.
Give one high-level navigation plan to further explore the scene in order to answer the User Query. 4.
If the answer to the question is fully evident from the current observation, provide it directly. Otherwise, set your current answer to “None”. Output Format: Return your response as a dictionary with the following structure: { ’Reason’: <Your visual reasoning and analysis>, ’Action Plan’: <Description of your next high-level navigation plan>, ’Chosen View’: <One of: ’front’, ’left’, ’right’, or ’back’, indicating the view you are going to further explore in your Action Plan>, ’Chosen Landmark’: <Index of the selected object landmark from the annotated stitched image, or ’None’> ’Answer’: <Your answer to the User Query, or ’None’> } Constraints: •
Provide exactly one high-level action, including one ’Chosen View’ and one ’Chosen Landmark’. •
If no suitable annotated object is available in your desired direction, set ’Chosen Landmark’ to ’None’ and describe your intended action in the ’Action Plan’ field. •
Each ’Action Plan’ should include a clear and executable instruction and stop condition. – Good Example: ’Action Plan’: "Pass through the doorway (object index "3") in the front view, and stop once inside the next room." – Good Example: ’Action Plan’: "Approach the sofa (object idx "10") in the left view, and stop once we can see the objects on it." – Bad Example: ’Action Plan’: "Move into the kitchen area visible in the view and stop once inside the kitchen." – kitchen area is not a specific object and not clear how to get there. •
If a landmark is selected, it must correspond to a visible, annotated object in the stitched image. •
Do not select unlabeled objects — they typically indicate previously visited or non-informative regions. •
Populate ’Answer’ only when you are confident the question can be answered from the current observation. Otherwise, set ’Answer’: ’None’ in the dictionary. Tips: •
If you observe a door in a closed state, it means you cannot pass through it. •
If the current observation shows that your previous plan has not yet been completed, it is acceptable to propose a similar plan again to continue pursuing the same goal. •
Leverage human spatial habits to guide your planning. For instance, if the goal involves finding a television, selecting a nearby sofa may be effective, as these often appear together in living spaces.
A-EQA Low-Level Planner Prompt
You are now performing low-level navigation action planning for an indoor scene exploration task. Inputs: You are provided with: 1.
An updated RGB image with annotations, representing the egocentric view of your current environment: •
Detected objects are annotated with contours and unique object indices with square text boxes. 2.
A high-level navigation plan represented as a dictionary with two fields: •
’Action Plan’: A description of the intended navigation strategy. •
’Chosen Landmark’: The object index of the selected landmark from the annotated image to approach, or ’None’ if no landmark is selected (in which case follow the ’Action Plan’ description). The current high-level plan is: {high_level_plan} Note all the provided images are in the format of {obs_key}. Task: Your task is to: 1.
Analyze the visual scene and identify your position relative to the goal. 2.
Determine the next low-level action(s) to take in sequence, up to a maximum of <{look_ahead_action_num}> steps. Constraints: •
You must generate less than {look_ahead_action_num} low-level actions. •
The actions sequence should align with the goal descripted in high-level ’Action Plan’ and ’Chosen Landmark’. •
If the navigation goal or selected landmark in the high-level plan is either: –
not visible in the current observation, or –
already reached (i.e., centered, unobstructed, and close), then your only action should be “stop”. Tips: •
If the landmark object is partially occluded or obstructed, consider repositioning to bypass the obstacle before approaching it directly. •
Choose actions that meaningfully move the agent toward the selected landmark or fulfill the intent of the high-level plan. •
Maintain spatial awareness: understand the relationship between your egocentric view and the direction of the target.
A-EQA High-Level Planner Additional Prompt (with WM Rollouts)
In addition to your current (real) observations, you are now provided with simulated outcomes —low-resolution reconstructions that represent the potential result of executing future navigation plans. These simulated outcomes are designed to help you better understand your surroundings and support more informed navigation planning. Each simulated outcome includes: •
Proposed High-Level Plan: A hypothetical navigation strategy used to generate the simulated result. •
Simulated Observation: A stitched panoramic image showing what the environment might look like after following the proposed plan. You should use this information to: •
Evaluate the potential effectiveness and correctness of the proposed high-level strategies. •
Make informed decisions by selecting your next high-level plan based on both the simulated information and your current real observation. Notes: •
Object indices remain consistent across simulated and real observations. •
Simulated outcomes are NOT fully accurate. If you believe you can answer the user query based on simulation alone, you should NOT provide a final answer yet. Instead, select a high-level plan that will lead to a real observation and validate your answer afterward. Your current simulated outcomes are:
F.4 Robotic Manipulation Prompt
Manipulation Planner Prompt
You are a Franka Panda robot with a parallel gripper. You can perform various tasks and output a sequence of gripper actions to accomplish a given task with images of your status. The input space, output action space and color space are defined as follows: Input Space You are given the following inputs: 1.
Human Instruction: A natural language command specifying the manipulation task goal. 2.
Object Dictionary: •
Each object is represented by a unique index (e.g., object 1) and mapped to a 3D discrete coordinate [X, Y, Z]. 3.
Annotated Scene Image: •
Each object in the image is annotated with: –
A circle point marker with –
A unique object index, which corresponds to the object dictionary. •
There is a red XYZ coordinate frame located in the top-left corner of the table. –
The XY plane represents the surface plane of the table (Z = 0). –
The valid coordinate range for X, Y, Z is: [0, {}]. Output Action Space •
Each output action is represented as a 7D discrete gripper action in the following format: [X, Y, Z, Roll, Pitch, Yaw, Gripper state]. •
X, Y, Z are the 3D discrete position of the gripper in the environment. It follows the same coordinate system as the input object coordinates. •
The allowed range of X, Y, Z is [0, {}]. •
Roll, Pitch, Yaw are the 3D discrete orientation of the gripper in the environment, represented as discrete Euler Angles. •
The allowed range of Roll, Pitch, Yaw is [0, {}] and each unit represents {} degrees. •
Gripper state is 0 for close and 1 for open. Color space •
Each object can only be described using one of the colors below: ["red", "maroon", "lime", "green", "blue", "navy", "yellow", "cyan", "magenta", "silver", "gray", "olive", "purple", "teal", "azure", "violet", "rose", "black", "white"], {}
Manipulation Planner Additional Prompt (with WM Rollouts)
You are now provided with simulated outcomes in addition to your real-time observations. These outcomes are low-resolution predictions of what the scene may look like after executing hypothetical action plans. They are intended to help you reason about the environment and make more informed decisions. Simulated Outcome Structure Each simulated-outcome item includes: •
Proposed Action Plan: The sequence of gripper actions that led to the simulated result. •
Simulated Observation: The simulated result after following the proposed plan. How to Use This Information You must consider both: 1.
Your current real observation of the environment, and 2.
The provided simulated outcomes. Use these to: •
Evaluate how well each proposed plan satisfies the task objective. •
Identify if any proposed plan fully achieves the instruction goal. •
If a proposed plan appears valid and effective, you may adopt it directly as your final response. •
If no plan fully meets the goal, generate a revised or entirely new action plan, guided by insights from the simulations and the real-world scene. Additional Notes •
Simulated outcomes are approximate. Treat them as helpful forecasts, not absolute truth. •
You must analyze these hypothetical action plans and their simulated outcomes in the reasoning_and_reflection field of the returned JSON (e.g., their differences and why you choose one over another). •
Always prioritize correctness and robustness in the final executable plan. You are now given the following simulated outcomes: