\n ## Diagram: World Models in Human Minds and Multimodal AI Reasoning ### Overview This image is a technical diagram or infographic illustrating the concept of "World Models" in human cognition and how similar principles are applied to reasoning in Multimodal AI. It is divided into three primary horizontal sections, each with a blue-bordered header. The diagram uses a combination of illustrations, text, equations, and example outputs to draw parallels between human mental processes and AI reasoning capabilities. ### Components/Axes The diagram is structured into three main sections: 1. **Top Section: "World Models in Human Minds"** * **Left Sub-section: "World Model: Mental Model of the World"** * Illustration: A person thinking, with a thought bubble containing a flowchart and a globe. * Text: "Approximate" and "Feedback" with arrows pointing to a realistic image of Earth. * **Right Sub-section: "Dual Representations of World Knowledge"** * Two columns: * **Left Column: "Verbal/Symbolic Knowledge"** * Illustration: A person shooting a basketball. * Mathematical equations: `y = ax² + bx + c` and `F = ma`. * Text: "Dislike in Daily Life" (in red). * **Right Column: "Visual/Imagery Knowledge"** * Illustration: A basketball trajectory diagram over a court. * A red YouTube play button icon. * Text: "Prefer in Daily Life" (in green). 2. **Middle Section: "Reasoning with Verbal World Modeling in Multimodal AI"** * This section is divided into three vertical columns, each with a title and an example. * **Column 1: "Mathematical Reasoning" & "Puzzle Solving"** * Contains a math problem and its step-by-step solution. * Contains a puzzle state description. * **Column 2: "Travel Planning"** * Contains a task description and a structured plan with `<think>`, `STATE`, and `ACTION` tags. * **Column 3: "Everyday Activity Planning"** * Contains a goal and a sequence of images showing cooking steps. * Contains a `<think>` block describing the cooking process. 3. **Bottom Section: "Reasoning with Visual World Modeling in Multimodal AI"** * **Title: "Real-World Spatial Reasoning"** * **Left:** A question box: "When you took the photo in Figure 1, where was the iron refrigerator located relative to you?" with two small interior photos below. * **Center & Right:** A sequence of three larger photos showing different views of a room (a doorway, a fireplace, a window). Each photo is accompanied by a `<think>` block of text describing the AI's reasoning process to locate the refrigerator. ### Detailed Analysis / Content Details **Top Section Content:** * **Text Transcription:** * Main Header: "World Models in Human Minds" * Left Sub-header: "World Model: Mental Model of the World" * Right Sub-header: "Dual Representations of World Knowledge" * Labels: "Approximate", "Feedback", "Verbal/Symbolic Knowledge", "Visual/Imagery Knowledge", "Dislike in Daily Life", "Prefer in Daily Life". * Equations: `y = ax² + bx + c`, `F = ma`. **Middle Section Content (Verbal World Modeling):** * **Mathematical Reasoning:** * **Question:** "If a > 1, then the sum of the real solutions of √(a - √(a + x)) = x is equal to" * **Response:** "<think> To solve the equation √(a - √(a + x)) = x, let's start by squaring both... (a - √(a + x)) = x² => √(a + x) = a - x². Rearrange to isolate the inner square root term: (a - x²)² = a + x => a² - 2ax² + (x²)² = a + x => x⁴ - 2ax² - x + a² - a = 0 ..." * **Puzzle Solving:** * **Text:** "<think>From S at [452, 59], the only available move is West to [359, 59] ......." * **Travel Planning:** * **Task:** "Plan a trip with a budget of $1,700." * **Plan:** "<think>STATE: Initial Budget: $1700, Spent: $0. Day 1: transportation: PENDING. Available: Flight F3573659: $474. ACTION: Plan day 1 transportation. Select Flight F3573659. NEXT STATE: After this action, you've spent $474, leaving you $1226 from your $1700 budget..." * **Legend:** Green box labeled "State/Observation", Pink box labeled "Action". * **Everyday Activity Planning:** * **Goal:** "Cooking tomato and eggs" * **Text:** "<think>... ACTION: Cook the eggs in the pan. STATE: This action changes the state of the eggs from liquid to a partially cooked state. The eggs are now partially cooked and are being transformed into curds. The heat from the pan cooks the eggs, and the stirring action ensures even cooking..." **Bottom Section Content (Visual World Modeling):** * **Question:** "When you took the photo in Figure 1, where was the iron refrigerator located relative to you?" * **Reasoning Sequence (Transcribed from `<think>` blocks):** 1. "<think>First, let's analyze the images. [...] It's not visible in that initial view, so I need to change my perspective." 2. "The refrigerator is not visible in this 90-degree left turn view, [...] To be thorough, I will also check the view to the right." 3. "[...] My initial turn was 90 degrees left, but the refrigerator isn't at that exact angle. Let's try a smaller turn to the left. [...]" ### Key Observations 1. **Structural Parallelism:** The diagram explicitly parallels human cognitive processes (top section) with AI reasoning frameworks (middle and bottom sections). The "Dual Representations" in humans map to "Verbal World Modeling" and "Visual World Modeling" in AI. 2. **Use of `<think>` Tags:** The AI reasoning examples in both verbal and visual sections are annotated with `<think>` blocks, simulating an internal monologue or chain-of-thought process, mirroring the human "thought bubble" in the top section. 3. **Preference Dichotomy:** The human section highlights a preference for visual/imagery knowledge in daily life ("Prefer in Daily Life") over verbal/symbolic knowledge ("Dislike in Daily Life"). 4. **Task Diversity:** The verbal modeling section demonstrates application across diverse domains: abstract math, logistics (travel), and procedural tasks (cooking). The visual section focuses on spatial reasoning. 5. **Iterative Process:** The visual reasoning example shows an iterative, hypothesis-testing approach ("not visible... need to change perspective... try a smaller turn"). ### Interpretation This diagram serves as a conceptual framework arguing that advanced AI, particularly multimodal AI, should emulate the structure of human world models to achieve robust reasoning. * **Core Thesis:** Human intelligence relies on an internal, approximate world model that integrates both verbal/symbolic and visual/imagery knowledge. The diagram posits that for AI to reason effectively about the real world, it must develop analogous internal models. * **Relationship Between Elements:** The top section establishes the human cognitive foundation. The middle section shows how AI can use a *verbal* world model (structured states, actions, and symbolic reasoning) to plan and solve problems. The bottom section shows how AI can use a *visual* world model (understanding spatial relationships and perspective from images) to answer questions about the physical world. * **Underlying Message:** The inclusion of the "Dislike/Prefer" labels suggests that while symbolic reasoning is powerful, grounding AI in visual and experiential knowledge (like human preference) may lead to more intuitive and applicable intelligence. The `<think>` tags are crucial—they indicate that the reasoning process itself, not just the final output, is a key component of the model. * **Investigative Reading:** The diagram implies that current AI often excels at one type of reasoning (e.g., verbal/symbolic for LLMs) but true "world modeling" requires the integration of both streams, much like the human mind. The travel planning example shows a stateful, grounded process, while the spatial reasoning example shows an embodied, perspective-taking process. Together, they illustrate a move from static pattern recognition towards dynamic, model-based reasoning.