2508.14160
Model: gemini-2.0-flash
# RynnEC: Bringing MLLMs into Embodied World
**Authors**: Ronghao Dang, Yuqian Yuan, Yunxuan Mao, Kehan Li, Jiangpin Liu, Zhikai Wang, Fan Wang, Deli Zhao, Xin Li
1]DAMO Academy, Alibaba Group 2]Hupan Lab 3]Zhejiang University \contribution [*]Equal contribution
(November 18, 2025)
Abstract
We introduce RynnEC, a video multimodal large language model designed for embodied cognition. Built upon a general-purpose vision-language foundation model, RynnEC incorporates a region encoder and a mask decoder, enabling flexible region-level video interaction. Despite its compact architecture, RynnEC achieves state-of-the-art performance in object property understanding, object segmentation, and spatial reasoning. Conceptually, it offers a region-centric video paradigm for the brain of embodied agents, providing fine-grained perception of the physical world and enabling more precise interactions. To mitigate the scarcity of annotated 3D datasets, we propose an egocentric video based pipeline for generating embodied cognition data. Furthermore, we introduce RynnEC-Bench, a region-centered benchmark for evaluating embodied cognitive capabilities. We anticipate that RynnEC will advance the development of general-purpose cognitive cores for embodied agents and facilitate generalization across diverse embodied tasks. The code, model checkpoints, and benchmark are available at: https://github.com/alibaba-damo-academy/RynnEC
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Visual Question Answering System
### Overview
The image presents a diagram illustrating a visual question answering (VQA) system. It outlines two tasks, each involving a series of steps and questions, processed by a large language model (LLM) with visual and mask encoders/decoders. The diagram shows the flow of information and the interaction between the model and visual inputs.
### Components/Axes
* **Header:** Contains the title "Q: Which is the nearest plant to <object mask>?" and the label "<video>".
* **Visual Encoder:** Processes video input.
* **Mask Encoder:** Processes object masks.
* **Large Language Model:** The central processing unit.
* **Mask Decoder:** Generates mask tokens.
* **Task 1:** Involves sticking window stickers, tidying the table, and organizing items on a bookshelf.
* **Task 2:** Involves stir-frying and cleaning the kitchen windowsill.
* **Steps:** Each task is broken down into numbered steps (Step 1, Step 2, etc.).
* **Questions (Q):** Each step poses a question.
* **Answers (A):** Each question has a corresponding answer.
* **Visual Input:** Each step is accompanied by a series of images representing the scene.
* **Arrows:** Indicate the flow of information between components.
### Detailed Analysis or Content Details
**Task 1:**
* **Step 1:**
* Q: How many panes of glass are there on the window? How many decorative window stickers are needed?
* A: Three
* Visual Input: Images showing a window with panes.
* **Step 2:**
* Q: How wide is the pad on the table? Can it be placed on the bookshelf without falling?
* A: The width of 6 is 20 cm, and it can be placed on the bookshelf.
* Visual Input: Images showing a pad on a table.
* **Step 3:**
* Q: Where are the teddy bear and the pillow located?
* A: The teddy bear is 7 and the pillow is 1.
* Visual Input: Images showing a teddy bear and a pillow.
* **Step 4:**
* Q: Do my feet need to move in the process of moving the teddy bear 7 onto the pillow 1?
* A: 7 needs to be moved 2.1 meters onto 1, so you will need to move your feet.
* Visual Input: Images showing feet and the teddy bear/pillow.
* **Step 5:**
* Q: Storage box on the bottom bookshelf
* A: 5
* Visual Input: Images showing a storage box on a bookshelf.
* **Step 4 (repeated):**
* Q: Will directly taking out the storage box 5 bump into other items?
* A: It will bump into a small doll, so you need to move it aside first.
* Visual Input: Images showing the storage box and other items.
**Task 2:**
* **Step 1:**
* Q: Where is the soy sauce?
* A: 3 is the soy sauce
* Visual Input: Images showing soy sauce.
* **Step 2:**
* Q: What is the function of 6?
* A: This object is a wok, which is used for stir-frying.
* Visual Input: Images showing a wok.
* **Step 3:**
* Q: How far is 3 from me? Can I reach it without moving?
* A: 3 is 1.3 meters away from me, and my arm is only 0.8 meters long, so I need to walk up to it to pick it up.
* Visual Input: Images showing the distance to an object.
* **Step 4:**
* Q: It is time to turn on the stove switch. Which stove is the frying pan 5 located on?
* A: 5 is on the stove on the left side.
* Visual Input: Images showing a stove and frying pan.
* **Step 5:**
* Q: I need to take the pot lid; in which direction relative to me is the lid 4 located?
* A: It is at my two o'clock position.
* Visual Input: Images showing a pot lid.
* **Step 6:**
* Q: Where is the degreasing spray bottle?
* A: 6 is the degreasing spray bottle
* Visual Input: Images showing a degreasing spray bottle.
* **Step 7:**
* Q: I am going to clean the kitchen windowsill. Which of the two objects, 1 or 2 needs to be removed?
* A: 2 needs to be removed
* Visual Input: Images showing the kitchen windowsill and objects.
**Flow of Information:**
1. The system receives a video input and a question related to an object mask.
2. The Visual Encoder processes the video.
3. The Mask Encoder processes the object mask.
4. The Large Language Model integrates the encoded information.
5. The Mask Decoder generates a mask token.
6. The system provides an answer based on the mask token.
### Key Observations
* The system uses visual and mask encoders to process visual information.
* The Large Language Model is central to the question answering process.
* The tasks involve understanding spatial relationships, object recognition, and following instructions.
* The system provides detailed answers based on the visual input and the questions asked.
### Interpretation
The diagram illustrates a VQA system designed to understand and respond to questions about visual scenes. The system combines visual processing with natural language understanding to perform tasks that require both perception and reasoning. The breakdown of tasks into steps and questions demonstrates a structured approach to problem-solving. The system's ability to provide detailed answers suggests a sophisticated understanding of the visual environment and the relationships between objects within it. The use of object masks allows the system to focus on specific regions of interest, improving accuracy and efficiency.
</details>
Figure 1: RynnEC is a video multi-modal large language model (MLLM) specifically designed for embodied cognition tasks. It can accept inputs interwoven from video, region masks, and text, and produce output in the form of text or masks based on the question. RynnEC is capable of addressing a diverse range of object and spatial questions within embodied contexts and plays a significant role in indoor embodied tasks.
1 Introduction
In recent years, Multi-modal Large Language Models (MLLMs) Wu et al. [2023], Zhang et al. [2024a] have experienced rapid development, leading to the emergence of models such as Gemini Team et al. [2024] and GPT-4o OpenAI et al. [2024] that can handle image and even video inputs. These MLLMs are attracting increasing attention from researchers due to their powerful contextual understanding Doveh et al. [2025] and generalization Zhang et al. [2024c] abilities. Researchers in embodied intelligence are also beginning to explore the use of MLLMs as the brains of robots Han et al. [2025b], Jin et al. [2024], enabling them to perceive the real world through visual inputs like humans do. However, the current mainstream MLLMs are trained on extensive internet images and lack the foundational visual cognition to match the physical world Dang et al. [2025], Yuan et al. [2025b].
Some works have begun exploring how MLLMs can be applied to ego-centric embodied scenarios. Models like Exo2Ego Zhang et al. [2025b] and EgoLM Hong et al. [2025] enhance the understanding of ego-centric dynamic environment interactions. SpatialVLM Chen et al. [2024a] and SpatialRGPT Cheng et al. [2024a] focus on addressing spatial understanding challenges within embodied contexts. However, these approaches are challenging to directly implement in physical robots to perform complex tasks. The main limitations are as follows:
1. Lack of flexible visual interaction: In complex embodied scenarios, relying solely on textual communication is prone to ambiguity or vagueness. Direct visual interaction references, such as masks or points, can more accurately and flexibly index entities within a scene, facilitating precise task execution.
1. Insufficient detailed understanding of objects: During task execution, objects typically serve as the smallest operational units, making comprehensive and detailed understanding of objects crucial. As illustrated in Task 1 Step 1 in Fig. 1, recognizing the number of panes in a window is essential to determine the quantity of window decals needed.
1. Absence of video-based coherent spatial awareness: For humans, spatial cognition arises from continuous visual perception Pasqualotto and Proulx [2012]. Current methods in spatial intelligence Zhang et al. [2025c], Xu et al. [2025] primarily focus on single or discrete images, lacking the capacity for spatial understanding in high-continuity videos. For example, in Task 1 Step 4 in Fig. 1, the absolute distance between the teddy bear and the pillow requires a spatial scale concept derived from the entire video to be properly inferred.
Thus, we propose RynnEC, an embodied cognitive MLLM designed to enhance robotic understanding of the physical world. As illustrated in Fig. 1, RynnEC is a large video understanding model whose visual encoder and foundational parameters are derived from VideoLLaMA3 Zhang et al. [2025a]. To enable flexible visual interaction, we incorporate an encoder and decoder specifically for region masks in videos, allowing RynnEC to achieve precise instance-level comprehension and grounding.
Within this framework, RynnEC is designed to perform diverse cognitive tasks in embodied scenarios. We categorize embodied cognitive abilities into two essential components: object cognition and spatial cognition. Object cognition necessitates MLLMs’ understanding of object attributes, quantities, and their relationships with the environment, alongside accurate object grounding. Spatial cognition is further divided into world-centric and ego-centric perspectives. World-centric spatial cognition requires the model to grasp absolute scales and relative positions within scenes, as exemplified by object size estimations in Task 1 Step 2 (Fig. 1). Ego-centric spatial cognition connects the robot’s physical embodiment with the world, thereby assisting in behavioral decisions. For example, as depicted in Fig. 1, the reachability estimation in Task 2 Step 3 and the orientation estimation in Task 2 Step 5 assist the robot in clearly defining its relationship with interactive objects. Equipped with enhanced object and spatial reasoning, RynnEC supports more efficient execution of complex, real-world robotic tasks.
Regrettably, the development of embodied cognition models has been slow due to a lack of ego-centric videos and high-quality annotations. Efforts such as Multi-SpatialMLLM Xu et al. [2025], Spatial-MLLM Wu et al. [2025a], and SpaceR Ouyang et al. [2025] leverage open-source datasets with comprehensive 3D point cloud and annotations to generate training data. However, in an era of scarce 3D annotations Hou et al. [2025], Lyu et al. [2024], this approach cannot achieve rapid and cost-effective expansion of data scale. Hence, we propose a data generation pipeline that transforms ego-centric RGB videos into embodied cognition question-answering datasets. This pipeline begins with instance segmentation from videos and diverges into two branches: one generating object cognition data and the other producing spatial cognition data. Ultimately, data from both branches are integrated into a comprehensive embodied cognition dataset. From over 200 households, we collect more than 20,000 egocentric videos. A subset from ten households is manually verified and balanced to create RynnEC-Bench, a fine-grained embodied cognition benchmark encompassing 22 tasks in object and spatial cognition.
Extensive experiments demonstrate that RynnEC significantly outperforms both general OpenAI et al. [2024], Bai et al. [2025], Zhu et al. [2025] and task-specific Yuan et al. [2025a, c], Team et al. [2025] MLLMs in cognitive abilities within embodied scenarios, showcasing scalable application potential. Additionally, we observe notable advantages in multi-task training with RynnEC and identify preliminary signs of emergence in more challenging embodied cognition tasks. Finally, we highlight the potential of RynnEC in facilitating robots to undertake large-scale, long-range tasks.
2 Related Work
2.1 MLLMs for Video Understanding
Early MLLMs primarily relied on sparse sampling and simple connectors, such as MLPs Lin et al. [2023], Ataallah et al. [2024], Maaz et al. [2023] and Q-Formers Zhang et al. [2023], Li et al. [2024b], to integrate visual representation with large language models. Subsequently, to tackle the problem of long video understanding, Zhang et al. [2024b] directly expanded the context window of language models, while Zhang et al. [2024d] introduced pooling in the spatial and temporal dimensions to compress the number of video tokens. As the need for more fine-grained understanding emerged, some studies (VideoRefer Yuan et al. [2025c], DAM Lian et al. [2025] and PAM Lin et al. [2025]) employed region-level feature encoders enabling video MLLMs to accept masked inputs and comprehend the semantic features of objects within the masks. Although these video MLLMs have demonstrated superior capabilities in high-level semantic capture and temporal modeling, they lack robust physical-world comprehension in egocentric embodied scenarios.
2.2 Embodied Scene Understanding Benchmarks
Some studies Ren et al. [2024a], Li et al. [2024c], Han et al. [2025a] have begun to explore leveraging MLLMs to assist robots in solving embodied tasks. However, determining whether these MLLMs possess the ability to understand and interact with the physical world is challenging. Consequently, several benchmarks have emerged to evaluate the capability of MLLMs to perceive the physical world. OpenEQA Majumdar et al. [2024] and IndustryEQA Li et al. [2025a] focus on several key competencies in home and industrial settings, respectively, and manually designed open-vocabulary questions. VSI-Bench Yang et al. [2025c] centers on assessing the spatial cognitive abilities of MLLMs. STI-Bench Li et al. [2025b] introduces more complex kinematic (e.g. velocity) problems. ECBench Dang et al. [2025] systematically categorizes embodied cognitive abilities into static environments, dynamic environments, and overcoming hallucinations, offering a comprehensive evaluation across 30 sub-competencies. While these benchmarks encompass a wide range of abilities, they are unable to assess more fine-grained, region-level understanding capabilities in embodied scenarios. Compared to purely textual question-answering, region-level visual interaction can more accurately refer to targets in the complex real world.
2.3 Improving MLLMs for Embodied Cognition
The aforementioned embodied benchmarks have highlighted the cognitive limitations of current MLLMs in embodied scenarios. Consequently, some studies have started to investigate diverse strategies for enhancing MLLMs’ understanding of the physical world. GPT4Scene Qi et al. [2025] improves MLLMs’ consistent global scene understanding by explicitly adding instance marks between video frames. SAT Ray et al. [2024] explores multi-frame dynamic spatial reasoning in simulated environments. Spatial-MLLM Wu et al. [2025a], Multi-SpatialMLLM Xu et al. [2025], and SpaceR Ouyang et al. [2025] leverage 3D datasets with detailed annotations (e.g., ScanNet Yeshwanth et al. [2023]) to construct the suite of spatial-intelligence tasks introduced in VSI-Bench. In contrast, our data generation pipeline based on RGB videos yields more realistic and scalable training data. More importantly, RynnEC is designed not just to handle selected capabilities in embodied scenarios, but to cover a broad swath of the world cognition required for embodied task execution under a single paradigm.
3 Methodology
RynnEC is a robust video embodied cognition model capable of processing and outputting various video object proposals. This enables it to flexibly address embodied questions about objects and space. Due to a paucity of research in this domain, we comprehensively present the construction process of RynnEC from four perspectives: data generation (Sec. 3.1), evaluation framework establishment (Sec. 3.2), model architecture (Sec. 3.3), and training (Sec. 3.4).
<details>
<summary>x2.png Details</summary>
### Visual Description
## Diagram: Video Instance Segmentation and Question Answering
### Overview
The image is a diagram illustrating a process for video instance segmentation and the generation of object and spatial question answering (QA). It outlines the steps from video input to generating questions about objects and their spatial relationships within the video.
### Components/Axes
* **Overall Structure:** The diagram is divided into three main sections, each enclosed in a dashed box: "Video Instance Segmentation" (left), "Generate Object QA" (top-right), and "Generate Spatial QA" (bottom-right).
* **Video Instance Segmentation:**
* Input: A series of video frames, indicated by a stack of images with a "40s" label suggesting a time interval.
* Process:
* "Extract Object Name" - A process to identify and name objects within the video frames.
* "Grounding DINO" and "Segment Anything 2" - These are looped processes, with "one second interval" indicating the frequency of iteration.
* Output: Segmented video frames.
* **Generate Object QA:**
* Input: "Keyframe of Objects" - Images of identified objects.
* Process:
* "Qwen 2.5-VL" - A prompt is given to a model.
* The model generates a "Caption" describing the object (e.g., "The object is a small footrest...") and poses questions (Q) about the object (e.g., "Is the <object> currently being exposed to sunlight?"). Answers (A) are provided.
* "Qwen 3" - Another model is used to generate object referring expressions.
* Output: "Object Referring Expression" - Examples include "[Simple Referring]: The green leather footstool beside the sofa" and "[Situational Referring]: A two-year old child is unable to climb onto the sofa. What can he use to get up there?".
* **Generate Spatial QA:**
* Input: 3D reconstruction of the scene, labeled "Mast3r-SLAM + Mask 2D to 3D". "Start Pos" and "End Pos" are marked with star icons.
* Process:
* "Ground Level Calibration" - The 3D scene is calibrated with X, Y, and Z axes indicated.
* "Template" - A template is used to generate spatial questions.
* Output: "Spatial Cognition Question" - Examples include "[Ego-Centric]: Q: Which of the two objects, <object 1> or <object 2>, is closer to me? A: <object1>" and "[World-Centric]: Q: What is the difference in height above the ground between <object1> and <object2>? A: 1.2 meters".
### Detailed Analysis or ### Content Details
* **Video Instance Segmentation:** The process starts with a video input, segments objects within the video using "Grounding DINO" and "Segment Anything 2", and extracts object names.
* **Generate Object QA:** Keyframes of objects are used as input to a model ("Qwen 2.5-VL"), which generates captions and questions about the objects. Another model ("Qwen 3") generates object referring expressions.
* **Generate Spatial QA:** A 3D reconstruction of the scene is used to generate spatial questions about the objects. Ground level calibration is performed, and a template is used to generate the questions.
### Key Observations
* The diagram illustrates a pipeline for generating object and spatial questions from video input.
* The process involves video instance segmentation, object recognition, and 3D reconstruction.
* The generated questions are designed to test spatial cognition and object understanding.
### Interpretation
The diagram presents a system that combines video analysis with question generation. The system aims to understand the content of a video, identify objects, and generate questions that require spatial reasoning and object understanding. This type of system could be used in various applications, such as robotics, virtual reality, and education. The use of models like "Grounding DINO", "Segment Anything 2", "Qwen 2.5-VL", and "Qwen 3" suggests the system leverages recent advances in computer vision and natural language processing. The inclusion of both ego-centric and world-centric spatial questions indicates an attempt to capture different perspectives and levels of spatial understanding.
</details>
Figure 2: Embodied Cognition Question-Answer (QA) Data Generation Pipeline: First, objects within the scene are segmented from the video. Subsequently, object and spatial QA pairs are generated via two distinct branches.
3.1 Embodied Cognition Data Generation
Our embodied cognition dataset construction (Fig. 2) begins with egocentric video collection and instance segmentation. One branch employs a human-in-the-loop streaming generation approach to construct various object cognition QA pairs. The other branch utilizes a monocular dense 3D reconstruction method and diverse question templates to generate spatial cognition task QA pairs.
3.1.1 Video Collection and Instance Segmentation
Our egocentric video collection encompasses $200+$ houses, with approximately 100 videos per house. To ensure video quality, we require a resolution of at least 1080p and a frame rate no less than 30fps, using a gimbal to maintain shooting stability. To achieve diversity among different video trajectories, each house is divided into multiple zones, with filming trajectories categorized into single-zone, dual-zone, and tri-zone types. Cross-zone filming enhances diversity by altering the sequence of traversed zones. Additionally, we randomly vary lighting conditions and camera height under different trajectories. We require that each video includes both vertical and horizontal rotations, as well as at least two close-ups of objects, simulating the variable field of view in robotic task execution. Ultimately, we collect 20,832 egocentric videos of indoor movement. To control video length, these videos are segmented every 40 seconds.
Previous works Luo et al. [2025], Wang et al. [2024] adopted a strategy of designing separate data generation processes for each task type, leading to limited data reusability and continuity. We aim to create a lineage among different types of foundational data to reduce unnecessary redundancy in data generation. Therefore, this paper proposes a mask-centric embodied cognition QA generation pipeline. This pipeline initiates with the generation of object masks from video instance segmentation within a scene. First, Qwen2.5-VL Bai et al. [2025] observes the raw video and outputs an object list containing the names of all entity categories in the scene. Utilizing this object list, Grounding DINO 1.5 Ren et al. [2024b] detects objects in key frames at one-second intervals. SAM2 Ravi et al. [2024] assists in segmenting and tracking the objects detected by Grounding DINO 1.5 during the intervening one-second interval. To ensure consistency of instance IDs, the tracking results of old instances are compared with the segmentation results of newly detected instances at key frames. If an instance is found to have overlapping masks (IOU > 0.5), it retains the ID of the old tracking instance. Due to the performance limitations of Grounding DINO 1.5, newly detected object instances may already have appeared in preceding frames yet were missed. Thus, SAM2 conducts a reverse four-second instance tracking for each new object in key frames, thereby achieving full lifecycle instance tracking. In total, we obtain 1.14 million video instance masks from all the egocentric videos.
3.1.2 Object QA Generation
In this work, we generate three types of object-related tasks: object captioning, object comprehension QA, and referring video object segmentation. For each instance, we first divide all frames containing the instance into eight equal parts in chronological order. Within each frame group, an instance key frame is selected based on two factors: the size of the instance in the frame and the distance between the instance center and the frame center. Consequently, each instance is associated with eight instance key frames, featuring good instance visibility and diverse viewing angles. Half of these frames have the instance cropped out using a mask, while the other four highlight the instance using a red bounding box and background dimming technique. The final set of object cue images is displayed within the blue box in Fig. 2.
Due to the limitation of SAM2 in consistent object tracking in egocentric videos, the same instance may be assigned multiple IDs if the instance appears intermittently in the video. We employ an object category filtering method that limits each video to a maximum of two instances per object category, thereby minimizing duplicate instances. The presence of multiple video segments per house leads to repeated occurrences of certain salient objects, causing a pronounced long-tail distribution. We downsample object categories that occur frequently to prevent extreme object distribution. After the aforementioned filtering, the cue image sets of retained instances are input into Qwen2.5-VL Bai et al. [2025], generating object caption and object comprehension QA through various prompts. It is noteworthy that in the object comprehension QA, counting QA task is particularly unique and requires specially designed prompts. Subsequently, based on each instance’s caption and QAs, Qwen3 Yang et al. [2025a] generates two types of referring expressions: simple referring expressions and situational referring expressions. Simple referring expressions identify objects through a combination of features such as spatial location and category. Situational referring expressions establish a task scenario, requiring the model to infer the instance needed by the user within this context. Each type of QA undergoes manual filtering post-output to ensure data quality. Detailed prompts are provided in the Appendix A.2.
3.1.3 Spatial QA Generation
Unlike object QA, spatial QA requires more precise 3D information concerning the global scene context. Therefore, we utilize MASt3R-SLAM Murai et al. [2025] to reconstruct 3D point clouds from RGB videos and obtain camera extrinsic parameters. Subsequently, by projecting 2D pixel points to 3D coordinates, the segmentation of each instance in the video can be mapped onto the point cloud. However, it is important to note that the world coordinate system established by MASt3R-SLAM for the 3D point cloud is not aligned with the floor. Therefore, the Random Sample Consensus (RANSAC) Fischler and Bolles [1981] algorithm is implemented to identify inlier points for plane fitting through ten iterative executions. In each iteration, the detected planar surface and its inliers are removed from the point cloud for subsequent plane detection. Given that the initial camera pose was approximately horizontal but not perpendicular to the ground, the ground plane is selected based on minimal angular deviation between its normal vector and the initial camera Y-axis orientation. The point cloud is then aligned to ensure orthogonality between the world coordinate Z-axis and the detected ground plane.
RynnEC dataset encompasses 10 fundamental spatial abilities, each of which is further divided into quantitative and qualitative variants. We construct spatial QA in a template-based manner. Diverse QA templates are designed according to the characteristics of each task, and the missing attributes within the templates (e.g., distance, height) can be calculated from the 3D point cloud. We denote each instance in the format <Object X>. Furthermore, to obtain purely textual spatial QA pairs, we replace <Object X> with simple referring expressions generated in the above object QA pipeline. These texts are then further refined and diversified using GPT-4o, resulting in the final natural language spatial QA data. With training on these data, RynnEC is able to answer spatial questions in various input forms. Examples of the generated spatial QAs are illustrated in Fig. 2, and more examples as well as detailed templates are provided in the Appendix A.3.
Building on insights from prior works Wu et al. [2025a], Ouyang et al. [2025], we recognize that spatial cognition tasks are highly challenging. Therefore, in addition to constructing a large-scale video-based spatial QA dataset, we also develop a relatively simpler image-based spatial QA dataset. This combination of tasks with varying levels of difficulty is intended to improve learning efficiency and enhance model robustness. Specifically, we collect 500k indoor images from 39k houses. Leveraging the single-image-to-3D reconstruction and calibration methods from SpatialRGPT Cheng et al. [2024a], we obtain the 3D spatial relationships between objects in each image. We then select tasks from the video-based spatial cognition set that can also be addressed via single images, and design corresponding QA templates. The format of the image-based spatial QA is kept consistent with that of the video-based spatial QA.
3.2 RynnEC-Bench
<details>
<summary>x3.png Details</summary>
### Visual Description
## Diagram: RynnEC Bench
### Overview
The image presents a diagram illustrating the RynnEC Bench, a framework for evaluating cognitive abilities related to object properties, spatial cognition, and referring object segmentation. The diagram is structured as a segmented circle, with each segment representing a different cognitive aspect. Each segment contains example questions and answers, along with associated images.
### Components/Axes
* **Central Text:** "RynnEC Bench"
* **Outer Ring Segments (Cognitive Aspects):**
* Referring Object Segmentation (Top)
* Object Properties Cognition (Left)
* Spatial Cognition (Right)
* **Sub-Segments within Object Properties Cognition (Left):**
* Category
* Color
* Material
* Shape
* State
* Position
* Function
* Surface Detail
* Size
* Counting
* **Sub-Segments within Spatial Cognition (Right):**
* Ego-Centric Spatial Cognition
* Present
* Future
* Historical
* World-Centric Spatial Cognition
* Positional Relationship
* Size
* Distance
* **Example Questions and Answers:** Each segment contains a question (Q) and an answer (A), along with associated images. These examples illustrate the type of cognitive task being evaluated.
### Detailed Analysis
**Referring Object Segmentation (Top):**
* **Sub-Categories:** Situational Referring, Direct Referring
* **Situational Referring Example:**
* Q1: If I want to travel and need to carry a lot of clothes, which item should I take?
* Q2: Where is the silver suitcase with a black bag on top?
* A: It is [image of a suitcase].
* **Direct Referring Example:**
* Images show a suitcase in a room from different angles.
**Object Properties Cognition (Left):**
* **Color Example:**
* Q: What is the color of 1?
* A: The 1 is red.
* Image shows an object labeled "1".
* **Function Example:**
* Q: What is the function of 0?
* A: It is to provide a flat, heat-resistant surface for efficiently ironing clothes and removing wrinkles.
* **Counting Example:**
* Q: How many white clothes are near the 1?
* A: 3.
* Images show clothes hanging in a room, with some labeled "1", "0", "2", "3".
**Spatial Cognition (Right):**
* **Ego-Centric Spatial Cognition Example:**
* Q: Upon making a 90-degree left turn, how will 0 be oriented with respect to you?
* A: 0 will located at 11 o'clock direction.
* Q: Is 0 on your left front or left rear?
* A: Left rear.
* Q: How far have you walked in total?
* A: 2.3m.
* Images show a room from different perspectives.
* **World-Centric Spatial Cognition Example:**
* Q: Among the three objects 0, 1, and 2, which one is the tallest?
* A: 1 reaches the greatest height.
* Q: What is the approximate height of 1?
* A: 1 is 0.6m tall.
* Q: Which is closer to 0, 1, or 2?
* A: 1 is closer.
* Images show objects in a room, labeled "0", "1", and "2".
### Key Observations
* The RynnEC Bench covers a range of cognitive abilities related to object recognition, spatial awareness, and language understanding.
* The examples provided are diverse and cover different scenarios.
* The use of images helps to ground the questions in a visual context.
### Interpretation
The RynnEC Bench appears to be a comprehensive framework for evaluating cognitive abilities in AI systems. By testing these abilities, researchers can gain insights into the strengths and weaknesses of different AI models and develop more robust and human-like AI systems. The framework's focus on object properties, spatial cognition, and referring object segmentation suggests that it is particularly well-suited for evaluating AI systems that interact with the physical world. The inclusion of both ego-centric and world-centric spatial cognition is important for evaluating AI systems that can understand and navigate their environment from both a first-person and third-person perspective.
</details>
Figure 3: Overview of embodied cognition dimensions in RynnEC-Bench. RynnEC-Bench includes two subsets: object cognition and spatial cognition, evaluating a total of 22 embodied cognitive abilities.
As this work is the first to propose a comprehensive set of fine-grained embodied video tasks, a robust evaluation framework for assessing MLLMs’ overall capabilities in this domain is currently lacking. To address this, we propose RynnEC-Bench, which evaluates fine-grained embodied understanding models from the perspectives of object cognition and spatial cognition in open-world scenarios. Fig. 3 provides a detailed illustration of the capability taxonomy in RynnEC-Bench.
3.2.1 Capability Taxonomy
Object cognition is divided into two tasks: object properties cognition and referring object segmentation. During embodied task execution, robots often require a clear understanding of key objects’ functions, locations, quantities, surface details, relationships with the surrounding environment, etc. Accordingly, the object properties recognition tasks comprehensively and meticulously construct questions in these aspects. In the processes of robotic manipulation and navigation, identifying operation instances and target instances is an essential step. Precise instance segmentation in videos serves as the best approach to indicate the positions of these key objects. Specifically, the referring object segmentation task is categorized into direct referring problems and situational referring problems. Direct referring problems involve only combinations of descriptions for the instance, while situational referring problems are set within a scenario, requiring MLLMs to perform reasoning in order to identify the target object.
Spatial cognition requires MLLMs to derive a 3D spatial awareness from egocentric video. We categorize it into ego-centric and world-centric spatial cognition. Ego-centric spatial cognition maintains awareness of agent-environment spatial relations and supports spatial reasoning and mental simulation; by temporal scope, we consider past, present, and future cases. World-centric spatial cognition focuses on understanding the 3D layout and scale of the physical world, which we further evaluate in terms of size, distance, and positional relations.
3.2.2 Data Balance
The videos in RynnEC-Bench are collected from ten houses that do not overlap with those in the training set. When evaluating object cognition, we observe substantial variation in object-category distributions across houses, making results highly sensitive to which houses are sampled. To mitigate this bias and better reflect real-world deployment, we introduce a physical-world-based evaluation protocol. We first define a taxonomy of 12 coarse and 119 fine-grained indoor object categories. Using GPT-4o, we then estimate an empirical category-frequency distribution by parsing 500,000 indoor images from 39,000 houses; given the scale, this serves as a close approximation to real-world indoor object frequencies. Finally, we perform frequency-proportional sampling so that the object-category distribution in RynnEC-Bench closely matches the empirical distribution, enabling a more objective and realistic evaluation. Specifically, counting questions with answers of 1 or 2 are reduced by 50% to achieve a more balanced difficulty distribution. All QA pairs in RynnEC-Bench are further subjected to meticulous human screening to ensure high quality. Additional implementation details are available in Appendix B.
3.2.3 Evaluation Framework
The questions are categorized into three types based on the nature of their answers: numerical questions, textual questions, and segmentation questions. For numerical questions such as distance estimation and direction estimation, we directly use the formula to calculate the precise indicators. For scale-related questions, Mean Relative Accuracy (MRA) Yang et al. [2025c], Everingham et al. [2010] is used to calculate the scores. Specifically, given a model’s prediction $\hat{y}$ , ground truth $y$ , and a confidence threshold $\theta$ , relative accuracy is calculated by considering $\hat{y}$ correct if the relative error rate, defined as $|\hat{y}-y|/y$ , is less than $1-\theta$ . As single-confidence-threshold accuracy only considers relative error within a narrow scope, the MRA averages the relative accuracy across a range of confidence thresholds $\mathcal{C}=\{0.5,\,0.55,\,...,\,0.95\}$ :
$$
MRA=\frac{1}{|\mathcal{C}|}\sum_{\theta\in\mathcal{C}}\mathbb{I}\Bigg(\frac{|\hat{y}-y|}{y}<1-\theta\Bigg) \tag{1}
$$
where $\mathbb{I}(·)$ is the indicator function. For angle-related questions, MRA is not suitable due to the cyclic nature of angular measurements. We therefore designed a rotational accuracy metric (RoA).
$$
RoA=1-min\Bigg(\frac{min(|\widehat{y}-y|,360-|\widehat{y}-y|)}{90},1\Bigg) \tag{2}
$$
RoA assigns a score only when the angular difference is less than 90 degrees, ensuring consistency in task difficulty across different settings.
Textual questions are further categorized into close-ended and open-ended questions. For the close-ended part, we prompt GPT-4o to assign a straightforward binary score of either 0 or 1. For the open-ended part, answers are scored by GPT-4o on a scale from 0 to 1 in increments of 0.2. This question-type-adaptive evaluation approach enables the metrics of RynnEC-Bench to be both precise and consistent.
For segmentation evaluation, prior work Yuan et al. [2025a], Yan et al. [2024] typically reports the $\mathcal{J}\&\mathcal{F}$ measure, combining region-overlap ( $\mathcal{J}$ ) and boundary-accuracy ( $\mathcal{F}$ ) scores. However, the conventional frame-averaged $\mathcal{J}\&\mathcal{F}$ treats empty frames (i.e., frames with no ground-truth mask) in a binary manner: if any predicted mask appears, the frame score is set to 0; otherwise it is set to 1. This evaluation method fails to account for the actual size of erroneous masks in empty frames, which can have a significant impact on embodied segmentation tasks. To address this, we propose the Global IoU metric, defined as
$$
\overline{\mathcal{J}}=\frac{\sum_{i=1}^{N}|\mathcal{S}_{i}\cap\mathcal{G}_{i}|}{\sum_{i=1}^{N}|\mathcal{S}_{i}\cup\mathcal{G}_{i}|}, \tag{3}
$$
where $N$ is the total number of video frames, $\mathcal{S}_{i}$ denotes the predicted segmentation mask for frame $i$ , and $\mathcal{G}_{i}$ denotes the ground truth mask for frame $i$ . For the boundary accuracy metric $\overline{\mathcal{F}}$ , we compute the average only over non-empty frames. The mean of $\overline{\mathcal{J}}$ and $\overline{\mathcal{F}}$ , denoted as $\overline{\mathcal{J}}\&\overline{\mathcal{F}}$ , provides an accurate reflection of segmentation quality, especially in egocentric videos where the target object appears in relatively few frames.
<details>
<summary>x4.png Details</summary>
### Visual Description
## Diagram: Multi-Stage Image Understanding Process
### Overview
The image illustrates a four-stage process for image understanding, involving mask alignment, object understanding, spatial understanding, and referring segmentation. Each stage includes a visual representation, a language model (LLM), and various encoders.
### Components/Axes
* **Stages:** The diagram is divided into four stages, labeled Stage 1 to Stage 4.
* **Stage Titles:**
* Stage 1: Mask Alignment
* Stage 2: Object Understanding
* Stage 3: Spatial Understanding
* Stage 4: Referring Segmentation
* **Image Representations:** Each stage contains an image with annotations or segmentations.
* **Language Model (LLM):** Each stage includes a block labeled "LLM" with a flame icon.
* **Encoders:** Each stage includes "Vision Encoder" and "Region Encoder" blocks, with the "Region Encoder" also having a flame icon.
* **Additional Components:** Stage 4 includes "Mask Decoder" and "+ LoRA" blocks, both with flame icons.
* **Icons:** Each stage has a set of icons at the bottom, including video play, image, and document icons.
### Detailed Analysis
**Stage 1: Mask Alignment**
* **Image:** A kitchen scene with a black kettle on a table, highlighted with a yellow bounding box.
* **Caption:** "A single black kettle on the table."
* **Components:** LLM, Vision Encoder, Region Encoder.
**Stage 2: Object Understanding**
* **Image:** A kitchen scene with various objects, including pots, pans, and utensils.
* **Question:** "Q: What is the purpose of <object>?"
* **Answer:** "A: It enhances flavor, adding umami and richness to dishes."
* **Components:** LLM, Vision Encoder, Region Encoder.
**Stage 3: Spatial Understanding**
* **Image:** A living room scene with furniture, including chairs and a table, highlighted with yellow bounding boxes.
* **Question:** "Q: What is the distance of <object0> and <object1>?"
* **Answer:** "A: 0.7m."
* **Components:** LLM, Vision Encoder, Region Encoder.
**Stage 4: Referring Segmentation**
* **Image:** A bedroom scene with a teddy bear on a table, segmented with a green mask.
* **Question:** "Q: Can you segment the the brown teddy bear located on the table in this video?"
* **Answer:** "A: Sure, it is [SEG]."
* **Components:** Mask Decoder, LLM + LoRA, Vision Encoder, Region Encoder.
### Key Observations
* The process progresses from simple object detection and description (Stage 1) to more complex spatial reasoning (Stage 3) and referring segmentation (Stage 4).
* The LLM is a central component in all stages, suggesting its role in understanding and generating responses.
* The Region Encoder is consistently associated with a flame icon, potentially indicating a "hot" or active component.
* Stage 4 introduces a Mask Decoder and LoRA, indicating a more specialized architecture for segmentation tasks.
### Interpretation
The diagram illustrates a multi-stage approach to image understanding, where each stage builds upon the previous one to achieve a higher level of comprehension. The use of LLMs and encoders suggests a deep learning-based architecture. The progression from simple object recognition to spatial reasoning and segmentation demonstrates the system's ability to perform complex visual tasks. The inclusion of LoRA in the final stage suggests fine-tuning or adaptation for specific segmentation tasks. The system appears to be designed to answer questions about images and perform segmentation based on natural language queries.
</details>
Figure 4: Training paradigm of RynnEC. The model is trained in four progressive stages: 1) Mask Alignment, 2) Object Understanding, 3) Spatial Understanding, and 4) Referring Segmentation.
3.3 RynnEC Architecture
RynnEC consists of three core components: the foundational vision-language model for basic multimodal comprehension, a region-aware encoder for fine-grained object-centric representation learning, an adaptive mask decoder for video segmentation tasks. Notably, the latter two modules are designed as plug-and-play components with independent parameter spaces, ensuring architectural flexibility and modular extensibility.
Foundational Vision-Language Model. We ultilize VideoLLaMA3-Image Zhang et al. [2025a] as the foundational vision-language model for RynnEC, which consists of three main modules: a Vision Encoder, the Projector and the Large Language Model (LLM). For the vision encoder, we use VL3-SigLIP-NaViT Zhang et al. [2025a], which leverages an any-resolution vision tokenization strategy to flexibly encode images of varying resolutions. As the LLM, we employ Qwen2.5-1.5B-Instruct Yang et al. [2024] and Qwen2.5-7B-Instruct Yang et al. [2024], enabling scalable trade-offs between performance and computational cost.
Region Encoder. Egocentric videos often feature cluttered scenes with similar objects that are difficult to distinguish using linguistic cues alone. To address this, we introduce a dedicated object encoder for specific object representation. This facilitates more precise cross-modal alignment during training and enables intuitive, fine-grained user interaction at inference time. Following Yuan et al. [2024, 2025c], we use a simple yet efficient MaskPooling for object tokenization, followed by a two-layer projector to align object features with LLM embedding space. During training, object masks spanning multiple frames in a video are utilized to achieve accurate representations. At inference, the encoder offers flexibility, operating effectively with either single-frame or multi-frame object masks.
Mask Decoder. Accurate object localization is critical for egocentric video understanding. To incorporate robust visual grounding capabilities without degrading the model’s pretrained performance, we fine-tune the LLM with LoRA. Our mask decoder is based on the architecture of SAM2 Ravi et al. [2024], which has demonstrated strong generalization capabilities and prior knowledge in purely visual segmentation tasks. For a given video and the instruction, we adpot a [SEG] token as a specifical token to trigger mask generation for the corresponding visual region. To facilitate this process, an additional linear layer is introduced to align the [SEG] token with SAM2’s feature space.
3.4 Training and Inference
As illustrated in Fig. 4, RynnEC is trained using a progressive four-stage pipeline: 1) Mask Alignment, 2) Object Understanding, 3) Spatial Understanding, and 4) Referring Segmentation. The first three stages are designed to incrementally enhance fine-grained, object-centric understanding, while the final stage focuses on equipping the model with precise object-level segmentation capabilities. This curriculum-based approach ensures gradual integration of visual, spatial, and grounding knowledge without overfitting to a single task. The datasets used in each stage are summarized in Tab. 1. The details of each training stage are as follows:
1) Mask Alignment. The goal of this initial stage is to encourage the model to attend to region-specific tokens rather than relying solely on global visual features. We fine-tune both the region encoder and the LLM on a large-scale object-level captioning dataset, where each caption is explicitly aligned with a specific object mask. This alignment training conditions the model to associate object-centric embeddings with corresponding linguistic descriptions, laying the foundation for localized reasoning in later stages.
2) Object Understanding. In this stage, the focus shifts to enriching the model’s egocentric object knowledge, encompassing attributes such as color, shape, material, size, and functional properties. The region encoder and the LLM are jointly fine-tuned to integrate this object-level information more effectively into the cross-modal embedding space. This stage is the basic for spatial understanding.
3) Spatial Understanding. Building on the previous stage, this phase equips the model with spatial reasoning abilities, enabling it to understand and reason about the relative positions and configurations of objects within a scene. We use a large amount of spatial QA we generated and the previous stage data as well as general VQA to maintain the ability to follow instructions.
4) Referring Segmentation. In the final stage, we integrate the Mask Decoder module after the LLM to endow the model with fine-grained referring segmentation capabilities. The LLM is fine-tuned via LoRA to minimize interference with its pretrained reasoning abilities. The training data includes not only segmentation-specific datasets but also samples from earlier stages to mitigate catastrophic forgetting. This multi-task mixture ensures that segmentation performance is improved without sacrificing the model’s object and spatial understanding.
Table 1: Datasets used at four training stages. IM and OM indicate whether the task involves the input mask and output mask, respectively.
| Training Stage | Task | IM | OM | # Samples | Datasets |
| --- | --- | --- | --- | --- | --- |
| Mask Alignment (Stage-1) | General Mask Captioning | ✓ | ✗ | 1.17M | RefCOCO Yu et al. [2016], Mao et al. [2016], VideoRefer-Caption Yuan et al. [2025c], DAM Lian et al. [2025], Osprey-Caption Yuan et al. [2024], MDVP-Data Lin et al. [2024], HC-STVG Tang et al. [2021] |
| Scene Instance Captioning | ✓ | ✗ | 0.14M | RynnEC-Caption | |
| Object Understanding (Stage-2) | Basic Properties QA | ✓ | ✗ | 1.49M | RynnEC-Object |
| Object-Centric Counting | ✓ | ✗ | 0.25M | RynnEC-Counting | |
| Spatial Understanding (Stage-3) | Our Stage-2 | ✓ | ✗ | 0.30M | RynnEC-Object, RynnEC-Counting |
| Spatial QA | ✓ | ✗ | 0.60M | RynnEC-Spatial (Image), RynnEC-Spatial (Video) | |
| ✗ | ✗ | 0.54M | VLM-3R-Data Fan et al. [2025] | | |
| General VQA | ✗ | ✗ | 0.74M | LLaVA-OV-SI Li et al. [2024a], LLaVA-Video Zhang et al. [2024e], ShareGPT-4o-video Chen et al. [2024b], VideoGPT-plus Maaz et al. [2024], FineVideo Farré et al. [2024], CinePile Rawal et al. [2024], ActivityNet Caba Heilbron et al. [2015], YouCook2 Zhou et al. [2018], LLaVA-SFT Liu et al. [2023] | |
| Referring Segmentation (Stage-4) | Our Stage-2 & Stage-3 | ✓ | ✗ | 0.60M | RynnEC-Object, RynnEC-Counting, RynnEC-Spatial |
| General Segmentation | ✗ | ✓ | 0.32M | ADE20K Zhou et al. [2017], COCOStuff Caesar et al. [2018], Mapillary Neuhold et al. [2017], PACO-LVIS Ramanathan et al. [2023], PASCAL-Part Chen et al. [2014] | |
| Embodied Segmentation | ✗ | ✓ | 0.31M | RynnEC-Segmentation | |
| General VQA | ✗ | ✗ | 0.80M | LLaVA-OV-SI Li et al. [2024a], LLaVA-Video Zhang et al. [2024e], ShareGPT-4o-video Chen et al. [2024b], VideoGPT-plus Maaz et al. [2024], FineVideo Farré et al. [2024], CinePile Rawal et al. [2024], ActivityNet Caba Heilbron et al. [2015], YouCook2 Zhou et al. [2018], LLaVA-SFT Liu et al. [2023] | |
4 Experiments
Table 2: Main evaluation results on RynnEC-Bench. We evaluate in two major categories: Object Cognition and Spatial Cognition. DR and SR represent Direct Referring and Situational Referring. PR represents Positional Relationship.
| Model | Overall Mean | Object Cognition | Spatial Cognition | | | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Object Properties | Segmentation | Mean | Ego-Centric | World-Centric | Mean | | | | | | | |
| DR | SR | His. | Pres. | Fut. | Size | Dis. | PR | | | | | |
| Proprietary Generalist MLLMs | | | | | | | | | | | | |
| GPT-4o OpenAI et al. [2024] | 28.3 | 41.1 | — | — | 33.9 | 13.4 | 22.8 | 6.0 | 24.3 | 16.7 | 36.1 | 22.2 |
| GPT-4.1 OpenAI et al. [2024] | 33.5 | 45.9 | — | — | 37.8 | 17.2 | 27.6 | 6.1 | 35.9 | 30.4 | 45.7 | 28.8 |
| Seed1.5-VL Guo et al. [2025] | 34.7 | 52.1 | — | — | 42.8 | 8.2 | 27.7 | 4.3 | 32.9 | 19.1 | 27.9 | 26.1 |
| Genimi-2.5 Pro Comanici et al. [2025] | 45.5 | 64.0 | — | — | 52.7 | 9.3 | 36.7 | 8.1 | 47.0 | 29.9 | 69.3 | 37.8 |
| Open-source Generalist MLLMs | | | | | | | | | | | | |
| VideoLLaMA3-7B Zhang et al. [2025a] | 27.3 | 36.7 | — | — | 30.2 | 5.1 | 26.8 | 1.2 | 30.0 | 19.0 | 34.9 | 24.1 |
| InternVL3-78B Zhu et al. [2025] | 29.0 | 45.3 | — | — | 37.3 | 9.0 | 31.8 | 2.2 | 10.9 | 30.9 | 26.0 | 20.0 |
| Qwen2.5-VL-72B Bai et al. [2025] | 36.4 | 54.2 | — | — | 44.7 | 11.3 | 24.8 | 7.2 | 27.2 | 22.9 | 83.7 | 27.4 |
| Open-source Object-Level MLLMs | | | | | | | | | | | | |
| DAM-3B Lian et al. [2025] | 15.6 | 22.2 | — | — | 18.3 | 2.8 | 14.1 | 1.3 | 28.7 | 6.1 | 18.3 | 12.6 |
| VideoRefer-VL3-7B Yuan et al. [2025c] | 32.9 | 44.1 | — | — | 36.3 | 5.8 | 29.0 | 6.1 | 38.1 | 30.7 | 28.8 | 29.3 |
| Referring Video Object Segmentation MLLMs | | | | | | | | | | | | |
| Sa2VA-4B Yuan et al. [2025a] | 4.9 | 5.9 | 35.3 | 14.8 | 9.4 | 0.0 | 0.0 | 1.3 | 0.0 | 0.0 | 0.0 | 0.0 |
| VideoGlaMM-4B Munasinghe et al. [2025] | 9.0 | 16.4 | 5.8 | 4.2 | 14.4 | 4.1 | 4.7 | 1.4 | 0.8 | 0.0 | 0.3 | 3.2 |
| RGA3-7B Wang et al. [2025] | 10.5 | 15.2 | 32.8 | 23.4 | 17.5 | 0.0 | 5.5 | 6.1 | 1.2 | 0.9 | 0.0 | 3.0 |
| Open-source Embodied MLLMs | | | | | | | | | | | | |
| RoboBrain-2.0-32B Team et al. [2025] | 24.2 | 25.1 | — | — | 20.7 | 8.8 | 34.1 | 0.2 | 37.2 | 30.4 | 3.6 | 28.0 |
| RynnEC-2B | 54.4 | 59.3 | 46.2 | 36.9 | 56.3 | 30.1 | 47.2 | 23.8 | 67.4 | 31.2 | 85.8 | 52.3 |
| RynnEC-7B | 56.2 | 61.4 | 45.3 | 36.1 | 57.8 | 40.9 | 50.2 | 22.3 | 67.1 | 39.2 | 89.7 | 54.5 |
4.1 Implementation Details
4.1.1 Training
In this part, we briefly introduce the implementation details of each training stage. For all stages, we adopt the cosine learning rate scheduler. The warm up ratio of the learning rate is set as 0.03. The maximum token length is set to 16384, while the maximum token length for vision tokens is set to 8192. In Stage 1, both the vision encoder and the LLM are initialized with pretrained weights from VideoLLaMA3-Image. During this stage, we train the LLM, the projector, and the region encoder, using learning rates of $1× 10^{-5}$ , $1× 10^{-5}$ , and $4× 10^{-5}$ , respectively. In Stages 2 and 3, the learning rates for the LLM, projector, and region encoder are adjusted to $4× 10^{-5}$ , $1× 10^{-5}$ , and $1× 10^{-5}$ , respectively. In the final stage, the LLM is fine-tuned using LoRA with the same learning rates as in Stage 3. The learning rate of Mask Decoder is set to $4× 10^{-5}$ .
4.1.2 Evaluation
We present a comprehensive evaluation of five MLLM categories on RynnEC-Bench, including both general-purpose models and those fine-tuned for region-level understanding and segmentation. For models that do not accept direct region-based inputs, we uniformly highlight target objects using bounding boxes in the video. Multiple objects are distinguished by different colored boxes, which are referenced in the question prompt. We observe that general-purpose MLLMs are incapable of localizing objects in videos; thus, only specialist models fine-tuned for this ability are evaluated on the RynnEC-Bench segmentation subset. To ensure a consistent evaluation protocol, videos are sampled at 1 fps up to a maximum of 30 frames. If the initial sampling exceeds the 30-frame limit, these target-containing frames are kept, and the remaining frames are selected via uniform sampling from the rest of the video.
<details>
<summary>x5.png Details</summary>
### Visual Description
## Radar Charts: Object and Spatial Cognition Performance
### Overview
The image presents two radar charts comparing the performance of six different models (Genimi-2.5-Pro, Qwen2.5-VL-72B, VideoRefer-VL3-7B, RoboBrain-2.0-32B, RGA3-7B, and RynnEC-7B) across various cognitive tasks. The left chart focuses on "Object Cognition," while the right chart focuses on "Spatial Cognition." Each axis represents a different task or attribute, and the distance from the center indicates the model's performance on that task.
### Components/Axes
**Legend (Top-Left):**
* **Orange:** Genimi-2.5-Pro
* **Green:** Qwen2.5-VL-72B
* **Blue:** VideoRefer-VL3-7B
* **Red:** RoboBrain-2.0-32B
* **Gray:** RGA3-7B
* **Purple:** RynnEC-7B (Ours)
**Left Chart - (a) Object Cognition:**
* **Axes (Clockwise from Top):** Category, Color, Material, Shape, State, Position, Function, Surface, Size, Situational Seg, Direct Seg, Counting
* **Scale:** The scale is implied to be from 0 to 100, based on the data values.
**Right Chart - (b) Spatial Cognition:**
* **Axes (Clockwise from Top):** Trajectory Review, Egocentric Direction, Egocentric Distance, Movement Imagery, Spatial Imagery, Object Height, Object Size, Object Distance, Absolute Position, Relative Position
* **Scale:** The scale is implied to be from 0 to 100, based on the data values.
### Detailed Analysis
**Left Chart - (a) Object Cognition:**
* **Category:**
* Genimi-2.5-Pro (Orange): ~28
* Qwen2.5-VL-72B (Green): ~69
* VideoRefer-VL3-7B (Blue): ~60
* RoboBrain-2.0-32B (Red): ~15
* RGA3-7B (Gray): ~11
* RynnEC-7B (Purple): ~50
* **Color:**
* Genimi-2.5-Pro (Orange): ~22
* Qwen2.5-VL-72B (Green): ~60
* VideoRefer-VL3-7B (Blue): ~63
* RoboBrain-2.0-32B (Red): ~0
* RGA3-7B (Gray): ~13
* RynnEC-7B (Purple): ~50
* **Material:**
* Genimi-2.5-Pro (Orange): ~14
* Qwen2.5-VL-72B (Green): ~66
* VideoRefer-VL3-7B (Blue): ~68
* RoboBrain-2.0-32B (Red): ~24
* RGA3-7B (Gray): ~7
* RynnEC-7B (Purple): ~56
* **Shape:**
* Genimi-2.5-Pro (Orange): ~16
* Qwen2.5-VL-72B (Green): ~60
* VideoRefer-VL3-7B (Blue): ~60
* RoboBrain-2.0-32B (Red): ~36
* RGA3-7B (Gray): ~22
* RynnEC-7B (Purple): ~66
* **State:**
* Genimi-2.5-Pro (Orange): ~3
* Qwen2.5-VL-72B (Green): ~50
* VideoRefer-VL3-7B (Blue): ~66
* RoboBrain-2.0-32B (Red): ~50
* RGA3-7B (Gray): ~33
* RynnEC-7B (Purple): ~66
* **Position:**
* Genimi-2.5-Pro (Orange): ~6
* Qwen2.5-VL-72B (Green): ~66
* VideoRefer-VL3-7B (Blue): ~70
* RoboBrain-2.0-32B (Red): ~46
* RGA3-7B (Gray): ~25
* RynnEC-7B (Purple): ~60
* **Function:**
* Genimi-2.5-Pro (Orange): ~44
* Qwen2.5-VL-72B (Green): ~63
* VideoRefer-VL3-7B (Blue): ~60
* RoboBrain-2.0-32B (Red): ~30
* RGA3-7B (Gray): ~54
* RynnEC-7B (Purple): ~63
* **Surface:**
* Genimi-2.5-Pro (Orange): ~77
* Qwen2.5-VL-72B (Green): ~49
* VideoRefer-VL3-7B (Blue): ~63
* RoboBrain-2.0-32B (Red): ~25
* RGA3-7B (Gray): ~75
* RynnEC-7B (Purple): ~63
* **Size:**
* Genimi-2.5-Pro (Orange): ~77
* Qwen2.5-VL-72B (Green): ~30
* VideoRefer-VL3-7B (Blue): ~46
* RoboBrain-2.0-32B (Red): ~30
* RGA3-7B (Gray): ~75
* RynnEC-7B (Purple): ~46
* **Situational Seg:**
* Genimi-2.5-Pro (Orange): ~28
* Qwen2.5-VL-72B (Green): ~56
* VideoRefer-VL3-7B (Blue): ~36
* RoboBrain-2.0-32B (Red): ~28
* RGA3-7B (Gray): ~44
* RynnEC-7B (Purple): ~36
* **Direct Seg:**
* Genimi-2.5-Pro (Orange): ~45
* Qwen2.5-VL-72B (Green): ~60
* VideoRefer-VL3-7B (Blue): ~23.5
* RoboBrain-2.0-32B (Red): ~82.8
* RGA3-7B (Gray): ~54
* RynnEC-7B (Purple): ~69
* **Counting:**
* Genimi-2.5-Pro (Orange): ~28
* Qwen2.5-VL-72B (Green): ~69
* VideoRefer-VL3-7B (Blue): ~15
* RoboBrain-2.0-32B (Red): ~11
* RGA3-7B (Gray): ~44
* RynnEC-7B (Purple): ~69
**Right Chart - (b) Spatial Cognition:**
* **Trajectory Review:**
* Genimi-2.5-Pro (Orange): ~30
* Qwen2.5-VL-72B (Green): ~97
* VideoRefer-VL3-7B (Blue): ~41
* RoboBrain-2.0-32B (Red): ~15
* RGA3-7B (Gray): ~29
* RynnEC-7B (Purple): ~90
* **Egocentric Direction:**
* Genimi-2.5-Pro (Orange): ~22
* Qwen2.5-VL-72B (Green): ~77
* VideoRefer-VL3-7B (Blue): ~31
* RoboBrain-2.0-32B (Red): ~21
* RGA3-7B (Gray): ~9
* RynnEC-7B (Purple): ~41
* **Egocentric Distance:**
* Genimi-2.5-Pro (Orange): ~27
* Qwen2.5-VL-72B (Green): ~40
* VideoRefer-VL3-7B (Blue): ~46
* RoboBrain-2.0-32B (Red): ~34
* RGA3-7B (Gray): ~12
* RynnEC-7B (Purple): ~5
* **Movement Imagery:**
* Genimi-2.5-Pro (Orange): ~6
* Qwen2.5-VL-72B (Green): ~25
* VideoRefer-VL3-7B (Blue): ~59
* RoboBrain-2.0-32B (Red): ~36
* RGA3-7B (Gray): ~3
* RynnEC-7B (Purple): ~15
* **Spatial Imagery:**
* Genimi-2.5-Pro (Orange): ~37
* Qwen2.5-VL-72B (Green): ~30
* VideoRefer-VL3-7B (Blue): ~15
* RoboBrain-2.0-32B (Red): ~8
* RGA3-7B (Gray): ~0
* RynnEC-7B (Purple): ~48
* **Object Height:**
* Genimi-2.5-Pro (Orange): ~47
* Qwen2.5-VL-72B (Green): ~39
* VideoRefer-VL3-7B (Blue): ~15
* RoboBrain-2.0-32B (Red): ~6
* RGA3-7B (Gray): ~0
* RynnEC-7B (Purple): ~21
* **Object Size:**
* Genimi-2.5-Pro (Orange): ~67
* Qwen2.5-VL-72B (Green): ~66
* VideoRefer-VL3-7B (Blue): ~59
* RoboBrain-2.0-32B (Red): ~36
* RGA3-7B (Gray): ~3
* RynnEC-7B (Purple): ~48
* **Object Distance:**
* Genimi-2.5-Pro (Orange): ~39
* Qwen2.5-VL-72B (Green): ~66
* VideoRefer-VL3-7B (Blue): ~46
* RoboBrain-2.0-32B (Red): ~34
* RGA3-7B (Gray): ~9
* RynnEC-7B (Purple): ~15
* **Absolute Position:**
* Genimi-2.5-Pro (Orange): ~66
* Qwen2.5-VL-72B (Green): ~93
* VideoRefer-VL3-7B (Blue): ~41
* RoboBrain-2.0-32B (Red): ~31
* RGA3-7B (Gray): ~15
* RynnEC-7B (Purple): ~59
* **Relative Position:**
* Genimi-2.5-Pro (Orange): ~93
* Qwen2.5-VL-72B (Green): ~97
* VideoRefer-VL3-7B (Blue): ~41
* RoboBrain-2.0-32B (Red): ~31
* RGA3-7B (Gray): ~15
* RynnEC-7B (Purple): ~90
### Key Observations
* **Qwen2.5-VL-72B (Green):** Generally performs well across both Object and Spatial Cognition tasks, often achieving the highest scores.
* **Genimi-2.5-Pro (Orange):** Shows variable performance, excelling in some Object Cognition tasks (e.g., Size, Surface) but lagging in others. It also shows good performance in Relative Position.
* **VideoRefer-VL3-7B (Blue):** Performance is generally moderate, with some strengths in specific areas.
* **RoboBrain-2.0-32B (Red):** Tends to have lower scores across most tasks, indicating weaker performance compared to other models.
* **RGA3-7B (Gray):** Generally exhibits low performance, particularly in Spatial Cognition.
* **RynnEC-7B (Purple):** Shows competitive performance, especially in Object Cognition, and performs well in Trajectory Review and Relative Position in Spatial Cognition.
### Interpretation
The radar charts provide a comparative analysis of the cognitive abilities of different models. Qwen2.5-VL-72B appears to be a strong all-around performer, while RoboBrain-2.0-32B and RGA3-7B generally underperform compared to the others. Genimi-2.5-Pro and RynnEC-7B show more specialized strengths. The data suggests that different models have varying proficiencies in different cognitive tasks, highlighting the importance of selecting the appropriate model based on the specific application. The "Ours" designation for RynnEC-7B suggests that this model is the focus of the study, and its performance is being compared against existing models.
</details>
Figure 5: More granular assessments of object cognition and spatial cognition. We compare the best-performing MLLM from each category with our RynnEC-7B.
4.2 Embodied Cognition Evaluation
4.2.1 Main Results
Tab. 2 presents the evaluation results of our RynnEC model and five categories of related MLLMs on the RynnEC-Bench. Although the RynnEC model contains only 7B parameters, it demonstrates robust embodied cognitive abilities, outperforming even the most advanced proprietary model, Gemini-2.5 Pro Comanici et al. [2025], by 10.7 points. Moreover, RynnEC achieves both balanced and superior performance across various tasks. For object cognition, RynnEC achieved a score of 61.4 and possesses the ability to both understand and segment objects. In terms of spatial cognition, RynnEC achieves a score of 54.5, which is 44.2% higher than that of Gemini-2.5 Pro. To support resource-constrained settings, we present a 2B-parameter RynnEC that delivers markedly lower inference latency while maintaining near-parity performance ( $<2$ percentage points drop), enabling on-device deployment for embodied applications. In the following sections, we will introduce the performance of different types of MLLMs on RynnEC-Bench in detail.
Proprietary Generalist MLLMs
Among the four leading proprietary generalist MLLMs evaluated, Gemini-2.5 Pro establishes a clear lead with an overall score of 45.5. This represents a substantial performance margin of 25% over the best open-source generalist MLLM and 38.3% over the premier open-source object-level MLLM. Even more notably, it achieves a remarkable score of 37.8 in the notoriously difficult domain of spatial cognition. This finding provides compelling evidence that spatial awareness can emerge as a byproduct of extensive training on video comprehension tasks.
Open-source Generalist MLLMs
Qwen2.5-VL-72B Bai et al. [2025] exhibits outstanding performance, achieving a score of 36.4 and surpassing GPT-4.1 OpenAI et al. [2024]. This suggests that, in specialized capabilities such as embodied cognition, the gap between open-source and proprietary MLLMs has been significantly narrowed. Furthermore, we observe that Qwen2.5-VL and InternVL3 Zhu et al. [2025] demonstrate superior performance in positional relationship (PR) and distance perception tasks, respectively, even outperforming Gemini-2.5 Pro. Such pronounced differences in various aspects of spatial cognition may be attributed to the distribution of training data.
Open-source Object-Level MLLMs
These MLLMs are capable of accepting region masks as input, enabling more direct localization of target objects and facilitating finer-grained object perception. VideoRefer-VL3-7B Yuan et al. [2025c] is a model fine-tuned from the base model VideoLLaMA3-7B Zhang et al. [2025a]. As shown in Tab. 2, VideoRefer-VL3-7B consistently outperforms VideoLLaMA3-7B in both object cognition and spatial cognition tasks. This demonstrates that, in embodied scenarios, integrating mask understanding within the model is superior to explicit visual prompting.
Referring Video Object Segmentation MLLMs
Recently, several studies have applied MLLMs to object segmentation tasks while retaining the original multimodal understanding capabilities of MLLMs. However, the best-performing model, RGA3-7B Wang et al. [2025], achieves only 15.2 points on the object properties task. Although these MLLMs can still address some general video understanding tasks, their task generalization ability is significantly diminished following segmentation training. In contrast, our RynnEC model, which is specifically designed for embodied scenarios, maintains strong object and spatial understanding capabilities even after segmentation training.
Open-source Embodied MLLMs
With the growing demand for highly generalizable cognitive abilities in the field of embodied intelligence, a number of studies have begun to develop MLLMs specifically tailored for embodied scenarios. A representative model is RoboBrain-2.0 Team et al. [2025], which achieves 24.2 even worse than general-purpose video models such as VideoLLaMA3-7B. There are two primary reasons for this: (1) Loss of object cognition: Embodied MLLMs typically emphasize spatial perception and task planning abilities, but tend to overlook the importance of detailed object understanding. (2) Lack of fine-grained perceptual capability: In egocentric videos, RoboBrain-2.0 demonstrates limited ability to interpret region-level features.
4.2.2 Object Cognition
Fig. 5 (a) presents a more comprehensive evaluation of RynnEC’s capability in object properties cognition from multiple dimensions. Since most object properties cognition abilities are encompassed by general video understanding skills, Gemini-2.5-Pro exhibits superior performance across various competencies. However, due to the high edge deployment requirements of embodied MLLMs, the inference speed of these large-scale models becomes a bottleneck. With only 7B parameters, RynnEC achieves object properties cognition comparable to that of Gemini-2.5-Pro in most categories. Notably, for attributes such as surface details, object state, and object shape, RynnEC-2B even surpasses all other MLLMs. Moreover, most MLLMs lack video object segmentation capabilities, whereas dedicated segmentation MLLMs often sacrifice understanding abilities. RynnEC, while maintaining strong comprehension capabilities, achieves 30.9% and 57.7% improvements over state-of-the-art segmentation MLLMs in direct referring and situational referring object segmentation tasks, respectively.
4.2.3 Spatial Cognition
Fig. 5 (b) demonstrates RynnEC’s spatial cognition capabilities through more fine-grained tasks. As spatial abilities have not been formally defined or systematically explored in previous work, different MLLMs only exhibit strengths in a limited set of specific skills. Overall, spatial cognition abilities such as Spatial Imagery, Movement Imagery, and Trajectory Review are typically absent in prior MLLMs. In contrast, RynnEC possesses a more comprehensive set of spatial abilities, which can facilitate embodied agents in developing spatial awareness within complex environments.
<details>
<summary>x6.png Details</summary>
### Visual Description
## Bar Chart: Model Performance Comparison
### Overview
The image is a bar chart comparing the performance of two models, VideoLLaMA3-7B and RynnEC-7B, across various tasks. The chart displays the scores achieved by each model on different categories, providing a visual comparison of their capabilities.
### Components/Axes
* **Y-axis:** "Score", ranging from 0 to 70 in increments of 10.
* **X-axis:** Categories: "Abs. Dist.", "Route Plan", "Rel. Dir. Hard", "Rel. Dist.", "Rel. Dir. Medium", "Rel. Dir. Easy", "Obj. Count", "Obj. Size", "Room Size", "Appear. Order".
* **Legend:** Located at the top of the chart.
* VideoLLaMA3-7B (Avg. Score: 35.8) - Represented by light blue bars.
* RynnEC-7B (Avg. Score: 45.8) - Represented by dark blue bars.
### Detailed Analysis
Here's a breakdown of the scores for each category, along with trend observations:
* **Abs. Dist.:**
* VideoLLaMA3-7B: 23.5
* RynnEC-7B: 25.4
* **Route Plan:**
* VideoLLaMA3-7B: 32
* RynnEC-7B: 38.7
* **Rel. Dir. Hard:**
* VideoLLaMA3-7B: 30
* RynnEC-7B: 42.9
* **Rel. Dist.:**
* VideoLLaMA3-7B: 39.4
* RynnEC-7B: 44.2
* **Rel. Dir. Medium:**
* VideoLLaMA3-7B: 46.3
* RynnEC-7B: 51.9
* **Rel. Dir. Easy:**
* VideoLLaMA3-7B: 45.2
* RynnEC-7B: 53.5
* **Obj. Count:**
* VideoLLaMA3-7B: 41.9
* RynnEC-7B: 58.5
* **Obj. Size:**
* VideoLLaMA3-7B: 42.2
* RynnEC-7B: 54.9
* **Room Size:**
* VideoLLaMA3-7B: 27.1
* RynnEC-7B: 42.7
* **Appear. Order:**
* VideoLLaMA3-7B: 31.4
* RynnEC-7B: 30.5
### Key Observations
* RynnEC-7B consistently outperforms VideoLLaMA3-7B across all categories.
* The largest performance difference is observed in "Obj. Count", where RynnEC-7B scores significantly higher (58.5) than VideoLLaMA3-7B (41.9).
* The smallest performance difference is observed in "Appear. Order", where VideoLLaMA3-7B scores 31.4 and RynnEC-7B scores 30.5.
### Interpretation
The bar chart provides a clear comparison of the two models' performance on various tasks. RynnEC-7B demonstrates a superior ability to handle these tasks compared to VideoLLaMA3-7B, as evidenced by its consistently higher scores. The "Obj. Count" category highlights a specific area where RynnEC-7B excels. The average scores provided in the legend (VideoLLaMA3-7B: 35.8, RynnEC-7B: 45.8) further support this conclusion. The chart suggests that RynnEC-7B is a more robust model for the tasks evaluated.
</details>
| Models | VSI-Bench |
| --- | --- |
| Qwen2.5-VL-7B Bai et al. [2025] | 35.9 |
| InternVL3-8B Zhu et al. [2025] | 42.1 |
| GPT-4o OpenAI et al. [2024] | 43.6 |
| Magma-8B Yang et al. [2025b] | 12.7 |
| Cosmos-Reason1-7B Azzolini et al. [2025] | 25.6 |
| VeBrain-8B Luo et al. [2025] | 26.3 |
| RoboBrain-7B-1.0 Ji et al. [2025] | 31.1 |
| RoboBrain-7B-2.0 Team et al. [2025] | 36.1 |
| M2-Reasoning-7B AI et al. [2025] | 42.3 |
| ViLaSR Wu et al. [2025b] | 45.4 |
| RynnEC-7B | 45.8 |
Figure 6: Performance on VSI-Bench Yang et al. [2025c]. Left: per-subtask comparison with VideoLLaMA3, the base model of our RynnEC. Right: overall comparison with generalist MLLMs and embodied MLLMs without explicit 3D encoding.
4.3 Generalization and Scalability
To investigate the generalizability of RynnEC, we conduct experiments on VSI-Bench Yang et al. [2025c], a purely textual spatial intelligence benchmark. As shown in Fig. 6, RynnEC-7B consistently surpasses VideoLLaMA3-7B across almost all capability dimensions. Notably, RynnEC is trained with a mask-centric spatial awareness paradigm, whereas all tasks in VSI-Bench involve purely textual spatial reasoning. This demonstrates that spatial awareness need not be constrained by the modality of representation, and spatial reasoning abilities can be effectively transferred across modalities. Further observation reveals substantial performance gains of RynnEC on the Route Planning task, despite this task not being included during training. This indicates that the navigation performance of embodied agents is currently constrained by foundational spatial perception capabilities, such as the understanding of direction, distance, and spatial relationships. Only with robust foundational spatial cognition can large embodied models achieve superior performance in high-level planning and decision-making tasks. Compared to other embodied MLLMs of comparable size, RynnEC-7B also achieves a leading score of 45.8.
Certain tasks, such as object segmentation and movement imagery, remain significant challenges for RynnEC. We hypothesize that the suboptimal performance on these tasks stems primarily from insufficient training data. To validate this, we conduct an empirical analysis of data scalability across different task categories. As the data volume increases progressively from 20% to 100%, the model’s performance on all tasks improves steadily. This observation motivates further expansion of the dataset to enhance RynnEC’s spatial reasoning capabilities. However, it is noteworthy that the marginal gains diminish as data volume grows, indicating a decreasing return on scale. Investigating strategies to enhance data diversity in order to sustain this scaling behavior remains a critical open challenge for future research.
<details>
<summary>x7.png Details</summary>
### Visual Description
## Task Execution Demonstration
### Overview
The image shows a sequence of steps for two tasks performed in a simulated environment. Task 1 involves placing a basketball into a white box near a tennis racket. Task 2 involves reducing the number of plates on a dining table to five and placing the removed plates to the left of a laptop. Each task is broken down into steps, with each step showing a visual representation of the environment, a question-and-answer exchange, and an action performed.
### Components/Axes
The image is divided into two rows, one for each task. Each task is further divided into three steps, each represented by a set of images and text.
**Task 1:**
* **Task Description:** "Put the basketball in the white box beside the tennis racket."
* **Step 1:** "Walk up to the basketball and pick it up"
* **Question:** "Where is the basketball?"
* **Answer:** "The basketball is <object1>"
* **Question:** "Is the <object1> above or below the table?"
* **Answer:** "<object1> is below the table"
* **Action:** "Go straight and then turn left", "Crouch down and pull the ball out from under the table."
* **Step 2:** "Walk up to the white box beside the tennis racket"
* **Question:** "Where is the white box beside the tennis racket?"
* **Answer:** "<object2> is directly behind me."
* **Question:** "How far is the <object2> from me?"
* **Answer:** "<object2> is 1.2 meters away from me."
* **Action:** "Turn around.", "Walk straight ahead for 1 meter."
* **Step 3:** "Put the basketball into the white box"
* **Question:** "How wide is the <object2>? Can it hold a basketball?"
* **Answer:** "The width of <object2> is approximately 0.22 meters. It can hold a basketball, but cannot accommodate it."
* **Action:** "Carefully place the basketball on the white box."
**Task 2:**
* **Task Description:** "Then reduce the number of plates on the dining table to five, and place the removed plates to the left of the laptop."
* **Step 1:** "Walk to the dining table and count how many plates there are."
* **Question:** "Where is the dining table?"
* **Answer:** "The dining table is <object3>"
* **Question:** "How many plates are there on the <object3>?"
* **Answer:** "Six"
* **Action:** "Turn right to view the entire dining table", "Since we need to leave 5 plates behind, we need to pick up one plate."
* **Step 2:** "Pick up the removed plate and walk over to the laptop."
* **Question:** "Which plate is the closest to me?"
* **Answer:** "<object4> is the closest to me"
* **Question:** "In what direction is the laptop located relative to me, and how far away is it?"
* **Answer:** "The laptop is in my one o'clock position and is 4.5 meters away from me."
* **Action:** "Pick up <object4>", "Turn right by 30 degrees and then go straight for 4 meters."
* **Step 3:** "Place the removed plate on the left side of the laptop."
* **Question:** "Has the plate been placed on the left side of the laptop?"
* **Answer:** "Yes"
* **Action:** "Send out a signal indicating that the task has been completed."
### Detailed Analysis or Content Details
The image provides a visual and textual representation of an agent performing tasks in a simulated environment. Each step includes a question-and-answer interaction, providing context and reasoning for the actions taken. The actions are described in text and visually demonstrated in the images.
### Key Observations
* The agent successfully completes both tasks.
* The agent uses a question-and-answer system to guide its actions.
* The actions are described in a clear and concise manner.
* The visual representations provide a clear understanding of the environment and the agent's actions.
### Interpretation
The image demonstrates the ability of an agent to perform tasks in a simulated environment. The question-and-answer system allows the agent to reason about its actions and make informed decisions. The visual representations provide a clear understanding of the environment and the agent's actions, making it easy to follow the task execution. The tasks are simple, but they demonstrate the potential for more complex tasks to be performed in a similar manner. The use of <object> tags suggests that the agent is able to identify and interact with specific objects in the environment. The distances provided in the answers suggest that the agent has a sense of spatial awareness.
</details>
Figure 7: The example of RynnEC assisting robots in performing long-range tasks. The robot accomplishes the two designated tasks within the RoboTHOR simulator Deitke et al. [2020]. RynnEC facilitates the robot in achieving fine-grained environmental cognition throughout the task execution.
4.4 Embodied Application
Recently, some works Cheng et al. [2024b], Xiong et al. [2024] have leveraged MLLMs as the "brain" to assist robots in planning tasks, perceiving environments, and making decisions. However, current MLLMs lack key capabilities such as spatial awareness, fine-grained perception, and instance localization, which restricts these applications to limited and simple tasks. As illustrated in Fig. 7, RynnEC demonstrates the potential to assist robots in accomplishing long-horizon tasks within complex environments. From two real-time tasks performed by the robot equipped with RynnEC, we observe the following roles that RynnEC plays in task execution: (1) Fine-grained object localization and understanding enable robots to more quickly identify target objects and assess their states; (2) Direction and distance perception of targets improves navigation efficiency and precision; (3) Spatial scale estimation empowers robots to perform more delicate manipulations; (4) Counting ability facilitates the completion of tasks requiring mathematical reasoning. It is important to emphasize that the role of RynnEC in embodied tasks is far from limited to these examples. We hope that more researchers will integrate RynnEC models into robotic systems across a wide range of tasks, thereby advancing embodied intelligence toward more valuable real-world applications.
5 Conclusion and Future Works
In this paper, we introduce RynnEC, a Video MLLM for embodied cognition. Through the architectural design of a region encoder and mask decoder, RynnEC achieves flexible, fine-grained visual interaction. Meanwhile, RynnEC demonstrates robust object and spatial cognitive abilities with compact size. To address the limitations of available scene data, we employ a data generation pipeline that relies solely on RGB videos. Furthermore, to supplement the lack of fine-grained embodied cognition benchmarks, we propose RynnEC-Bench, which covers 22 categories of object and spatial cognitive abilities. During training, RynnEC progressively integrates diverse skills through a four-stage capability injection process. Importantly, we advocate that fine-grained video-based visual understanding is key to achieving generalizable cognition in the physical world. RynnEC will enable robots to accomplish more precise cognitive tasks, thereby advancing the practical development of embodied intelligence.
We regard RynnEC as a foundational step toward developing a general embodied intelligence model. Looking ahead, we plan to further advance RynnEC along two primary directions.
- Enhancing Reasoning Capabilities: Robust visual reasoning is essential for solving any complex embodied task. An important research direction is how to effectively integrate RynnEC’s diverse abilities to perform joint reasoning, thereby enabling the resolution of higher-level embodied problems.
- Unified Perception and Planning Framework: Recent studies Team et al. [2025] have started to explore training unified embodied intelligence models that combine perception and planning. However, these approaches are limited in their ability to facilitate fine-grained, video-based visual interactions. In the future, we aim to endow RynnEC with more flexible planning abilities and integrate it with VLA models to form a closed-loop embodied system.
References
- AI et al. [2025] Inclusion AI, Fudong Wang, Jiajia Liu, Jingdong Chen, Jun Zhou, Kaixiang Ji, Lixiang Ru, Qingpei Guo, Ruobing Zheng, Tianqi Li, et al. M2-reasoning: Empowering mllms with unified general and spatial reasoning. arXiv preprint arXiv:2507.08306, 2025.
- Ataallah et al. [2024] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024.
- Azzolini et al. [2025] Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558, 2025.
- Bai et al. [2025] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. https://arxiv.org/abs/2502.13923.
- Caba Heilbron et al. [2015] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
- Caesar et al. [2018] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018.
- Chen et al. [2024a] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024a.
- Chen et al. [2024b] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472–19495, 2024b.
- Chen et al. [2014] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1971–1978, 2014.
- Cheng et al. [2024a] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.01584, 2024a.
- Cheng et al. [2024b] Kai Cheng, Zhengyuan Li, Xingpeng Sun, Byung-Cheol Min, Amrit Singh Bedi, and Aniket Bera. Efficienteqa: An efficient approach for open vocabulary embodied question answering. arXiv preprint arXiv:2410.20263, 2024b.
- Comanici et al. [2025] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
- Dang et al. [2025] Ronghao Dang, Yuqian Yuan, Wenqi Zhang, Yifei Xin, Boqiang Zhang, Long Li, Liuyi Wang, Qinyang Zeng, Xin Li, and Lidong Bing. Ecbench: Can multi-modal foundation models understand the egocentric world? a holistic embodied cognition benchmark. arXiv preprint arXiv:2501.05031, 2025.
- Deitke et al. [2020] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. Robothor: An open simulation-to-real embodied ai platform, 2020.
- Doveh et al. [2025] Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision and language models. In European Conference on Computer Vision, pages 250–267. Springer, 2025.
- Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–338, 2010.
- Fan et al. [2025] Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Dilin Wang, Zhicheng Yan, et al. Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction. arXiv preprint arXiv:2505.20279, 2025.
- Farré et al. [2024] Miquel Farré, Andi Marafioti, Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Finevideo. https://huggingface.co/datasets/HuggingFaceFV/finevideo, 2024.
- Fischler and Bolles [1981] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381–395, 1981.
- Guo et al. [2025] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1.5-vl technical report. arXiv preprint arXiv:2505.07062, 2025.
- Han et al. [2025a] Xiaofeng Han, Shunpeng Chen, Zenghuang Fu, Zhe Feng, Lue Fan, Dong An, Changwei Wang, Li Guo, Weiliang Meng, Xiaopeng Zhang, et al. Multimodal fusion and vision-language models: A survey for robot vision. arXiv preprint arXiv:2504.02477, 2025a.
- Han et al. [2025b] Xiaofeng Han, Shunpeng Chen, Zenghuang Fu, Zhe Feng, Lue Fan, Dong An, Changwei Wang, Li Guo, Weiliang Meng, Xiaopeng Zhang, et al. Multimodal fusion and vision-language models: A survey for robot vision. arXiv preprint arXiv:2504.02477, 2025b.
- Hong et al. [2025] Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5344–5354, 2025.
- Hou et al. [2025] Xiaolu Hou, Mingcheng Li, Dingkang Yang, Jiawei Chen, Ziyun Qian, Xiao Zhao, Yue Jiang, Jinjie Wei, Qingyao Xu, and Lihua Zhang. Bloomscene: Lightweight structured 3d gaussian splatting for crossmodal scene generation. arXiv preprint arXiv:2501.10462, 2025.
- Ji et al. [2025] Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1724–1734, 2025.
- Jin et al. [2024] Shiyu Jin, Jinxuan Xu, Yutian Lei, and Liangjun Zhang. Reasoning grasping via multimodal large language model. arXiv preprint arXiv:2402.06798, 2024.
- Li et al. [2024a] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
- Li et al. [2024b] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024b.
- Li et al. [2024c] Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18061–18070, 2024c.
- Li et al. [2025a] Yifan Li, Yuhang Chen, Anh Dao, Lichi Li, Zhongyi Cai, Zhen Tan, Tianlong Chen, and Yu Kong. Industryeqa: Pushing the frontiers of embodied question answering in industrial scenarios. arXiv preprint arXiv:2505.20640, 2025a.
- Li et al. [2025b] Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. Sti-bench: Are mllms ready for precise spatial-temporal world understanding? arXiv preprint arXiv:2503.23765, 2025b.
- Lian et al. [2025] Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504.16072, 2025.
- Lin et al. [2023] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
- Lin et al. [2024] Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want. arXiv preprint arXiv:2403.20271, 2024.
- Lin et al. [2025] Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos, 2025.
- Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.
- Luo et al. [2025] Gen Luo, Ganlin Yang, Ziyang Gong, Guanzhou Chen, Haonan Duan, Erfei Cui, Ronglei Tong, Zhi Hou, Tianyi Zhang, Zhe Chen, et al. Visual embodied brain: Let multimodal large language models see, think, and control in spaces. arXiv preprint arXiv:2506.00123, 2025.
- Lyu et al. [2024] Ruiyuan Lyu, Jingli Lin, Tai Wang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, and Jiangmiao Pang. Mmscan: A multi-modal 3d scene dataset with hierarchical grounded language annotations. Advances in Neural Information Processing Systems, 37:50898–50924, 2024.
- Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
- Maaz et al. [2024] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.09418, 2024.
- Majumdar et al. [2024] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16488–16498, 2024.
- Mao et al. [2016] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016.
- Munasinghe et al. [2025] Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. Videoglamm: A large multimodal model for pixel-level visual grounding in videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19036–19046, 2025.
- Murai et al. [2025] Riku Murai, Eric Dexheimer, and Andrew J Davison. Mast3r-slam: Real-time dense slam with 3d reconstruction priors. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 16695–16705, 2025.
- Neuhold et al. [2017] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990–4999, 2017.
- OpenAI et al. [2024] OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Alan Hayes. Gpt-4o system card, 2024.
- Ouyang et al. [2025] Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Spacer: Reinforcing mllms in video spatial reasoning, 2025. https://arxiv.org/abs/2504.01805.
- Pasqualotto and Proulx [2012] Achille Pasqualotto and Michael J Proulx. The role of visual experience for the neural basis of spatial cognition. Neuroscience & Biobehavioral Reviews, 36(4):1179–1187, 2012.
- Qi et al. [2025] Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01428, 2025.
- Ramanathan et al. [2023] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7141–7151, 2023.
- Ravi et al. [2024] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.
- Rawal et al. [2024] Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024.
- Ray et al. [2024] Arijit Ray, Jiafei Duan, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, Kuo-Hao Zeng, et al. Sat: Spatial aptitude training for multimodal language models. arXiv preprint arXiv:2412.07755, 2024.
- Ren et al. [2024a] Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confident: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403.15941, 2024a.
- Ren et al. [2024b] Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300, 2024b.
- Tang et al. [2021] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8238–8249, 2021.
- Team et al. [2025] BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, et al. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.02029, 2025.
- Team et al. [2024] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, and Damien Vincent. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
- Wang et al. [2025] Haochen Wang, Qirui Chen, Cilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie, and Stratis Gavves. Object-centric video question answering with visual grounding and referring. arXiv preprint arXiv:2507.19599, 2025.
- Wang et al. [2024] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19757–19767, 2024.
- Wu et al. [2025a] Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505.23747, 2025a.
- Wu et al. [2023] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE, 2023.
- Wu et al. [2025b] Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, and Tieniu Tan. Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing. arXiv preprint arXiv:2506.09965, 2025b.
- Xiong et al. [2024] Chuyan Xiong, Chengyu Shen, Xiaoqi Li, Kaichen Zhou, Jiaming Liu, Ruiping Wang, and Hao Dong. Autonomous interactive correction mllm for robust robotic manipulation. In 8th Annual Conference on Robot Learning, 2024.
- Xu et al. [2025] Runsen Xu, Weiyao Wang, Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Dahua Lin, Matt Feiszli, and Kevin J Liang. Multi-spatialmllm: Multi-frame spatial understanding with multi-modal large language models. arXiv preprint arXiv:2505.17015, 2025.
- Yan et al. [2024] Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In European Conference on Computer Vision, pages 98–115. Springer, 2024.
- Yang et al. [2024] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
- Yang et al. [2025a] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
- Yang et al. [2025b] Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14203–14214, 2025b.
- Yang et al. [2025c] Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025c.
- Yeshwanth et al. [2023] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias NieSSner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes, 2023.
- Yu et al. [2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European conference on computer vision, pages 69–85. Springer, 2016.
- Yuan et al. [2025a] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025a.
- Yuan et al. [2024] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202–28211, 2024.
- Yuan et al. [2025b] Yuqian Yuan, Ronghao Dang, Long Li, Wentong Li, Dian Jiao, Xin Li, Deli Zhao, Fan Wang, Wenqiao Zhang, Jun Xiao, and Yueting Zhuang. Eoc-bench: Can mllms identify, recall, and forecast objects in an egocentric world?, 2025b. https://arxiv.org/abs/2506.05287.
- Yuan et al. [2025c] Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video llm. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18970–18980, 2025c.
- Zhang et al. [2025a] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025a. https://arxiv.org/abs/2501.13106.
- Zhang et al. [2024a] Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024a.
- Zhang et al. [2023] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023.
- Zhang et al. [2025b] Haoyu Zhang, Qiaohui Chu, Meng Liu, Yunxiao Wang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding. arXiv preprint arXiv:2503.09143, 2025b.
- Zhang et al. [2025c] Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, et al. From flatland to space: Teaching vision-language models to perceive and reason in 3d. arXiv preprint arXiv:2503.22976, 2025c.
- Zhang et al. [2024b] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024b.
- Zhang et al. [2024c] Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the out-of-distribution generalization of multimodal large language models. arXiv preprint arXiv:2402.06599, 2024c.
- Zhang et al. [2024d] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024d.
- Zhang et al. [2024e] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024e.
- Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
- Zhou et al. [2018] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
- Zhu et al. [2025] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, and Jiahao Wang. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. https://arxiv.org/abs/2504.10479.
\beginappendix
A Implementation Details for Data Pipeline
A.1 Instance Segmentation
As described in Section 3.1.1, instance segmentation and tracking in videos require a three-stage collaborative process. The first stage involves the extraction of an object list, which should comprehensively include the names of all objects present in the video scene. After evaluating multiple approaches, we find that directly leveraging Qwen2.5-VL to extract object category names from video frames achieves the highest efficiency and accuracy. Specifically, we uniformly sample 16 frames from each video, dividing them into two groups: even-numbered frames and odd-numbered frames. Each group is then processed independently by Qwen2.5-VL to generate a list of object category names. The prompt used to guide the extraction of the object list is in Tab. 3.
System Prompt: Please analyze the image sequence captured as I move through an indoor environment and provide a concise list of major distinct physical objects that can be detected and segmented in these scenes. You need to pay attention to the following points (1) Focus on tangible items such as furniture, appliances, and tools. Avoid nouns that denote locations and rooms like "kitchen" or "bedroom". (2) Limit the list to a maximum of 20 objects, and avoid including specific components or parts of these objects. (3) Ensure the list does not have duplicates. Your output must be a series of nouns separated by semicolons
Table 3: Prompts for object list extraction.
During our experimentation, we observe that Qwen2.5-VL occasionally produces repeated instances of the same object name or phrases sharing the same object name as a prefix. To address this, we apply a post-processing step to remove duplicate and semantically similar phrases from the model outputs, thereby ensuring the diversity and conciseness of the object list. The final object list is obtained by taking the union of the results from the odd-numbered and even-numbered frame groups, yielding a more comprehensive and robust set of detected objects. Furthermore, generic scene-level categories such as "wall" and "floor" are explicitly excluded from the final object list, as they are not considered relevant instances for downstream instance-level tracking and segmentation tasks.
A.2 Object QA Generation
We generate three categories of object-related tasks: object caption, object comprehension QA, and referring video object segmentation. The pre-annotation prompts for object caption and object comprehension QA are presented in Tab. 4. Both tasks take as input a set of keyframes in which the target object is highlighted; the only difference lies in the task-specific instruction prompts.
The referring video object segmentation task requires generating unique referring expressions for objects. We aggregate the QAs generated in the previous stage for each object, representing various attributes of the object. Subsequently, Qwen3 utilizes these QAs to generate both direct referring expressions and situational referring expressions. The specific prompt is shown in Tab. 5.
Crop Image Prompt: The above four images show a crop of the object we need to describe. Bbox Image Prompt: The four images above highlight the target object with a red bounding box and dimmed background. Task Prompt: Caption Task: Please provide a detailed description of the specified object, focusing on its color, material, shape, state, position, function, surface detail and other information. (1) Stick to a narrative format for descriptions, avoiding list-like itemizations. (2) Just output the information you are sure of, if you output the wrong information you will be fired. Comprehension QA Task: I need you to generate a series of question pairs for me about this object, using <object> to represent the object in the question pairs. You can focus on its category, color, material, shape, state, position, function, surface detail, size and other information. "Output example" Question: What color is the <object>? Answer: Mainly red, with some blue as decoration. Notes: (1) The object in all images is the same; QA should focus solely on it, without referencing specific images. (2) Ask as many questions as needed—the more details, the better. (3) Prioritize reasoning and spatial understanding questions over simple ones. (4) You can ask questions about the target object by associating it with the surrounding objects (e.g., comparison, spatial relationship, functional relationship, quantitative relationship, etc.). # Python code together with above text prompts are directly sent to LLaMA ⬇ messages = [{"role": "user", "content": [ {"type": "text", "text": "Crop Image Prompt"} + crop_image_list + {"type": "text", "text": "Bbox Image Prompt"} + bbox_image_list + {"type": "text", "text": "Task Prompt"}]}]
Table 4: Prompts for object caption and comprehension QA generation. Separate textual instructions are provided for the cropped images and the images highlighting the object via bounding boxes, respectively.
System Prompt: You are analyzing indoor objects. Given a series of QAs about a single object (marked as <object>), use the information to generate two referring expressions that uniquely identify it. The two expressions should be: •
One simple referring expression, using attributes such as category, color, material, spatial location, or function. •
One situational referring expression, involving contextual reasoning and diverse sentence structures. Input Example: Question: What is the primary function of the <object>? Answer: The <object> is primarily used for holding writing instruments like pens and pencils. (Additional QA pairs continue in a similar fashion—omitted for brevity.) Output Example: [simple expression] The cylindrical light brown pen holder on the top shelf of the desk. [complex expression] If I finish writing with a pencil, where is the best place to store it?
Table 5: Prompt for object referring expression generation.
A.3 Spatial QA Generation
As outlined in Section 3.1.3, we adopt a template-based approach for generating spatial QA. Specifically, we define 14 core spatial abilities and create a total of 30 distinct templates, with each template encompassing at least three different question structures. Some examples of QA templates are provided in Listing LABEL:lst:qa_template.
Listing 1: Template examples for Spatial QA generation.
⬇
camera_distance_questions = [
" How far have you walked in total?",
" What is the total distance you have covered walking?",
" What is the overall distance you have walked?"
]
closer_to_camera_questions = [
" Which is closer to you, [A] or [B]?",
" Between [A] and [B], which one is nearer to you?",
" Which one is closer to you, [A] or [B]?"
]
closest_to_camera_questions = [
" Which is closest to you, [A] or [B] or [C]?",
" Among [A], [B], and [C], which one is nearer to you?",
" Which of [A], [B], or [C] is closest to you?"
]
future_direction_camera_questions = [
" After you turn 90 degrees to the left, where will [A] be in relation to you?",
" If you turn left by 90- degree, in which direction will [A] be positioned?",
" Upon making a 90- degree left turn, how will [A] be oriented with respect to you?"
]
future_direction_camera_rotate_questions = [
" How many degrees clockwise do you need to turn to face the direction of [A]?",
" To face towards [A], how many degrees should you rotate in a clockwise direction?",
" What degree of clockwise rotation is necessary for you to face [A]’ s direction?"
]
distance_questions_3 = [
" Which of the three objects, [A], [B], or [C], is closest to you?",
" Among [A], [B], and [C], which object is nearest to you?",
" Between [A], [B], and [C], which one is the closest to you?",
]
height_from_ground_questions = [
" What is the height difference above ground level between [A] and [B]?",
" How much higher or lower is [A] compared to [B] above the ground?",
" By what amount does the elevation of [A] differ from that of [B]?"
]
center_distance_questions = [
" What is the distance between the centers of [A] and [B]?",
" How far apart are the centers of [A] and [B]?",
" What is the separation between the central points of [A] and [B]?"
]
tall_choice_questions_3 = [
" Among the three objects [A], [B], and [C], which one is the tallest?",
" Which of the three objects [A], [B], and [C] is tallest?",
" Out of the three objects [A], [B], and [C], which one is the tallest?",
]
above_predicate_questions = [
" Is [A] above [B]?",
" Does [A] appear over [B]?",
" Can you confirm if [A] is positioned above [B]?",
]
B Details of RynnEC-Bench Construction
As described in Section 3.2.2, we adjust the object distribution in the object properties understanding evaluation set of RynnEC-Bench based on real-world object category frequencies. The detailed object categorization strategy is presented in Tab. 6. We classify common indoor objects into 12 coarse categories and 119 fine-grained categories. Objects not falling into any of these predefined categories are assigned to an "other" class. A function-centered taxonomy is adopted: objects with similar appearances but distinct functional roles are categorized separately.
To construct this evaluation set, we follow a two-stage process. First, an initial, oversized pool of 20,000 question-answer (QA) pairs is randomly generated without distributional constraints. Following this, we downsample this pool to a target size of 10,000 pairs. The sampling is performed according to the real-world object distribution outlined previously. Specifically, we calculate the target number of samples for each object category by multiplying its frequency in the distribution by the total target size (10,000). The final dataset is then constructed by drawing the calculated number of QA pairs for each category from the initial 20,000-pair pool. This stratified sampling strategy ensures that the final evaluation set’s composition accurately mirrors the specified real-world object frequencies.
| Category | Fine-Grained Classes |
| --- | --- |
| Furniture | Bed, Chair, Sofa, Table, Nightstand, Cabinet, Shelf, Headboard, Wardrobe, Drawer, Wall, Door, Window, Mirror, Hanger, Hook, Handle, Hinge, Railing, Radiator, Light Switch |
| Appliances & Electronics | Outlet, Refrigerator, Washing Machine, Air Conditioner, Monitor, Television, Control Panel, Fan, Speaker, Lamp, Charger, Router, Cable, Oven, Toaster, Microwave, Water heater, Range Hoods, Remote Control |
| Kitchenware & Tableware | Spice Jar, Pot, Kettle, Cup, Jar, Bowl, Spoon, Knife, Plate, Chopping board, Chopstick, Stove, Rice Cooker |
| Containers | Bag, Box, Basket, Bucket, Bottle, Trash Can, Can, Lid, Ashtray |
| Bathroom & Cleaning | Faucet, Sink, Toilet, Toilet Seat, Toilet Lid, Shower, Bathtub, Mop, Broom, Brush, Sponge, Towel, Toothbrush, Toothpaste, Comb, Soap, Toilet Paper, Hose, Razor, Hair Dryer |
| Textiles & Bedding | Quilt, Blanket, Carpet, Curtain, Pillow, Cushion, Mattress |
| Stationery & Office Supplies | Book, Clock, Calendar, Pen, Sharpener, Scissors, Calculator, Mouse, Mousepad, Keyboard, LaptopPanel, Tablet Computer |
| Decor & Art | Plant, Painting, Picture, Poster, Label, Calendar, Vase |
| Daily Necessities | Phone, Hat, Slipper, Shoe, Umbrella, Headphones, Glove |
| Food | Fruit, Vegetable |
| Clothing | Shirt, Pants, Dress, Skirt, Coat, Shorts, Socks, Underwear |
| Fitness & Recreation | Treadmill, Dumbbells, Piano, Toy |
Table 6: Object category taxonomy.
C Qualitative examples
In section ˜ 4.2.1 and table ˜ 2, we show that our model can handle different types of object and spatial cognition tasks. In this section, we show more detailed qualitative examples for different abilities of our model.
C.1 Object Cognition
- Properties (figure ˜ 8, figure ˜ 9). The model discerns a wide range of object properties, including physical attributes such as size, color, and surface details, as well as functional affordances.
- Segmentation (figure ˜ 8). The system performs both simple and situational referring expression segmentation, enabling it to isolate target objects in the scene based on natural language queries.
C.2 Spatial Cognition
- Trajectory Review (figure ˜ 8). The model perceives the distance traversed by its own camera, allowing for a review of its past trajectory.
- Egocentric Direction (figure ˜ 9). It successfully determines the direction of objects relative to its own perspective.
- Egocentric Distance (figure ˜ 9). The system is capable of estimating the egocentric distance from itself to surrounding objects in the environment.
- Movement Imagery (figure ˜ 8). A key capability is the imagination of prospective movements, allowing the model to reason about future paths.
- Spatial Imagery (figure ˜ 8). The model demonstrates an ability for spatial imagination, such as inferring the layout of unseen areas.
- Object Size (figure ˜ 9). Its spatial understanding extends to estimating the absolute sizes of objects and performing relative size comparisons between them.
- Object Height (figure ˜ 8). Similarly, the model predicts and compares the heights of different objects.
- Object Distance (figure ˜ 9). The system accurately gauges the distance between multiple objects within the scene (i.e., inter-object distance).
- Absolute Position (figure ˜ 8). The model can ascertain the absolute positional relationships between objects.
- Relative Position (figure ˜ 9). Furthermore, it demonstrates a robust understanding of the relative positions of objects with respect to one another.
<details>
<summary>figures/append-qual/qual-all-1.png Details</summary>
### Visual Description
## Image Analysis: Object and Spatial Cognition Tasks
### Overview
The image presents a series of object and spatial cognition tasks, each consisting of a question (Q) and an answer (A) related to objects depicted in accompanying images. The tasks cover various dimensions of cognition, including color, category, shape, function, state, material, absolute position, object height, movement imagery, trajectory review, size, position, spatial imagery, and object segmentation. Each task is presented with a set of images showing the object or scene in question.
### Components/Axes
The image is organized into six rows and two columns, each containing a separate task. Each task includes the following components:
* **Dimension:** The type of cognitive task being assessed (e.g., "Object Cognition----Color").
* **Question (Q):** A question related to the object or scene.
* **Answer (A):** The answer to the question.
* **Images:** A set of images depicting the object or scene relevant to the question.
### Detailed Analysis or Content Details
Here's a breakdown of each task:
**Row 1, Column 1:**
* **Dimension:** Object Cognition----Color
* **Q:** What is the primary color of <object0>?
* **A:** The object is primarily light brown.
* **Images:** Three images of a small, light brown wooden table.
**Row 1, Column 2:**
* **Dimension:** Spatial Cognition----Spatial Imagery
* **Q:** Positioned at <object0> with your view directed towards <object1>, in which direction is <object2> situated?
* **A:** Right front.
* **Images:** Three images of a room, with a table and chairs.
**Row 2, Column 1:**
* **Dimension:** Object Cognition----Category
* **Q:** What category does <object0> belong to?
* **A:** The object is a piece of furniture, specifically a small wooden table or stand.
* **Images:** Three images of a small, light brown wooden table.
**Row 2, Column 2:**
* **Dimension:** Object Cognition----Function
* **Q:** What is the function of <object0>?
* **A:** The object is used for storage.
* **Images:** Three images of a wall-mounted cabinet.
**Row 3, Column 1:**
* **Dimension:** Object Cognition----Shape
* **Q:** What is the shape of <object0>?
* **A:** The object has a rectangular shape with a flat top and open sides.
* **Images:** Three images of a small, light brown wooden table.
**Row 3, Column 2:**
* **Dimension:** Object Cognition----State
* **Q:** What can be inferred about the state of the <object2>?
* **A:** The object appears to be in a stationary state, not currently in use.
* **Images:** Three images of a printer.
**Row 4, Column 1:**
* **Dimension:** Spatial Cognition----Absolute Position
* **Q:** Which one is above, <object0> or <object1>?
* **A:** <object0>.
* **Images:** Three images of a shelf above a trash bin.
**Row 4, Column 2:**
* **Dimension:** Object Cognition----Material
* **Q:** What material is <object1> likely made of?
* **A:** <object1> is likely made of plastic.
* **Images:** Three images of a trash bin.
**Row 5, Column 1:**
* **Dimension:** Spatial Cognition----Object Height
* **Q:** How much higher or lower is <object1> compared to <object0> above the ground?
* **A:** 1.03 meters.
* **Images:** Three images of a person standing in a room.
**Row 5, Column 2:**
* **Dimension:** Spatial Cognition----Trajectory Review
* **Q:** How far did you walk?
* **A:** 1.83 meters.
* **Images:** Three images of a room.
**Row 6, Column 1:**
* **Dimension:** Spatial Cognition----Movement Imagery
* **Q:** After you turn 90 degree to the left, where will <object1> be in relation to you?
* **A:** <object1> will situate at the 6 o'clock direction from me.
* **Images:** Three images of a room.
**Row 6, Column 2:**
* **Dimension:** Object Cognition---- Object Segmentation
* **Q:** If I want to drink water, which object should I look for?
* **A:**
* **Images:** Three images of a table with a water bottle.
**Row 7, Column 1:**
* **Dimension:** Object Cognition-Size
* **Q:** How does the size of the <object0> compare to the other items on the table?
* **A:** The object is relatively small compared to the other items on the table, such as the coffee machine and the trash bin.
* **Images:** Three images of a hand sanitizer.
**Row 7, Column 2:**
* **Dimension:** Object Cognition-Position
* **Q:** What is the position of <object0>?
* **A:** The object is mounted on the wall above the countertop.
* **Images:** Three images of a hand sanitizer.
### Key Observations
* The tasks cover a wide range of cognitive dimensions related to object recognition and spatial reasoning.
* The answers are generally straightforward and based on visual information present in the images.
* Some tasks involve relative comparisons (e.g., size, height, position).
* The use of placeholders like `<object0>`, `<object1>`, and `<object2>` suggests a standardized format for the questions.
### Interpretation
The image demonstrates a set of cognitive tasks designed to assess an AI's ability to understand and reason about objects and their spatial relationships. The tasks require the AI to:
* Identify objects.
* Categorize objects.
* Describe object properties (e.g., color, shape, material).
* Understand object functions.
* Reason about spatial relationships between objects.
* Infer object states.
* Estimate distances and trajectories.
The tasks could be used to evaluate the performance of computer vision models, robotics systems, or other AI agents that need to interact with the physical world. The standardized format of the questions allows for easy creation of new tasks and comparison of results across different systems.
</details>
Figure 8: Visualization of question answering examples. Part 1 out of 2.
<details>
<summary>figures/append-qual/qual-all-3.png Details</summary>
### Visual Description
## Image Analysis: Spatial and Object Cognition Scenarios
### Overview
The image presents a series of scenarios focusing on spatial and object cognition. Each scenario includes a set of images depicting a scene, followed by a question and answer related to spatial relationships, object properties, or object function. The scenarios cover aspects like distance estimation, object shape recognition, relative positioning, and object categorization.
### Components/Axes
Each scenario block contains the following components:
1. **Dimension Label**: Indicates the type of cognitive task being assessed (e.g., "Spatial Cognition----Egocentric Distance", "Object Cognition----Surface Detail").
2. **Images**: A set of images depicting the scene from different perspectives.
3. **Question (Q)**: A question related to the scene, probing spatial or object understanding.
4. **Answer (A)**: The correct answer to the question.
5. **Object Highlighting**: Some objects in the images are highlighted with colored outlines to indicate the object being referenced in the question.
### Detailed Analysis
**Scenario 1 (Top-Left)**
* **Dimension**: Spatial Cognition----Egocentric Distance
* **Question**: What is the distance between me and <object0>?
* **Answer**: 1.63m.
* **Object**: A computer monitor is highlighted in cyan.
* **Trend**: Measures the distance between the viewer and the monitor.
**Scenario 2 (Top-Right)**
* **Dimension**: Spatial Cognition----Object Distance
* **Question**: What is the distance between <object0> and <object1>?
* **Answer**: It is 1.23 meters
* **Object0**: A wooden dresser highlighted in green.
* **Object1**: A small bookshelf highlighted in purple.
* **Trend**: Measures the distance between two pieces of furniture.
**Scenario 3 (Middle-Left)**
* **Dimension**: Object Cognition----Surface Detail
* **Question**: What's the surface detail of <object0>?
* **Answer**: The surface of <object0> is smooth and reflective.
* **Object**: A computer monitor is highlighted in cyan.
* **Trend**: Identifies the surface properties of the monitor.
**Scenario 4 (Middle-Right)**
* **Dimension**: Spatial Cognition----Object Size
* **Question**: How tall is <object1>?
* **Answer**: It is 1.02 meters
* **Object1**: A small bookshelf highlighted in purple.
* **Trend**: Measures the height of the bookshelf.
**Scenario 5 (Second Row, Left)**
* **Dimension**: Spatial Cognition----Egocentric Direction
* **Question**: Is <object0> on your left front or right front in the last frame?
* **Answer**: Left front.
* **Object**: A computer monitor is highlighted in cyan.
* **Trend**: Determines the relative direction of the monitor from the viewer's perspective.
**Scenario 6 (Second Row, Right)**
* **Dimension**: Spatial Cognition----Relative Position
* **Question**: Is <object0> directly above <object1>?
* **Answer**: No, they are on the same height.
* **Object0**: A wooden dresser highlighted in green.
* **Object1**: A small bookshelf highlighted in purple.
* **Trend**: Assesses the vertical relationship between the dresser and bookshelf.
**Scenario 7 (Third Row, Left)**
* **Dimension**: Object Cognition----Shape
* **Question**: What is the shape of <object0>?
* **Answer**: The object has a classic teddy bear shape with a round head and body, and limbs.
* **Object**: A teddy bear is highlighted in yellow.
* **Trend**: Describes the shape of the teddy bear.
**Scenario 8 (Third Row, Right)**
* **Dimension**: Spatial Cognition----Relative Position
* **Question**: Is <object0> between <object1> and <object2>?
* **Answer**: No.
* **Object0**: A plant highlighted in blue.
* **Object1**: A calendar highlighted in yellow.
* **Object2**: A small green object.
* **Trend**: Determines the relative position of the plant with respect to the calendar and the small green object.
**Scenario 9 (Fourth Row, Left)**
* **Dimension**: Object Cognition----Size
* **Question**: What is the size of <object0> compare to the Ipad on the desk?
* **Answer**: The object is larger in size compared to the Ipad on the desk.
* **Object**: A teddy bear is highlighted in yellow.
* **Trend**: Compares the size of the teddy bear to an iPad.
**Scenario 10 (Fourth Row, Right)**
* **Dimension**: Spatial Cognition---- Egocentric Distance
* **Question**: Among <object0>, <object1>, and <object2> which one is nearer to you?
* **Answer**: <object0>.
* **Object0**: A plant highlighted in blue.
* **Object1**: A calendar highlighted in yellow.
* **Object2**: A small green object.
* **Trend**: Determines which object is closest to the viewer.
**Scenario 11 (Fifth Row, Left)**
* **Dimension**: Object Cognition----Size
* **Question**: What category does <object0> belong to?
* **Answer**: The object is a stuffed toy, specifically a teddy bear.
* **Object**: A teddy bear is highlighted in yellow.
* **Trend**: Categorizes the teddy bear.
**Scenario 12 (Fifth Row, Right)**
* **Dimension**: Spatial Cognition---- Egocentric Distance
* **Question**: Among <object0>, <object1>, and <object2> which one is nearer to you?
* **Answer**: <object0>.
* **Object0**: A plant highlighted in blue.
* **Object1**: A calendar highlighted in yellow.
* **Object2**: A small green object.
* **Trend**: Determines which object is closest to the viewer.
**Scenario 13 (Sixth Row, Left)**
* **Dimension**: Object Cognition---- Object Segmentation
* **Question**: If I want to check the current weather and time while sitting at the desk, where should I look?
* **Answer**: The answer is an image of the computer monitor.
* **Object**: The computer monitor.
* **Trend**: Identifies the location to find specific information.
**Scenario 14 (Sixth Row, Right)**
* **Dimension**: Object Cognition----Function
* **Question**: What is the function of <object0> ?
* **Answer**: The object provides water, which can be used for drinking or cooking.
* **Object**: A water container highlighted in yellow.
* **Trend**: Describes the function of the water container.
**Scenario 15 (Seventh Row, Left)**
* **Dimension**: Spatial Cognition---- Egocentric Distance
* **Question**: Among <object0>, <object1>, and <object2> which one is nearer to you?
* **Answer**: <object0>.
* **Object0**: A water container highlighted in yellow.
* **Object1**: A wooden cutting board.
* **Object2**: A green container.
* **Trend**: Determines which object is closest to the viewer.
### Key Observations
* The scenarios cover a range of cognitive tasks related to spatial and object understanding.
* The use of highlighted objects helps to focus attention on the relevant items in the scene.
* The questions vary in complexity, ranging from simple distance estimation to more complex relative positioning and object categorization.
### Interpretation
The image demonstrates a system designed to assess spatial and object cognition. The scenarios presented are designed to test a user's ability to understand spatial relationships, identify object properties, and categorize objects. The system could be used for training or evaluation purposes in fields such as robotics, computer vision, or cognitive science. The consistent format and clear questions and answers make the system easy to understand and use. The variety of scenarios ensures a comprehensive assessment of cognitive abilities.
</details>
Figure 9: Visualization of question answering examples. Part 2 out of 2.