# Grounding Video Reasoning in Physical Signals
## Abstract
Physical video understanding requires more than naming an event correctly. A model can answer a question about pouring, sliding, or collision from textual regularities while still failing to localize the event in time or space. We introduce a grounded benchmark for physical video understanding that extends the what–when–where evaluation structure of V-STaR [5] to four video sources, six physics domains, three prompt families (physics, vstar_like, and neutral_rstr), and four input conditions (original, shuffled, ablated, and frame-masked). The benchmark contains 1,560 base video clips from SSV2 [11], YouCook2 [32], HoloAssist [24], and Roundabout-TAU [16]. Each clip is first converted into a shared grounded event record, and the three query families are derived from that record. Temporal and spatial targets are shared across prompt families, while the non-physics families use deterministic family-appropriate semantic a_what targets derived from the same record. Across models and prompt families, physics remains the strongest regime overall, vstar_like is the clearest non-physics semantic comparison, and neutral_rstr behaves as a harder templated control. Prompt-family robustness is selective rather than universal, perturbation gains cluster in weak original cases, and spatial grounding is the weakest across settings. These results suggest that video Q $\&$ A reasoning benchmarks shall report physically grounded, prompt-aware, and perturbation-aware diagnostics alongside aggregate accuracy.
## 1 Introduction
In a video question answering task, a correct answer to a physical video question does not guarantee visually grounded understanding. A model can answer a question about pouring, sliding, or collision because the wording narrows the event type, even if it never identifies when the event happens or where the relevant objects are in a video. Standard answer-only benchmarks do not separate these cases. They tell us whether the final response matches the label, not whether the model uses video in a grounded way [4].
This matters most in physical scenes. Motion, contact, force, and state change unfold over time, and many of them are spatially localized. Recognising a clip contains a “collision” from the question text is different from locating the interaction in time and grounding the participating objects in space. In robotics, video moment retrieval, embodied decision-making, the first kind of success is not enough.
Existing benchmarks cover only part of this problem. Physical reasoning datasets such as CLEVRER [29], IntPhys [21], and PhysBench [6] probe physical understanding, but they mostly rely on categorical outputs. Grounded video benchmarks [31, 13, 27] evaluate temporal or spatial localisation, but they are not organized around physical event structure. V-STaR [5] took an important step by showing that strong performance on what can coexist with weak performance on when and where. That raises a useful diagnostic question for physical video understanding: when a model answers correctly, is it actually grounded in the video?
We study this question by introducing a physics-focused benchmark. Our benchmark keeps the grounded what–when–where structure of V-STaR, but moves it to six physics domains: Gravity, Fluids, Collisions, Deformation, Friction, and State Changes. Each sample is organized around a shared grounded event record with an event description, temporal span, and bounding box. The three prompt families— physics, vstar_like, and neutral_rstr —are alternative query formulations derived from that same record, and each is evaluated under four input conditions—original, shuffled, ablated, and frame-masked. The physics family is the main benchmark regime, vstar_like is a V-STaR-style semantic comparison rather than an exact reconstruction of the original benchmark prompts, and neutral_rstr is a neutral wording control that functions as a templated ablation.
The grounded event record stays fixed across prompt families, but the semantic field is expressed differently across them. Both a_when and a_where are scored against shared targets throughout. For neutral_rstr and vstar_like, a_what is scored against shorter family-appropriate semantic targets derived from the same record. Under this protocol, physics remains the strongest overall regime, vstar_like is the main non-physics comparison, and neutral_rstr remains the harder control. These shifts are not uniform across models. Some remain comparatively strong across formulations, while others depend more heavily on the cues made explicit by physics framing.
Aggregate scores still hide much of this behavior. Two models with similar overall performance can react very differently when the wording changes or the visual evidence is perturbed. Figure 1 previews the setup on one shared clip. The prompt families ask about the same event, but the ground-truth and model box overlays show that plausible answers can still hide very different spatial grounding.
<details>
<summary>2604.21873v1/Figures/fig_merged_prompt_model_qualitative.png Details</summary>
### Visual Description
## Screenshot: Shared Clip Qualitative Example
### Overview
The image displays a qualitative analysis of a video clip showing a human hand interacting with a stapler on a yellow surface. It includes:
- A sequence of 5 frames with timestamps (0.3s to 4.0s)
- Model prediction boxes overlaid on the clip
- Text-based question-answering sections
- Model performance comparisons
### Components/Axes
1. **Header**:
- Title: "Shared Clip Qualitative Example"
- File identifier: "SSV2 153091_original"
- Note: "GT boxes over the clip, model where-answers shown on the t = 1.0s frame"
2. **Main Content**:
- **Frames**:
- 5 sequential images showing hand/stapler interaction
- Timestamps: t = 0.3s, 1.1s, 1.9s, 2.8s, 4.0s
- Model prediction boxes (colored) overlaid on objects
- **Legend**:
- Position: Bottom center
- Color coding:
- Black: GT (Ground Truth)
- Green: Gemma 4
- Orange: Qwen2.5-VL
- Purple: Qwen3.5
3. **Text Sections**:
- **Left Panel**:
- Three question categories:
1. **Physics**:
- What: "How does the stapler transition from sliding to falling?"
- When: "When does the stapler lose support and transition from sliding to to..."
- Where: "Where is the stapler during the sliding-and-falling event?"
2. **V-STaR-like**:
- What: "What does the stapler do?"
- When: "When is the moment 'the stapler slides off the edge and falls'..."
- Where: "Where is the stapler during the event of sliding off the edge and..."
3. **Neutral-RSTR**:
- What: "What happens to the stapler after it starts to slide?"
- When: "When does the stapler begin sliding off the edge?"
- Where: "Where is the stapler during the relevant time span?"
- **Right Panel**:
- **Physics Answers**:
- Prompt: "How does the stapler transition from sliding to falling?"
- Ground Truth (GT): "The stapler slides across the surface with deceleration due to friction until its center of mass passes the edge, causing a..."
- Gemma 4: "The stapler transitions from sliding to falling when its center of mass moves past the edge of the surface, losing the..."
- Qwen2.5-VL: "The stapler transitions from sliding to falling."
- Qwen3.5: "The stapler is pushed and slides off the edge of the chair."
### Detailed Analysis
1. **Frame Analysis**:
- At t=0.3s: Stapler initially placed on surface
- t=1.1s: Multiple model boxes (GT, Gemma 4, Qwen2.5-VL, Qwen3.5) show varying predictions
- t=1.9s: GT box maintains accurate position
- t=2.8s: Model boxes diverge in position prediction
- t=4.0s: Stapler partially off surface; models show different interpretations
2. **Model Performance**:
- GT (Ground Truth) consistently most accurate
- Gemma 4 provides detailed physical explanation
- Qwen2.5-VL gives concise answer
- Qwen3.5 introduces positional inaccuracy ("chair" instead of "surface")
3. **Question-Answer Structure**:
- All question categories follow "What/When/Where" format
- Physics questions focus on mechanics of motion
- V-STaR-like emphasizes event description
- Neutral-RSTR focuses on object state changes
### Key Observations
1. Model predictions show increasing divergence over time
2. GT maintains consistent accuracy across frames
3. Qwen3.5 introduces positional error in final frame
4. Answer quality correlates with box prediction accuracy
5. V-STaR-like questions require more descriptive answers
### Interpretation
This visualization demonstrates:
1. **Temporal Tracking Performance**: Models struggle with long-term object tracking, particularly during motion transitions
2. **Physics Understanding**: GT and Gemma 4 show better grasp of physical principles (friction, center of mass)
3. **Answer Quality Gradient**: More accurate predictions correlate with more detailed/accurate answers
4. **Model Specialization**: Qwen3.5's "chair" error suggests potential domain confusion
5. **Evaluation Framework**: The structured question categories (Physics/V-STaR/Neutral) provide multidimensional assessment of model capabilities
The data suggests that while all models can identify basic object interactions, only GT and Gemma 4 demonstrate nuanced understanding of physical principles governing the stapler's motion. The increasing prediction errors over time highlight challenges in maintaining object tracking during dynamic events.
</details>
Figure 1: One shared clip across prompt families and grounded outputs. Top: matched SSV2 frames with the ground-truth box trajectory and representative model a_where predictions. Bottom left: the three query families derived from the same event record. Bottom right: representative physics answers from the ground truth and three models.
#### Research questions.
We organize this study around four questions. RQ1 asks whether the grounded failure pattern identified by V-STaR persists in physical video understanding. RQ2 asks how stable grounded performance remains across physics, vstar_like, and neutral_rstr when the underlying event record is fixed. RQ3 asks what perturbation gains and losses reveal beyond original-condition performance. RQ4 asks which weaknesses remain even for strong models and favorable prompt families.
We keep the hypotheses narrow. We expect the V-STaR failure pattern to persist in physics-focused video, with spatial grounding remaining the weakest component. We also expect physics to be strongest overall, vstar_like to be the strongest non-physics comparison, and neutral_rstr to behave as a harder templated control. Finally, we expect perturbation gains to concentrate in weak or mid-baseline cases and to function as diagnostics of evidence sensitivity rather than as simple robustness wins.
Our main contributions are:
- A grounded benchmark for physical video understanding, built from 1,560 base clips from SSV2 [11], YouCook2 [32], HoloAssist [24], and Roundabout-TAU [16], and organized into six physics domains.
- A three-prompt evaluation design built on the same shared event record, separating the main physics regime from a V-STaR-style prompt family (vstar_like) and a neutral wording control (neutral_rstr).
- A perturbation analysis framework that combines shuffled, ablated, and frame-masked inputs with component-wise metrics and diagnostic indices to interpret changes under degraded evidence.
- Empirical evidence that prompt-family robustness is selective across models, perturbation gains concentrate in weaker original cases, and spatial grounding remains the most persistent weakness across model families.
## 2 Related Work
### 2.1 Video-LLM evaluation
Recent video-LLMs combine a visual encoder with a pretrained language model and instruction tuning over mixed-modality datasets [18, 15]. Systems such as Qwen2.5-VL [3], Qwen3-VL [2], VideoLLaMA3 [30], InternVideo2.5 [25], InternVL3.5 [23], and MiniCPM-o [28] perform well on standard benchmarks including Video-MME [8], MVBench [14], and LongVideoBench [26]. Those benchmarks are useful for broad coverage, but they score the final answer rather than the grounding process. A correct answer does not tell us whether the model located the relevant event in time and space or whether it relied on prompt cues and dataset regularities.
### 2.2 Physical reasoning benchmarks
Physical reasoning has been studied in synthetic and real-world settings for a long time. IntPhys [21] evaluates physical plausibility, while CLEVRER [29] probes causal and counterfactual reasoning in a controlled collision world. More recent benchmarks such as PhysBench [6] evaluate multimodal LLMs on physical concepts including gravity, collision, and material behavior. The common limitation is the answer format. Most of these benchmarks use categorical or multiple-choice outputs, so a model can score well without grounding the event itself in time or space.
### 2.3 Grounded video understanding
Grounded video understanding is usually studied through temporal grounding, spatio-temporal grounding, and grounded question answering. TALL and Charades-STA formalized temporal moment localization in untrimmed video [9], while ActivityNet Captions extended that setting to dense event description [12]. VidSTG [31] made spatio-temporal grounding a standard evaluation problem, and TVQA+ [13] and NExT-GQA [27] added temporal and spatial annotations to video Q $\&$ A. These datasets make grounding visible, but they are not designed around physical event structure or prompt-sensitive diagnosis.
### 2.4 Diagnostic evaluation and V-STaR
Several recent works argue that strong aggregate scores can conceal brittle behavior. Buch et al. [4] showed that some video-language benchmarks can be solved from linguistic structure alone. Bagad et al. [1] found that leading video-language models remain weak at chronological reasoning when frame order is manipulated. In other modalities, Winoground [22] and CheckList [20] showed the value of controlled diagnostic testing instead of relying only on headline benchmark scores.
V-STaR [5] is the closest prior benchmark to this study. It introduced a grounded what–when–where evaluation structure and showed that video-LLMs often perform much better on what than on when or where. Our benchmark uses the same diagnostic logic. Against physical reasoning benchmarks, we add grounded outputs. Against standard grounding benchmarks, we add a physics-domain layer and controlled perturbations. We also include a V-STaR-style prompt family explicitly, not leaving that comparison implicit.
## 3 Benchmark design
### 3.1 Task format and prompt families
Our benchmark inherits the grounded what–when–where evaluation structure introduced by V-STaR [5]. Given a video clip and a physical question, the model must produce a single structured prediction with three fields:
- a_what: a short text description of the physical event,
- a_when: a temporal interval [start_sec, end_sec],
- a_where: a normalised bounding box x, y, w, and h.
Requiring all three parts in one response is deliberate. Recognition, temporal grounding, and spatial grounding can each be gamed differently if they are evaluated in isolation. By forcing the model to commit to one coherent account of the event, we make it harder to score well by naming the event while ignoring where it happened or when it started.
We evaluate the same grounded event record under three prompt families:
- physics: the main benchmark regime, written to foreground physical dynamics and event descriptions.
- vstar_like: questions written in the style of the original V-STaR prompts using the existing annotations only. It is a continuity control, not an exact reconstruction of original V-STaR semantics.
- neutral_rstr: a neutral wording control that preserves the same grounded event and output schema while removing physics-specific phrasing. It is best read as a templated ablation rather than as the paper’s main semantic baseline.
The prompt families are not separate annotation pipelines. All three are derived from the same grounded event record, which fixes the event identity, temporal span, and spatial reference. For cross-family evaluation, a_when and a_where are therefore scored against the same targets in every family. The text field works slightly differently. The physics family uses the reference event description directly, while neutral_rstr and vstar_like use shorter family-appropriate semantic a_what targets derived from that same record. This keeps the semantic target close to the answer style requested by the non-physics prompts while preserving the shared grounded event.
Figure 1 shows an example of this shared-record design. The wording changes across prompt families, but the grounded event, temporal span, and spatial reference do not.
### 3.2 Perturbation conditions
Each base clip is evaluated under four input conditions:
- Original: the unmodified RGB video.
- Shuffled: frames are randomly permuted while the set of frames is kept fixed.
- Ablated: each frame is converted to greyscale and blurred to suppress color and fine texture while preserving coarse spatial structure.
- Frame-Masked: half of the frames are replaced by black frames while video length and frame rate remain unchanged.
Each condition targets a different component of the visual signal. Shuffling removes temporal order while preserving the frame set. Ablation removes color and fine appearance detail while preserving coarse structure. Frame masking removes evidence intermittently while preserving the temporal structure. Together, these conditions let us ask whether a model depends on temporal order, appearance detail, or persistent visual evidence, instead of collapsing those effects into one original-input score.
Table 1 summarizes what each condition is meant to probe, and Figure 2 shows the same clip under all four variants. The ablated row is especially useful because object extent remains visible while fine appearance cues are suppressed.
| Original | unmodified video | baseline performance |
| --- | --- | --- |
| Shuffled | frame order permuted | temporal order sensitivity |
| Ablated | greyscale + blur | appearance dependence |
| Frame-Masked | 50% frames replaced by black | robustness to missing evidence |
Table 1: The four evaluation conditions and the diagnostic role each is intended to probe. The grounded target is held fixed across conditions.
<details>
<summary>2604.21873v1/Figures/fig1_conditions.png Details</summary>
### Visual Description
## Screenshot: YouCook2 Dataset Visualization
### Overview
The image displays a sequence of six frames from a video clip (ID: `316_FiE1KczH4pc`) in the YouCook2 dataset, annotated with timestamps (19.6s to 30.6s) and labeled with processing methods. The central object is a blender, and the frames are organized into four rows representing different processing techniques: **Original**, **Shuffled**, **Ablated**, and **Frame-Masked**. Yellow vertical lines indicate the ground-truth bounding box (bbox) for the blender.
### Components/Axes
- **Top Caption**:
- Text: `"YouCook2 clip: 316_FiE1KczH4pc | object: blender"`
- Position: Center-top of the image.
- **Timestamps**:
- Values: `19.6s`, `20.4s`, `21.3s`, `22.2s`, `23.1s`, `30.6s`
- Position: Top of each column, aligned with corresponding frames.
- **Row Labels (Left Side)**:
- **Original**: Blue box with white text.
- **Shuffled**: Orange box with white text.
- **Ablated**: Black box with white text.
- **Frame-Masked**: Black box with white text.
- **Legend**:
- Yellow lines labeled `"yellow = ground-truth bbox"`
- Position: Right side of the image, spanning all rows.
### Detailed Analysis
- **Original Row**:
- Color: Full-color images.
- Content: A person in a blue tank top interacting with a white blender on a kitchen countertop. The blender’s transparent container holds ingredients.
- Ground-truth bbox: Yellow vertical lines align with the blender’s position in each frame.
- **Shuffled Row**:
- Color: Grayscale images.
- Content: Same scene as Original, but with pixelated/shuffled regions outside the blender’s bbox.
- **Ablated Row**:
- Color: Grayscale images.
- Content: Blender and person visible, but background elements (e.g., cabinets, countertop) are blurred or removed.
- **Frame-Masked Row**:
- Color: Black frames for all timestamps except `22.2s` and `23.1s`, where partial blender visibility remains.
- Content: Most frames are entirely black, suggesting object masking.
### Key Observations
1. **Temporal Consistency**: The blender’s position (yellow bbox) remains consistent across Original, Shuffled, and Ablated rows, indicating stable object detection.
2. **Processing Effects**:
- Shuffled: Artifacts outside the bbox suggest data augmentation or noise introduction.
- Ablated: Background simplification implies feature isolation for object-focused analysis.
- Frame-Masked: Selective masking (e.g., `22.2s`, `23.1s`) may test robustness to partial occlusions.
3. **Temporal Gaps**: The Frame-Masked row has missing frames (`19.6s`, `20.4s`, `30.6s`), possibly due to processing limitations.
### Interpretation
This visualization demonstrates how different preprocessing methods affect video analysis for object detection tasks. The **Original** row serves as the ground truth, while **Shuffled** and **Ablated** rows test model resilience to noise and background complexity. The **Frame-Masked** row evaluates handling of occlusions or missing data. The ground-truth bbox (yellow lines) provides a reference for evaluating detection accuracy. Notably, the Frame-Masked row’s incomplete frames suggest potential challenges in real-time processing or edge-case handling.
The dataset’s structure highlights the importance of robust object tracking across varying temporal and spatial conditions, critical for applications like automated cooking tutorials or robotic assistance.
</details>
Figure 2: The four input conditions applied to the same YouCook2 clip at matched timestamps. Yellow boxes show the ground-truth object reference across rows. Shuffling preserves the frame set while breaking temporal order; ablation suppresses color and fine texture while preserving coarse object extent; frame masking removes evidence intermittently.
### 3.3 Metrics and reporting conventions
We score the three output fields with grounded metrics. Let $Acc$ be text accuracy for a_what, $tIoU$ be temporal intersection-over-union for a_when, and $sIoU$ be spatial intersection-over-union for a_where. Following V-STaR [5], we combine these three components with the Logarithmic Geometric Mean:
$$
\displaystyleLGM=-\frac{1}{3}\Big[ \displaystyle\log(1-Acc+ε) \displaystyle+\log(1-tIoU+ε) \displaystyle+\log(1-sIoU+ε)\Big] \tag{1}
$$
where $ε$ is a small constant for numerical stability.
LGM is high only when all three components are strong. A model that achieves good semantic accuracy while failing at temporal or spatial localisation will still score poorly. In this setting, that matters because strong a_what performance would otherwise hide weak a_when or a_where predictions.
There is one reporting detail worth stating explicitly. V-STaR applies an additional linear readability scaling to LGM [5]. Our pipeline reports LGM directly on the normalized component scale. This changes the displayed magnitude, but not the ranking or the perturbation ratios.
### 3.4 Diagnostic indices
We derive three indices from the four condition scores:
$$
\displaystyleSBI \displaystyle=1-(LGM_orig-LGM_shuf), \displaystylePRI \displaystyle=\frac{LGM_abl}{LGM_orig}, \displaystyleSPI \displaystyle=\frac{LGM_mask}{LGM_orig}. \tag{2}
$$
PRI and SPI are retention ratios relative to the original condition, so a value of $1$ means that the perturbed input matches the original-condition score. SBI is written on the same centered scale, with $1$ again meaning no change under temporal shuffling. Larger values therefore indicate greater robustness to the corresponding perturbation, while smaller values indicate loss under the perturbed input. We use these indices as descriptive diagnostics rather than standalone evidence: they are interpreted together with component deltas, baseline strata, validity transitions, and qualitative examples.
### 3.5 Validation-aware scoring
Model outputs are sometimes empty, malformed, or refusal-like. The evaluation pipeline counts those cases in the denominator and scores them as zero rather than silently dropping them. This matters for interpretation. If failed outputs were excluded, perturbation deltas would look artificially favorable for models that simply stop producing valid answers under harder settings.
## 4 Dataset construction
### 4.1 Sources and scale
The benchmark is constructed from four video sources that differ in viewpoint, event type, and annotation style: SSV2 [11], YouCook2 [32], HoloAssist [24], and Roundabout-TAU [16]. In total, the benchmark contains 1,560 base clips. Each base clip is expanded into four perturbation conditions, producing 6,240 scored video-condition pairs.
Table 2 shows the source breakdown. SSV2 and YouCook2 provide the largest portions of the data. HoloAssist contributes egocentric manipulation scenes in which hands and objects overlap under camera motion. Roundabout-TAU contributes overhead traffic footage where appearance cues are weak and motion structure matters more than texture.
| SSV2 YouCook2 HoloAssist | 600 600 275 |
| --- | --- |
| Roundabout-TAU | 85 |
| Total | 1,560 |
Table 2: Source breakdown. Counts are for base clips before perturbation expansion.
### 4.2 Physics domains
Every clip is assigned to one of six physics domains: Gravity, Fluids, Collisions, Deformation, Friction, and State Changes. The domain layer matters because it lets us ask not only whether a model fails, but where it fails. A model may perform well on State Changes because those events are temporally salient and semantically distinctive, yet still struggle on Friction or Collisions where localization is harder and the critical evidence can be brief or spatially small.
Table 3 shows the domain distribution across base clips. No domain is severely underrepresented, so per-domain analysis remains meaningful.
| Gravity Fluids Collisions | 248 283 296 |
| --- | --- |
| Deformation | 248 |
| Friction | 196 |
| State Changes | 289 |
| Total | 1,560 |
Table 3: Domain breakdown for the benchmark base clips.
### 4.3 Annotation pipeline
Each base clip is converted into a common grounded record containing a reference event description, a temporal span, a bounding box, and a physics-domain label. The key design choice is that this record is prompt-family-agnostic. It defines which event should be grounded, when it happens, and where it occurs, before any particular query wording is chosen. Text annotations are produced with a local Qwen3.5-based generator [19], which rewrites source metadata and event windows into that shared event description. Temporal spans come from the source adapters and event windows rather than from a second prompt-generation stage.
Spatial annotations are generated on the original clip with GroundingDINO [17] and then reused across all four perturbation conditions. This keeps the target object reference fixed when the input is shuffled, ablated, or frame-masked. The prompt families are layered on top of the same grounded record. They do not regenerate temporal spans or spatial boxes. The semantic text target is handled more carefully: physics keeps the longer reference event description, while neutral_rstr and vstar_like derive shorter prompt-aligned semantic a_what targets from the same grounded record.
### 4.4 Source adaptation and Roundabout selection
The four sources are not simply merged. Each one is adapted into the same grounded annotation format and mapped into the six-domain taxonomy. SSV2 contributes short object-manipulation clips in which temporal order is often decisive. YouCook2 contributes longer procedural clips with extended state changes and fluid events. HoloAssist contributes first-person manipulation in which object visibility and grounding are harder because the camera moves with the actor.
Roundabout-TAU requires an additional filtering step. Much of the raw traffic footage shows ordinary circulation rather than a localized physical interaction. We therefore map the source event labels into our physics taxonomy and exclude normal traffic by default. Clips are retained only when the event can be grounded as a localized interaction, such as collision-like behavior or a physically meaningful maneuver conflict. This is also how we justify using Roundabout for the Collisions domain: we do not treat every traffic clip as a collision example, only the subset whose event annotation corresponds to an interaction that can be temporally and spatially grounded.
### 4.5 Annotations and perturbation reuse
Each sample stores a reference event description, a temporal interval, a spatial annotation, and a domain label, and the query families are rendered from that shared record. The temporal interval and bounding box are used directly to score a_when and a_where. The text field serves two roles: it is the base event description for the physics family, and it is the source from which shorter deterministic semantic a_what targets are derived for neutral_rstr and vstar_like. Bounding boxes are generated on the original clip and then reused across all perturbation conditions. The same underlying event is therefore evaluated across original, shuffled, ablated, and frame-masked inputs.
The query families are rendered automatically from the shared record rather than rewritten manually for each sample. That keeps supervision aligned across prompt families, but it also means that some individual prompts are terser or less natural than others. For that reason, we interpret cross-family results primarily through aggregate behavior over shared targets rather than through any single prompt instance.
This reuse matters for interpretation. Changes across prompt families or perturbation conditions reflect changes in question formulation, visual evidence, and answer-style alignment rather than a separate manual annotation pipeline for each family. The benchmark is designed so that a score shift can be traced back to model behavior without arbitrary drift in the underlying grounded event.
## 5 Experimental Setup
### 5.1 Models
Table 4 lists the ten-model suite, spanning general-purpose VLMs, video-native models, a compact multimodal model, and recent open multimodal LLMs.
| VideoLLaMA3-7B Qwen3-VL-8B-Instruct Molmo2-8B | 7B 8B 8B | video-centric model general-purpose VLM multimodal LLM |
| --- | --- | --- |
| Qwen2.5-VL-7B-Instruct | 7B | general-purpose VLM |
| Gemma4-26B-A4B-IT | 26B | multimodal LLM |
| MiniCPM-o 2.6 | 2.6B | compact multimodal model |
| Qwen3-VL-8B-Thinking | 8B | thinking-variant VLM |
| InternVideo2.5-Chat-8B | 8B | video-native model |
| InternVL3.5-8B | 8B | image-centric VLM |
| Qwen3.5-9B | 9B | native multimodal LLM |
Table 4: Ten-model suite spanning general-purpose VLMs, video-native models, compact multimodal models, and open multimodal LLMs.
We evaluate Qwen2.5-VL [3], Qwen3-VL and Qwen3-VL-Thinking [2], VideoLLaMA3 [30], InternVideo2.5 [25], InternVL3.5 [23], MiniCPM-o [28], Qwen3.5 [19], Gemma 4 [10], and Molmo2 [7]. The suite gives us useful contrasts: two Qwen generations, two video-native models, one image-centric VLM, a compact multimodal model, and several recent open multimodal LLMs.
### 5.2 Inference setup
All models return one JSON object containing a_what, a_when, and a_where. We evaluate all four perturbation conditions, and cross-family comparisons use the full completed model set. Frame budgets follow native presets rather than a single shared budget because the benchmark is meant to measure model behavior in its normal operating mode. The most informative comparisons are therefore within-model changes across prompt families and perturbations, not perfectly matched frame counts across architectures.
### 5.3 Evaluation and analysis protocol
For each sample, the prediction is compared to the reference annotation using text accuracy, temporal IoU, spatial IoU, and LGM. Condition-level means are computed first, and SBI, PRI, and SPI are derived from those means. Missing, malformed, or refusal-like outputs are scored as zero and kept in the denominator. Otherwise the benchmark would overstate models that simply stop returning usable answers under harder conditions.
The main original-condition physics comparison uses all ten integrated models. Prompt-family comparisons use the same suite, with shared a_when and a_where targets and prompt-aligned non-physics a_what targets derived from the same event record.
We interpret perturbation behavior with analyses beyond the aggregate indices. For selected models and prompt families, we use per-sample $Δ$ LGM, bootstrap confidence intervals, sign tests, baseline-stratified summaries, leave-one-dataset-out checks, and qualitative examples to interpret gains and losses.
## 6 Results
### 6.1 Main physics benchmark
Table 5 reports original-condition grounded metrics in the physics prompt family together with the three perturbation indices.
| VideoLLaMA3 Qwen3-VL Molmo2 | 0.319 0.687 0.548 | 0.547 0.183 0.322 | 0.042 0.056 0.023 | 2.634 2.551 2.515 | 0.959 1.076 1.326 | 0.926 1.231 1.087 | 1.032 1.337 0.974 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2.5-VL | 0.378 | 0.560 | 0.073 | 2.399 | 1.076 | 1.073 | 1.091 |
| Gemma4 | 0.481 | 0.455 | 0.034 | 2.150 | 1.173 | 1.047 | 0.862 |
| MiniCPM-o 2.6 | 0.232 | 0.373 | 0.033 | 1.396 | 0.947 | 0.946 | 0.706 |
| Qwen3-VL-Thinking | 0.635 | 0.126 | 0.051 | 1.332 | 1.106 | 1.304 | 1.132 |
| InternVideo2.5 | 0.189 | 0.285 | 0.027 | 1.223 | 1.290 | 1.064 | 0.705 |
| InternVL3.5 | 0.393 | 0.263 | 0.036 | 0.828 | 1.302 | 1.058 | 1.001 |
| Qwen3.5 | 0.310 | 0.323 | 0.037 | 0.766 | 1.309 | 1.093 | 0.965 |
Table 5: Physics-prompt results averaged over the four sources. Acc, tIoU, sIoU, and LGM are original-condition scores; SBI, PRI, and SPI summarize perturbation response. Values above $1$ mean the perturbed condition outscored the original. Models are ordered by LGM values.
Table 5 shows the same failure pattern that motivated V-STaR: strong semantic accuracy does not guarantee equally strong temporal or spatial grounding. Qwen3-VL leads Acc, Qwen2.5-VL leads tIoU and sIoU, and VideoLLaMA3 leads LGM. These models arrive there in different ways: VideoLLaMA3 is strongest temporally, Qwen3-VL is strongest semantically, and Qwen2.5-VL is the most balanced on temporal and spatial localization. Molmo2 joins this top group with a distinct perturbation profile. Spatial grounding remains the clearest weakness, with no model exceeding 0.073 mean sIoU.
### 6.2 Prompt-family changes and V-STaR-style prompting
Table 6 reports original-condition LGM across prompt families on the full ten-model set.
| VideoLLaMA3 Qwen3-VL Molmo2 | 2.634 2.551 2.515 | 2.171 0.116 1.384 | 1.181 0.114 0.866 |
| --- | --- | --- | --- |
| Qwen2.5-VL | 2.399 | 1.058 | 0.750 |
| Gemma4 | 2.150 | 0.945 | 1.315 |
| MiniCPM-o 2.6 | 1.396 | 1.824 | 0.515 |
| Qwen3-VL-Thinking | 1.332 | 0.242 | 0.102 |
| InternVideo2.5 | 1.223 | 0.839 | 0.304 |
| InternVL3.5 | 0.828 | 0.448 | 0.297 |
| Qwen3.5 | 0.766 | 0.524 | 0.214 |
| Mean over 10-model set | 1.779 | 1.139 | 0.547 |
Table 6: Original-condition LGM across prompt families on the full ten-model set. Non-physics rows use prompt-aligned semantic a_what; a_when and a_where remain shared across families. Models are ordered by physics LGM values.
Table 6 and Figure 1 show the same ordering: physics is strongest overall, vstar_like sits between it and neutral_rstr, and mean LGM falls from 1.779 to 1.139 to 0.547, making vstar_like the main non-physics comparison.
Cross-family shifts are model-specific rather than uniform. VideoLLaMA3 remains strong under vstar_like, and MiniCPM-o 2.6 is the clearest positive case: its vstar_like score exceeds its physics score. Molmo2 also recovers meaningfully under vstar_like, while Gemma4 is the cleanest neutral_rstr case. The strongest negative cases are Qwen3-VL and Qwen3-VL-Thinking, which both drop sharply outside physics. Qwen3.5 follows the same pattern at a lower level. High performance under physics-framed queries therefore does not necessarily transfer to alternative semantic formulations.
### 6.3 Interpreting perturbation indices
By construction, PRI and SPI compare ablated and masked performance to the original condition, and SBI compares shuffled performance to the original on the same centered scale. Values above $1$ mean that the perturbed input outscored the original, but they should be read as diagnostics rather than blanket robustness. Across the analyses, low- and mid-baseline rows are more likely to improve than high-baseline rows.
| Qwen3-VL / physics | ablated | +0.589 | high-baseline gain |
| --- | --- | --- | --- |
| Molmo2 / physics | shuffled | +0.326 | mid-baseline temporal gain |
| Gemma4 / neutral_rstr | shuffled | +0.347 | control-family gain |
| InternVL3.5 / physics | shuffled | +0.302 | low-baseline gain |
Table 7: Representative positive-response cases. Positive $Δ$ LGM is descriptive, not a general robustness claim.
Figure 2 and Table 7 show why the sign alone is not enough under physics. The same sign can arise in different regimes, so these cases should not be read as one uniform notion of robustness. Qwen3-VL and VideoLLaMA3 make the contrast clear: their original-condition LGM values are close, but Qwen3-VL often improves under perturbation whereas VideoLLaMA3 is flatter or negative on shuffled and ablated inputs.
### 6.4 Domain-level findings
Domain leaders vary by domain: Qwen3-VL leads Gravity and Friction, VideoLLaMA3 leads Fluids, and Qwen2.5-VL leads Collisions, Deformation, and State Changes. There is no winner-take-all leader. Domain-level LGM changes across models, the underlying limitation does not: spatial grounding remains weak even when a model leads a domain.
## 7 Limitations
Automatic supervision. Event descriptions, temporal spans, and spatial boxes are produced automatically rather than fully verified by human annotators. That gives the benchmark scale, but it also introduces noise. The problem is most likely in egocentric and traffic video, where the grounded target can be ambiguous even for a human reader. Human spot-checking of hard cases would improve confidence in the labels.
Annotation-style bias. The text annotations are generated with an LLM-based stage, and one evaluated model family is closely related to that generator. We therefore use shorter non-physics semantic a_what targets derived from the shared event record instead of reusing the longer physics-style answers verbatim. That keeps cross-family semantics closer to the answer style requested by the prompts, but the non-physics targets remain heuristic deterministic derivations rather than a manually curated multi-family gold set. Large cross-family drops should therefore not be over-attributed to any single cause: instruction-following preferences, answer-style priors, and deeper failures in grounded video reasoning can all contribute.
Source imbalance. The benchmark is reasonably balanced at the domain level, but not at the source level. Roundabout-TAU contributes only 85 clips, compared with 600 each from SSV2 and YouCook2, so its source-level findings should be read cautiously.
Scope of the task. The benchmark evaluates grounded event understanding: what happened, when it happened, and where it happened. It does not directly test causal forecasting, counterfactual physics, or long-horizon planning. A model that localises a collision correctly can still fail to predict what follows or to reason about alternative physical outcomes.
Diagnostic, not causal, interpretation. Perturbations expose behavior that original-condition scores hide, but they do not identify a single underlying cause. A gain under ablation can reflect reduced distractors, weak original baselines, or a cleaner temporal guess, so we treat the perturbation analysis as a diagnostic framework rather than a causal explanation of how each model reasons internally.
## 8 Conclusion
We introduced a grounded benchmark for physical video understanding that extends the what–when–where diagnostic idea of V-STaR [5] to physics-focused video, multiple prompt families, and controlled perturbations built on a shared grounded event record. The experiments support three conclusions: physics is strongest overall, vstar_like is the strongest non-physics comparison, neutral_rstr is better read as a harder control, and prompt-family robustness is selective rather than universal. Spatial grounding remains the weakest component, so answer accuracy alone still hides too much behavior. A model can still recover the event label while failing to ground it in time or space, and prompt-family changes and input perturbations make that failure easier to see.
## References
- [1] P. Bagad, M. Tapaswi, and C. G. M. Snoek (2023) Test of time: instilling video-language models with a sense of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2503–2516. Cited by: §2.4.
- [2] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §2.1, §5.1.
- [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Cited by: §2.1, §5.1.
- [4] S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, L. Fei-Fei, and J. C. Niebles (2022) Revisiting the “video” in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2907–2917. Cited by: §1, §2.4.
- [5] Z. Cheng, J. Hu, Z. Liu, C. Si, W. Li, and S. Gong (2025) V-star: benchmarking video-llms on video spatio-temporal reasoning. arXiv preprint arXiv:2503.11495. External Links: 2503.11495 Cited by: §1, §2.4, §3.1, §3.3, §3.3, §8.
- [6] W. Chow, J. Mao, B. Li, D. Seita, V. Guizilini, and Y. Wang (2025) PhysBench: benchmarking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501.16411. Cited by: §1, §2.2.
- [7] C. Clark, J. Zhang, Z. Ma, J. S. Park, M. Salehi, R. Tripathi, S. Lee, Z. Ren, C. D. Kim, Y. Yang, V. Shao, Y. Yang, W. Huang, Z. Gao, T. Anderson, J. Zhang, J. Jain, G. Stoica, W. Han, A. Farhadi, and R. Krishna (2026) Molmo2: open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611. External Links: 2601.10611 Cited by: §5.1.
- [8] C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, R. Ji, and X. Sun (2025) Video-MME: the first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24108–24118. Cited by: §2.1.
- [9] J. Gao, C. Sun, Z. Yang, and R. Nevatia (2017) TALL: temporal activity localization via language query. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5277–5285. Cited by: §2.3.
- [10] Google DeepMind (2026) Gemma-4-26b-a4b-it. Hugging Face. Note: https://huggingface.co/google/gemma-4-26B-A4B-it Cited by: §5.1.
- [11] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic (2017) The “something something” video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5842–5850. Cited by: 1st item, §4.1.
- [12] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles (2017) Dense-captioning events in videos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 706–715. Cited by: §2.3.
- [13] J. Lei, L. Yu, T. L. Berg, and M. Bansal (2020) TVQA+: spatio-temporal grounding for video question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8211–8225. Cited by: §1, §2.3.
- [14] K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, L. Wang, and Y. Qiao (2024) MVBench: a comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22195–22206. Cited by: §2.1.
- [15] B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan (2023) Video-LLaVA: learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. Cited by: §2.1.
- [16] Y. Lin, K. Chen, S. Lockyer, A. Yadav, M. Sui, S. Zhang, Y. Shi, B. Wang, Y. Zhang, M. Zarbock, F. Stanek, A. Evans, W. Li, Y. Wang, and N. Zhang (2026) TAU-R1: visual language model for traffic anomaly understanding. arXiv preprint arXiv:2603.19098. Cited by: 1st item, §4.1.
- [17] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang (2023) Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499. Cited by: §4.3.
- [18] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan (2024) Video-ChatGPT: towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12585–12602. Cited by: §2.1.
- [19] Qwen Team (2026) Qwen3.5-9b-base. Hugging Face. Note: https://huggingface.co/Qwen/Qwen3.5-9B-Base Cited by: §4.3, §5.1.
- [20] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh (2020) Beyond accuracy: behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4902–4912. Cited by: §2.4.
- [21] R. Riochet, M. Ynocente Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux (2021) IntPhys: a framework and benchmark for visual intuitive physics reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). Cited by: §1, §2.2.
- [22] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross (2022) Winoground: probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5238–5248. Cited by: §2.4.
- [23] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. (2025) InternVL3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Cited by: §2.1, §5.1.
- [24] X. Wang, T. Kwon, M. Rad, B. Pan, I. Chakraborty, S. Andrist, D. Bohus, A. Feniello, B. Tekin, F. V. Frujeri, N. Joshi, and M. Pollefeys (2023) HoloAssist: an egocentric human interaction dataset for interactive AI assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20270–20281. Cited by: 1st item, §4.1.
- [25] Y. Wang, X. Li, Z. Yan, Y. He, J. Yu, X. Zeng, C. Wang, C. Ma, H. Huang, J. Gao, M. Dou, K. Chen, W. Wang, Y. Qiao, Y. Wang, and L. Wang (2025) InternVideo2.5: empowering video MLLMs with long and rich context modeling. arXiv preprint arXiv:2501.12386. Cited by: §2.1, §5.1.
- [26] H. Wu, D. Li, B. Chen, and J. Li (2024) LongVideoBench: a benchmark for long-context interleaved video-language understanding. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Cited by: §2.1.
- [27] J. Xiao, A. Yao, Y. Li, and T. Chua (2024) Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13204–13214. Cited by: §1, §2.3.
- [28] Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, et al. (2024) MiniCPM-o: a gpt-4o level mllm for vision, speech and multimodal live streaming. arXiv preprint arXiv:2408.01800. Cited by: §2.1, §5.1.
- [29] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum (2020) CLEVRER: collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.2.
- [30] B. Zhang, K. Li, Z. Cheng, Z. Hu, Y. Yuan, G. Chen, S. Leng, Y. Jiang, H. Zhang, X. Li, P. Jin, W. Zhang, F. Wang, L. Bing, and D. Zhao (2025) VideoLLaMA 3: frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106. Cited by: §2.1, §5.1.
- [31] Z. Zhang, Z. Zhao, Y. Zhao, Q. Wang, H. Liu, and L. Gao (2020) Where does it exist: spatio-temporal video grounding for multi-form sentences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10668–10677. Cited by: §1, §2.3.
- [32] L. Zhou, C. Xu, and J. J. Corso (2018) Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: 1st item, §4.1.