# OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
**Authors**: {i2486147, miya}@cc.kyoto-su.ac.jp
## Abstract
Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is $N$ -th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude (from small numbers to extreme cases up to 300), (ii) arrangement complexity (from single loops to maze-like paths), and (iii) object count. The benchmark provides 39,000 question–answer pairs, each annotated with a ground-truth reasoning trajectory, balanced across difficulty, supporting controlled yet large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured, stepwise traces of the counting process and supplies an open evaluation toolkit measuring both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench offers a reproducible testbed and actionable diagnostics to drive development of VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io
## 1 INTRODUCTION
*[Error downloading image: 2603.07786v1/x5.png]*
Figure 1: Overview of OrdinalBench. We illustrate representative instances of Single-Loop (2D/3D) and Maze-Loop (2D), where models follow explicit traversal rules (start position, facing direction, stride, and blocked-turn policy) to count along a path and predict the identifier at the $N$ -th position. Each example includes the input image and question, as well as a step-wise trace output, enabling evaluation of procedural execution beyond the final answer. For brevity, the instruction prompts shown are abridged examples of the actual inputs used in our evaluation.
In recent years, Vision-Language Models (VLMs) have advanced markedly and achieve strong performance across multimodal tasks [Li et al., 2025]. Yet they still exhibit unexpected errors in fundamental visual reasoning, including accurate object enumeration (cardinal understanding) and spatial/ordinal relations [Pothiraj et al., 2025, Zhang et al., 2025, Du et al., 2024]. These failures suggest fragile generalization, potentially amplified by the next-token prediction objective underlying many models [Qi et al., 2025].
Generalization weaknesses are particularly salient in ordinal number understanding, which is crucial for real applications yet underexplored. Ordinals help uniquely identify targets when attributes are insufficient (e.g., “the black car three vehicles ahead of the white car”) and require sequential tracking, sustained attention, and state updates. For instance, in user-interface automation, an agent may be instructed to click the $N$ -th icon in a toolbar after navigating a menu hierarchy; in embodied robotics, it may need to grasp the $N$ -th part along a conveyor by following a belt or wiring route. However, existing benchmarks are not designed to diagnostically probe the generalization of such procedural ordinal number understanding.
We view ordinal number understanding not as memorized ordinal vocabulary, but as the generalization of procedural reasoning: executing “find the $N$ -th” step-by-step from visual input. We therefore introduce OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task. The core task traverses objects from a start point under a rule (e.g., clockwise; a maze traversal rule in Section 3.2) to identify the $N$ -th target. We also introduce skip counting with stride $k>1$ (count every $k$ steps) to further require algorithmic execution.
Nth-object identification comprises two processes: (i) understanding and executing the rule that determines the counting order, and (ii) proceeding correctly until reaching the specified ordinal $N$ . OrdinalBench diagnoses these independently via a spatial axis (simple loops to maze structures) and a numerical axis (small to very large $N$ ), and additionally controls visual load by varying object count (or grid size).
The contributions of this research are summarized in the following three points.
1. We develop and release OrdinalBench, a diagnostic benchmark that systematically evaluates ordinal generalization under rare large $N$ , complex loop/maze traversal, and skip counting.
1. We identify characteristic weaknesses and capability limits of current VLMs via zero-shot evaluation on OrdinalBench.
1. We discuss implications and future directions for model design, training strategies, and evaluation methods toward more robust VLMs, grounded in observed failure modes.
We describe the benchmark design and evaluation metrics, present experimental results, discuss implications and limitations, and conclude.
## 2 RELATED WORK
Many benchmarks evaluate VLM capabilities, but most do not target the generalization of procedural ordinal number understanding studied here. Existing benchmarks broadly focus on compositional reasoning or cardinal counting. For example, generalization benchmarks such as CLEVR [Johnson et al., 2017] emphasize compositional reasoning over object attributes and relations. Numerical cognition benchmarks such as TallyQA [Acharya et al., 2018] and HowManyQA [Trott et al., 2018] focus on static questions of “how many” (cardinal understanding). In contrast, identifying the $N$ -th object requires sequential rule application, maintaining internal state, and algorithmic execution, and a framework to systematically evaluate ordinal generalization under large numerical scales and complex paths has been lacking.
OrdinalBench addresses this gap with parameter-controllable synthetic data. Synthetic data is effective for precise diagnostic evaluation [Johnson et al., 2017, Mumuni et al., 2024], and strict control over factors such as ordinal magnitude and path complexity enables us to identify performance limits.
Prior work also reports that VLMs struggle with number sense and ambiguities in spatial/frame-of-reference reasoning [Du et al., 2024, Pothiraj et al., 2025, Zhang et al., 2025]. OrdinalBench fills the remaining gap of evaluation specifically focused on ordinal generalization and contributes to deeper understanding of VLM capabilities.
## 3 BENCHMARK DATASET
### 3.1 Design Philosophy
OrdinalBench is a large-scale benchmark designed to diagnose ordinal number understanding in Vision-Language Models (VLMs), particularly the generalization limits of procedural reasoning. Under controlled synthetic settings, it systematically combines: (i) complex spatial structures such as maze-loops, (ii) large ordinals that rarely appear in training data (e.g., $N\geq 100$ ), and (iii) algorithmic rules such as skip counting.
OrdinalBench follows the CLEVR-style diagnostic benchmark philosophy [Johnson et al., 2017, Mumuni et al., 2024]. By intentionally removing recognition confounders such as lighting and occlusion, it isolates ordinal number understanding as the evaluation target. To achieve this goal, we adopt the following design principles.
Targetedness
We use simple, easily identifiable visual identifiers so that errors can be attributed to reasoning rather than visual ambiguity.
Controllability & Systematization
We control difficulty systematically across three axes (arrangement complexity, object count, ordinal magnitude) and additionally introduce skip counting ( $k>1$ ) to disentangle failure factors.
Challenge
We intentionally include difficult configurations (maze structures, very large ordinals, skip counting) to expose limitations of current SOTA models and clarify directions for improvement.
Scalability
An automated generation pipeline enables continual creation of harder evaluation sets for future, stronger models [Mumuni et al., 2024].
Interpretability
Beyond final correctness, we design outputs to identify where the reasoning process breaks, enabling detailed analysis of procedural failures.
### 3.2 Task Definition
The core task is Nth-object identification. Given an image $I$ and traversal rule $R$ , a model counts starting from a reference $o_{start}$ (counted as 1st) and identifies the target at ordinal position $N$ . This task is not static pattern recognition; it requires sequential tracking under a rule and maintaining an internal state (the current count), directly probing procedural reasoning.
We formalize the task as a mapping where a VLM $f_{VLM}$ outputs a final target $\hat{o}_{N}$ and a reasoning trace $\hat{T}$ :
$$
(\hat{o}_{N},\hat{T})=f_{VLM}(I,R,o_{start},N,k).
$$
The inputs are:
- $I$ : input image (e.g., 2D maze, 3D scene)
- $R$ : traversal rule described in text (e.g., “go clockwise”; for maze-loops, see the rule below)
- $o_{start}$ : starting object (counted as 1st)
- $N$ : target ordinal position ( $N\geq 1$ )
- $k$ : counting interval (stride, $k\geq 1$ )
For maze-loop tasks, we define $R$ as follows: At each step, move forward if the cell ahead is free; otherwise, turn to the preferred side (right/left) if possible, and if that is blocked, turn to the opposite side. Mazes contain no dead ends, so this rule defines a unique next move. The outputs are:
- $\hat{o}_{N}$ : predicted $N$ -th target
- $\hat{T}$ : structured reasoning trace $\hat{T}=(\hat{t}_{1},\ldots,\hat{t}_{N})$
In our evaluation, we require models to output $(\hat{o}_{N},\hat{T})$ as a structured JSON object to make parsing and scoring unambiguous.
Stride $k$ controls cognitive load. With $k{=}1$ , counting advances at every step. With $k{>}1$ , the model must execute an algorithm that counts “every $k$ steps.” Concretely, we count $o_{start}$ as 1st; the position after $k$ steps under $R$ is 2nd, after another $k$ steps is 3rd, and so on. This diagnoses internal state management and algorithm execution beyond simple instruction following.
By requiring the trace $\hat{T}$ , we can quantify where reasoning breaks (nLCP) and how faithful the overall process is (STA) (Section 3.6).
### 3.3 Systematic Difficulty Control
OrdinalBench controls difficulty to diagnose generalization limits of procedural reasoning. The core consists of three independent axes: Arrangement Complexity, Object Count, and Ordinal Magnitude. This separation helps attribute degradation to spatial structure understanding, working-memory load, or numerical-scale generalization. We also combine these with skip counting ( $k>1$ ) for more fine-grained diagnostics.
Arrangement Complexity
We vary the visual and structural complexity of the traversal path to evaluate ordinal tracking under complex visual structures.
- Level 1 - Single-Loop: a single closed loop; evaluates basic path tracking.
- Level 2 - Maze-Loop: an algorithmically generated complex path (always a loop, with no dead ends); evaluates maintaining ordinal state over hard-to-verbalize global topology.
Object Count
We vary the total number of objects (or grid size) to evaluate attention and working-memory load in larger scenes.
- Few / Medium / Many: Single-Loop uses 5/10/20 objects; Maze-Loop uses $7{\times}7/11{\times}11/21{\times}21$ grids.
Ordinal Magnitude
We vary the target ordinal number $N$ to evaluate generalization across numerical scales. We define a baseline $S$ as “object count” for Single-Loop and “grid size” for Maze-Loop.
- Level 1 (Within): $2\leq N\leq S$
- Level 2 (Exceed): $S<N\leq 99$
- Level 3 (Large Scale): $100\leq N\leq 300$
### 3.4 Data Generation Pipeline and Scalability
OrdinalBench images and QA pairs are automatically generated by parameter-controllable programs. The pipeline consists of three stages that progressively instantiate the difficulty factors.
Scene Generation
First, we construct the visual scenes. 2D images are generated with Python libraries, and 3D scenes are built on publicly available codebases (e.g., CLEVR-Ref+ [Liu et al., 2019]). To prevent recognition difficulty from becoming noise in reasoning evaluation, we use easily identifiable labels (e.g., A00--Z99) and 48 geometric objects.
Question–Answer Pair Generation
Second, we generate QA pairs from templates based on the task definition and difficulty axes. We systematically sample the starting point, traversal rule, target ordinal $N$ , and stride $k$ to produce structurally consistent questions.
Reasoning Trace Generation
Third, for each QA, we simulate the traversal and produce a ground-truth reasoning trace. The trace records step states (identifier, position, step count, ordinal position, etc.). Even when $k>1$ , we record only the steps where a count occurs (every $k$ steps), so the trace length is always $N$ .
### 3.5 Dataset Statistics
We generate and release a reproducible evaluation split, public_test. It covers 2D Single-Loop, 2D Maze-Loop, and 3D Single-Loop, totaling 2,600 images and 39,000 QA pairs. The split is balanced across combinations of the difficulty factors (spatial structure, numerical scale, rules, and object count), making it suitable for diagnostic evaluation.
### 3.6 Evaluation Protocol and Metrics
Beyond providing the dataset, OrdinalBench contributes an evaluation protocol and metrics that diagnose the structural correctness of the reasoning process, not only final correctness. We use the following metrics.
Final Accuracy (Acc@N)
Final Accuracy measures whether the final prediction $\hat{o}_{N}$ matches the ground-truth target $o_{N}$ :
$$
\mathrm{Acc@N}=\frac{1}{M}\sum_{i=1}^{M}\mathds{1}\!\left[\hat{o}_{N}^{(i)}=o_{N}^{(i)}\right].
$$
Normalized Longest Correct Prefix (nLCP)
nLCP measures the normalized length of the longest correct prefix $LCP_{i}$ of the predicted trace, averaged over samples:
$$
\mathrm{nLCP}=\frac{1}{M}\sum_{i=1}^{M}\frac{LCP_{i}}{|T_{i}|}.
$$
Stepwise Trace Accuracy (STA)
STA evaluates whether each step of the predicted trace matches the ground truth, averaged over steps and samples:
$$
\mathrm{STA}=\frac{1}{M}\sum_{i=1}^{M}\frac{1}{|T_{i}|}\sum_{t=1}^{|T_{i}|}\mathds{1}\!\left[\hat{t}_{t}^{(i)}=t_{t}^{(i)}\right].
$$
Trace Coverage (Cov.)
Trace Coverage measures the fraction of samples for which a model returns at least one valid step in the structured JSON trace:
$$
\mathrm{Cov}=\frac{1}{M}\sum_{i=1}^{M}\mathds{1}\!\left[\,\lvert\hat{T}_{i}\rvert>0\,\right].
$$
Evaluation Toolkit
We provide an evaluation toolkit that takes model outputs (structured JSON traces) and automatically computes Acc@N, nLCP, STA, and Cov., facilitating reproducible comparisons and detailed diagnostic analysis.
## 4 EXPERIMENTS
### 4.1 Experimental Setup
In this section, we describe the experimental setup for evaluating how well current state-of-the-art VLMs can generalize procedural reasoning on OrdinalBench, with a focus on its limitations. The overall experimental design follows the research objectives presented in the introduction, aiming to fairly and rigorously test generalization to novel scenarios (e.g., complex spatial structures or unfamiliar numbers).
All results reported in this section are obtained on the public_test split generated and released with this work (Section 3.5). For API-based models (GPT-5 and Gemini 2.5), we additionally report results on stratified subsets sampled from public_test (2,500 samples for each 2D task and 1,500 for 3D) to reduce inference cost. Stratification preserves distributions over ordinal level and stride, and task-specific factors.
Evaluated Models
We evaluate API-based proprietary models (OpenAI GPT-5 [OpenAI, 2025], Gemini 2.5 Flash Lite [Comanici et al., 2025]) and major, widely used open-source VLM suites (Qwen2.5-VL [Bai et al., 2025], InternVL3.5 [Wang et al., 2025], Molmo [Deitke et al., 2025]). This selection is justified on the following three points: (1) to clarify the performance limits at the current state of the art; (2) to verify whether weaknesses are specific to particular models or are general challenges by comparing models with different architectures and training data; and (3) to enable future additional evaluations and analyses by including open-source models.
Evaluation Setting
To evaluate the true generalization capability of the models, all tasks were conducted in a zero-shot setting [Qi et al., 2025]. Zero-shot refers to a setting where models tackle tasks at inference time without any prior fine-tuning or in-context examples. This is a rigorous method for measuring the ability to handle unseen patterns, such as unfamiliar numbers or complex structures.
Implementation Details
To ensure fair comparison and reproducibility across models, the prompt design and inference parameters were carefully standardized. Each model was given a basic prompt containing the task rules, traversal method, starting point, and target ordinal number. This design follows the standard for diagnostic benchmarks, employing an identical format for all models [Johnson et al., 2017]. We also instructed the models to output their reasoning in a structured JSON format as described in Section 3.2. For inference parameters, the temperature was fixed at 0.0 to ensure deterministic outputs. This allows for an accurate comparison of the models’ intrinsic reasoning capabilities, minimizing the influence of prompt design differences and sampling stochasticity.
Evaluation Metrics
We report the four metrics defined in Section 3.6: Final Accuracy (Acc@N) and three trace-based metrics (nLCP, STA, and Cov.) to assess procedural faithfulness and output validity. Unless noted, all metrics are macro-averaged across QA instances.
### 4.2 Quantitative Results and Analysis
In this section, we quantitatively analyze the performance of VLMs evaluated using OrdinalBench. First, we provide an overview of the overall performance of all models, and then we conduct a detailed performance analysis along each difficulty axis to reveal the limits of current VLMs’ ordinal number understanding capabilities.
#### 4.2.1 Overall Performance Overview
Table 1 summarizes the overall scores of the four evaluation metrics (Acc@N, nLCP, STA, and Cov.) for all evaluated models across three task categories: 2D Single-Loop, 3D Single-Loop, and 2D Maze-Loop. We first observe a dramatic impact of task complexity on model performance. In the simplest 2D Single-Loop task, Qwen2.5-VL-32B achieved the highest Acc@N of 32.83%, and several models significantly surpassed the Chance Level (approx. 11.7%). However, in the 3D Single-Loop task, which introduces additional visual complexity, many models could not maintain their performance, although Qwen2.5-VL-72B performed well at 31.33%. Furthermore, performance dropped significantly in the most complex 2D Maze-Loop task, where even the top-scoring GPT-5 achieved only 11.04% Acc@N. Although this is above the Chance Level (approx. 2.5%), it indicates that the challenge of navigating a maze structure is extremely difficult for current VLMs.
Additionally, the Molmo series performed at or below the Chance Level on all tasks. This result is strongly correlated with their notably lower Trace Coverage (Cov.). While other models achieved near 100% coverage, the Molmo models remained in the 70-80% range for many tasks. This is due to a high number of parse failures, suggesting the Molmo models struggle with the basic instruction-following and output formatting required by the task.
Table 1: Overall performance of evaluated VLMs on OrdinalBench. We report Final Accuracy (Acc@N), Stepwise Trace Accuracy (STA), Normalized Longest Correct Prefix (nLCP), and Trace Coverage (Cov.) in percentages (%). All metrics are macro-averaged across QA instances; the highest score in each column is highlighted in bold. Note that results for GPT-5 and Gemini 2.5 are based on stratified subsets sampled from public_test (2,500/1,500; seed = 0), while others are based on the full 15,000/9,000-sample test set.
| | 2D Single-Loop | | 3D Single-Loop | | 2D Maze-Loop | | | | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Acc@N | STA | nLCP | Cov. | | Acc@N | STA | nLCP | Cov. | | Acc@N | STA | nLCP | Cov. |
| Proprietary models | | | | | | | | | | | | | | |
| GPT-5 | $15.64$ | $18.70$ | $17.07$ | $93.80$ | | $13.80$ | $12.05$ | $8.78$ | $100.00$ | | 11.04 | 17.33 | 15.45 | $99.40$ |
| Gemini 2.5 Flash Lite | $21.68$ | $21.13$ | $18.11$ | $100.00$ | | $10.93$ | $9.36$ | $6.75$ | $100.00$ | | $9.28$ | $16.79$ | $14.66$ | $99.48$ |
| Open-source models | | | | | | | | | | | | | | |
| Qwen2.5-VL-72B | $31.16$ | $31.89$ | $24.38$ | $100.00$ | | 31.33 | 33.25 | $22.82$ | $100.00$ | | $5.07$ | $11.47$ | $9.67$ | $99.54$ |
| Qwen2.5-VL-32B | 32.83 | $34.34$ | $23.79$ | $100.00$ | | $30.27$ | $31.93$ | 23.07 | $99.93$ | | $4.29$ | $10.38$ | $8.71$ | $99.32$ |
| Qwen2.5-VL-7B | $32.76$ | 36.47 | 24.53 | $96.69$ | | $14.24$ | $12.10$ | $9.80$ | $99.80$ | | $3.96$ | $9.41$ | $7.55$ | $96.91$ |
| InternVL3.5-14B | $32.45$ | $36.38$ | $22.12$ | $100.00$ | | $19.79$ | $23.98$ | $12.14$ | $99.99$ | | $3.23$ | $9.02$ | $7.28$ | $99.73$ |
| InternVL3.5-8B | $29.18$ | $31.86$ | $19.34$ | $99.44$ | | $12.36$ | $13.13$ | $8.70$ | $99.26$ | | $3.52$ | $9.40$ | $7.67$ | $96.35$ |
| Molmo-72B | $4.53$ | $9.68$ | $7.09$ | $77.37$ | | $6.79$ | $11.33$ | $6.13$ | $76.80$ | | $0.31$ | $4.94$ | $4.84$ | $74.98$ |
| Molmo-7B | $5.67$ | $9.15$ | $6.92$ | $70.37$ | | $5.41$ | $8.70$ | $6.14$ | $89.09$ | | $1.58$ | $5.99$ | $5.46$ | $82.35$ |
| Chance Level | 11.70 | $\approx 0$ | $\approx 0$ | — | | 11.70 | $\approx 0$ | $\approx 0$ | — | | 2.50 | $\approx 0$ | $\approx 0$ | — |
Chance Level. For the 2D/3D Single-Loop tasks we average the per-instance probability of uniform random guessing, $1/\text{{object\_count}}$ , which yields 11.7%. For the 2D Maze-Loop task we instead average $1/\text{{path\_length}}$ from each maze layout, resulting in roughly 2.5%. STA and nLCP chance values remain near 0% because randomly generated traces almost never match the correct step sequence.
#### 4.2.2 Performance Analysis by Difficulty Axis
Performance Degradation in Challenging Tasks (Maze Structures and skip counting)
One of the most significant findings of this study is the sharp decline in performance when models are required to perform challenging procedural reasoning. Figure 2 (right) shows that as the cognitive load increases from simple sequential tracking (stride $=1$ ) to skip counting (stride $>1$ ), the performance of all models drops substantially. For example, in the 2D Single-Loop task, Qwen2.5-VL-7B achieved an Acc@N of 42.62% with stride $=1$ , but this plummeted to 11.80% with stride $=2$ , approaching the Chance Level. This trend is even more pronounced in the nLCP metric, which indicates the robustness of the thought process, with many models dropping to the 5–6% range under stride $>1$ conditions. This is strong evidence that models struggle to reliably execute algorithmic instructions.
Impact of Ordinal Magnitude
Figure 2 (left) shows the impact of ordinal magnitude $N$ (Within / Exceed / Large Scale) on performance. In the 2D Single-Loop task, many models showed a performance decline as N increased from Within Objects to Large Scale. This trend becomes even more prominent in the 2D Maze-Loop task, where the average-performance heatmap in Figure 3 (middle) shows many models dropping to below 3% Acc@N under Large Scale conditions. This quantitatively demonstrates that VLMs have limitations in their sustained attention capacity for maintaining long-distance counts.
Impact of Scene Scale
As the total number of objects or the grid size increases, there is a tendency for model performance to decrease more sharply than the corresponding drop in the Chance Level. The heatmaps in Figure 3 show a consistent performance degradation as the number of objects (or grid size) increases. Notably, in the 3D Single-Loop task, the average Acc@N drops to single digits under heavy clutter (20 objects), approaching the Chance Level (5.00%) (Figure 3, right). This suggests that in large-scale scenes with increased visual clutter, it is difficult for models to maintain attention and execute an accurate counting strategy. *[Error downloading image: 2603.07786v1/x6.png]*
Figure 2: Performance degradation with increasing cognitive load in 2D Single-Loop. (Left) Accuracy consistently declines as the ordinal level increases from Within to Large Scale. (Right) Skip counting (stride $>1$ ) causes a sharp drop, pushing many models toward chance level. *[Error downloading image: 2603.07786v1/x7.png]*
Figure 3: Average Acc@N performance heatmaps across all models in the three task categories. Each heatmap displays the average accuracy of all evaluated models using color, with the y-axis representing the Ordinal Level and the x-axis representing either the number of objects or grid size.
## 5 DISCUSSION
We focus on the underlying causes of sharp performance degradation and the implications for future VLM development.
### 5.1 Summary and Implications
Our results reveal a fundamental weakness shared across all evaluated VLMs. While many models are strong at static pattern recognition, their procedural and algorithmic reasoning —executing a precise, multi-step procedure sequentially over visual inputs—remains limited. This weakness can be explained from two perspectives.
Language Bottleneck
VLMs may reason by internally converting visual information into linguistic representations, which can amplify reference-resolution ambiguity [Zhang et al., 2025]. Complex visual structures such as mazes are hard to transcribe faithfully into language, and information loss or ambiguity during the visual-to-language conversion can cause long-horizon reasoning to break down. Consistent failures on simple branching decisions in maze tasks suggest that models may not form stable internal representations of global spatial structure.
Lack of Procedural Knowledge
Current VLMs are strong in declarative knowledge (e.g., what an object is), yet often struggle to execute procedural knowledge (algorithms) such as “follow a maze traversal rule” or “count by skipping every three steps.” The sharp performance drops for stride $>1$ across all models strongly support this hypothesis and align with broader evidence of fragile generalization [Qi et al., 2025]. In other words, models appear to have difficulty precisely managing and updating internal state (e.g., “which ordinal are we currently counting”) while executing procedures.
Implications for Robust VLM Development
Based on the observed failure modes, our findings suggest that robustness may benefit from: (i) explicit, structured state representations to reduce the language bottleneck; (ii) training signals and curricula that emphasize step-level procedural execution (short to long horizons; stride $=1$ to $>1$ ); (iii) schema-constrained and validated generation to reduce parse failures; and (iv) evaluations beyond Acc@N using trace metrics (nLCP/STA/Cov.) and systematic stress tests.
Overall, for VLMs to function as “visual agents” rather than mere recognizers, improving the ability to accurately manipulate visual information and stably carry out multi-step reasoning is essential.
### 5.2 Limitations and Future Work
While this study provides diagnostic insights into ordinal number understanding in VLMs, it has several limitations.
Reliance on Synthetic Data
OrdinalBench is synthetic by design to isolate ordinal number understanding, and thus it does not include the full complexity of real images (e.g., lighting, occlusion, and diverse textures). Future work should extend evaluation to real-world images where recognition and reasoning are intertwined.
Static Environments
Our evaluation targets still images and does not address dynamic situations that involve temporal ordering (e.g., videos). Reasoning evaluation in dynamic environments is an important direction for applications such as autonomous driving and robotics.
## 6 CONCLUSION
In this work, we present OrdinalBench, a large-scale diagnostic benchmark for evaluating VLMs’ generalization in ordinal number understanding. It provides systematic difficulty control across 2D Single-Loop, 2D Maze-Loop, and 3D Single-Loop tasks, a skip counting task that probes procedural reasoning, and an evaluation protocol with nLCP and STA that scores step-by-step traces beyond final accuracy.
Across extensive experiments, even state-of-the-art VLMs show marked degradation under complex path tracing and algorithmic conditions, revealing a core limitation. We expect OrdinalBench to serve as a reliable yardstick that catalyzes progress toward more robust, instruction-following VLMs and, ultimately, more general visual intelligence.
## ACKNOWLEDGEMENTS
This work was partially supported by JSPS KAKENHI Grant Number 23K11342.
We used ChatGPT (OpenAI) to assist with drafting parts of this manuscript and polishing English expressions; the authors take full responsibility for the final content.
## REFERENCES
- Acharya et al., 2018 Acharya, M., Kafle, K., and Kanan, C. (2018). TallyQA: Answering complex counting questions. In Proceedings of the European Conference on Computer Vision (ECCV).
- Bai et al., 2025 Bai, S., Chen, K., Liu, X., et al. (2025). Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.
- Comanici et al., 2025 Comanici, G., Bieber, E., Schaekermann, M., et al. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.
- Deitke et al., 2025 Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., Lu, J., Anderson, T., Bransom, E., Ehsani, K., Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison-Burch, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjonsberg, S., Schmitz, M., Sarnat, A., Bischoff, B., Walsh, P., Newell, C., Wolters, P., Gupta, T., Zeng, K.-H., Borchardt, J., Groeneveld, D., Nam, C., Lebrecht, S., Wittlif, C., Schoenick, C., Michel, O., Krishna, R., Weihs, L., Smith, N. A., Hajishirzi, H., Girshick, R., Farhadi, A., and Kembhavi, A. (2025). Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 91–104.
- Du et al., 2024 Du, Y., Zhai, Q., Dai, W., and Li, X. (2024). Teach clip to develop a number sense for ordinal regression (NumCLIP). In European Conference on Computer Vision (ECCV).
- Johnson et al., 2017 Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. (2017). CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Li et al., 2025 Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H., and Shi, G. (2025). A survey of state of the art large vision language models: Alignment, benchmarks, evaluations and challenges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 10663–10672.
- Liu et al., 2019 Liu, R., Liu, C., Bai, Y., and Yuille, A. L. (2019). CLEVR-Ref+: Diagnosing visual reasoning with referring expressions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4185–4194.
- Mumuni et al., 2024 Mumuni, A., Mumuni, F., and Gerrar, N. K. (2024). A survey of synthetic data augmentation methods in computer vision. arXiv preprint arXiv:2403.10075.
- OpenAI, 2025 OpenAI (2025). GPT-5 System Card. https://openai.com/index/gpt-5-system-card/. Accessed 22 Oct 2025.
- Pothiraj et al., 2025 Pothiraj, A., Stengel-Eskin, E., Cho, J., and Bansal, M. (2025). CAPTURe: Evaluating spatial reasoning in vision-language models via occluded object counting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9994–10003.
- Qi et al., 2025 Qi, Z., Luo, H., Huang, X., Zhao, Z., Jiang, Y., Fan, X., Lakkaraju, H., and Glass, J. (2025). Quantifying Generalization Complexity for Large Language Models. In International Conference on Learning Representations (ICLR).
- Trott et al., 2018 Trott, A., Xiong, C., and Socher, R. (2018). Interpretable counting for visual question answering (HowManyQA Dataset). In International Conference on Learning Representations (ICLR).
- Wang et al., 2025 Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. (2025). InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
- Zhang et al., 2025 Zhang, Z., Hu, F., Lee, J., Shi, F., Kordjamshidi, P., Chai, J., and Ma, Z. (2025). Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities. In International Conference on Learning Representations (ICLR).