# VisGYM: Diverse, Customizable, Scalable Environments for Multimodal Agents Zirui Wang^†, Junyi Zhang^†, Jiaxin Ge^†, Long Lian, Letian Fu, Lisa Dunlap, Ken Goldberg, XuDong Wang, Ion Stoica, David M. Chan, Sewon Min, Joseph E. Gonzalez UC Berkeley ^†Equal contribution. Modern Vision–Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGYM, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: [VisGym.github.io](https://github.com/VisGym). Correspondence: {zwcolin, junyizhang, gejiaxin}@eecs.berkeley.edu The diagram illustrates the structure of VisGYM environments. On the left, 17 environments are shown, each with an Initial Frame and a Goal Frame. The environments include: Colorization, Counting, Pick & Place, Reach, Jigsaw, Matchstick Equation, Matchstick Rotation, Mental Rotation 2D, Mental Rotation 3D (Cube), Patch Reassembly, Referring Dot-Pointing, Sliding Block, Maze 2D, Maze 3D, Video Unshuffle, Zoom-In Puzzle, and Mental Rotation 3D (Objaverse). On the right, an example trajectory for the Maze 3D navigation task is shown. It consists of three frames: Current State (a maze with a red dot), Current Observation (a 3D rendering of the maze), and a sequence of actions: "Navigate the maze to find the red dot", "('move', 0)", "('turn', 1)", and "('stop', 'stop')". Figure 1. **An overview of VisGYM.** (Left) VisGYM consists of 17 diverse, long-horizon environments designed to systematically evaluate, diagnose, and train VLMs on visually interactive tasks with different domains, levels of state observability, and types of observations. (Right) An example trajectory for the Maze 3D navigation task illustrates a partially observable environment consisting of non-structured synthetic renderings. Here, a VLM is prompted with (1) the task description (*simplified in the figure*) and (2) a set of available actions to use (*not shown in the figure for simplicity*). The agent must select each action conditioned on both its past actions and observation history for its decision-making.Table 1. **Comparison among frameworks for visually interactive decision-making.** Struct. Obs. and Non-struct. Obs. indicate whether visual inputs can be parsed into structured text. POMDP denotes partial observability with hidden states. Multi-Domain covers diversity across domains (*e.g.*, robotics, computer use, games, puzzles). Scalable Episodes marks automatic, large-scale generation. SFT and Online RL show support for finetuning and reinforcement learning.

Framework	# Tasks	Environments					Training
Framework	# Tasks	Struct. Obs.	Non-struct. Obs.	POMDP	Multi Domain	Scalable Episodes	SFT	Online RL
Evaluation-only
OSWorld (Xie et al., 2024)	369	✓		✓
LIBERO (Liu et al., 2023)	130		✓	✓		✓
VideoGameBench (Zhang et al., 2025)	23	✓	✓	✓
LMGame-Bench (Hu et al., 2025)	6	✓	✓	✓
Evaluation and Training
VLABench (Zhang et al., 2025)	100		✓	✓		✓	✓	✓
VLM-Gym (Chen et al., 2025)	4	✓				✓	✓	✓
KORGym (Shi et al., 2025)	6	✓		✓		✓		✓
VisualAgentBench (Liu et al., 2024)	5	✓	✓	✓	✓		✓	✓
VAGEN (Wang et al., 2025)	5	✓	✓	✓	✓	✓		✓
VisGYM (Ours)	17	✓	✓	✓	✓	✓	✓	✓

## 1. Introduction Humans navigate complex tasks in visually rich and interactive settings: manipulating objects, using devices, or exploring unfamiliar environments. Success in these settings hinges on the tight coupling of perception, memory, and action over multiple steps (Gibson, 1979; Henderson, 2001). Foundation Vision–Language Models (VLMs) have made remarkable progress on static vision–language benchmarks (Yue et al., 2024; Lu et al., 2023; Wang et al., 2024) and on text-based multi-step tasks such as web browsing and coding (Sirdeshmukh et al., 2025; Wei et al., 2025; Jimenez et al., 2023). Yet when visual observations must be integrated into multi-step decision-making, their behavior remains far less understood. Recent evaluations across robotic manipulation, computer-use agents, and gaming agents highlight a range of challenges for visually interactive decision-making, including low task success rates, brittle visual grounding, and weak generalization (Zhang et al., 2025; Xie et al., 2024; Zhang et al., 2025; Liu et al., 2023; Hu et al., 2025; Chen et al., 2025; Shi et al., 2025; Liu et al., 2024). Although these insights are valuable, they tend to be domain-specific and observational, offering limited *systematic, controlled* diagnosis of how domain-agnostic factors—such as context length, representation modality, feedback design, or goal visibility—affect model performance across tasks. We introduce VisGYM, a highly diverse, scalable, and customizable gymnasium with 17 long-horizon environments designed to isolate what limits interactive decision-making across domains and to expose where current VLMs break down. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation tasks, each with distinct observability and dynamics and equipped with oracle multi-step solvers for supervised finetuning (framework comparison in Tab. 1). Crucially, VisGYM provides fine-grained controls over input representation, difficulty, history length, planning horizon, and feedback, enabling domain-agnostic, systematic analysis of model behavior. Building on prior domain-specific studies, we conduct cross-domain controlled experiments that examine how these factors, together with module finetuning and data curation, affect performance in multi-step visual decision-making. Across 12 state-of-the-art models, even the strongest achieve only 46.61% and 26.00% success in the easy and hard settings, respectively. Our analyses reveal several concrete, cross-domain failure modes: (1) models struggle to effectively leverage long-term context, showing a reversed-U relationship where performance degrades as the context grows unbounded; (2) VLMs struggle with low-level perceptual grounding, a limitation highlighted by symbolic variants of tasks being substantially easier than their visually rendered counterparts; (3) models struggle to infer task states and outcomes from purely visual transitions, consistently relyingTable 2. **VisGYM environments**. For each environment, we specify (1) **Domain**: whether observations come from **Real** or **Synthetic** images, (2) **Observability (Obs.)**: **Full** or potentially **Partial**, (3) **Dynamics (Dyn.)**: **Known** vs. **Unknown** dynamics, (4) **Parameters (P.)**: number of difficulty parameters, and (5) **Available Actions**.

Environment	Domain	Obs.	Dyn.	P.	Available Actions
Colorization (103)	Real	Full	Known	1	rotate( $\theta$ ); saturate( $\delta$ ); stop()
Counting (30)	Real	Full	Known	2	mark( $x, y$ ); undo(); guess( $N$ ); stop()
Jigsaw (27)	Real	Full	Known	2	swap( $(r_1, c_1), (r_2, c_2)$ ); reorder([...]); stop()
Matchstick Equation (42)	Synthetic	Full	Known	1	move([ $i, s, j, t$ ]); undo(); stop()
Matchstick Rotation (44)	Synthetic	Full	Unknown	3	move([ $dx, dy, d\theta$ ]); stop()
Maze 2D (43)	Synthetic	Full	Known	2	move( $d$ ); stop()
Maze 3D (43)	Synthetic	Partial	Known	2	move(0); turn( $d$ ); stop()
Mental Rotation 2D (18)	Real	Full	Known	1	rotate( $\theta$ ); stop()
Mental Rotation 3D (CUBE) (66; 70)	Synthetic	Partial	Known	3	rotate([ $dy, dp, dr$ ]); stop()
Mental Rotation 3D (OBJAVERSE) (70; 20)	Synthetic	Partial	Known	1	rotate([ $dr, dp, dy$ ]); stop()
MuJoCo Fetch (PICK-AND-PLACE) (86)	Synthetic	Partial	Unknown	0	move([ $x, y, z$ ]); gripper( $g$ ); stop()
MuJoCo Fetch (REACH) (86)	Synthetic	Partial	Unknown	0	move([ $x, y, z$ ]); stop()
Patch Reassembly (28)	Synthetic	Full	Known	2	place( $p, r, c$ ); remove( $p$ ); stop()
Referring Dot-Pointing (39)	Real	Full	Known	0	mark( $x, y$ ); stop()
Sliding Block (75)	Synthetic	Full	Known	1	move( $b, d$ ); stop()
Video Unshuffle (29; 60)	Real	Full	Known	3	swap( $i, j$ ); reorder([...]); stop()
Zoom-In Puzzle (6)	Real	Full	Known	5	swap( $i, j$ ); reorder([...]); stop()

on explicit textual feedback to boost performance; (4) the benefit of providing explicit goal observations is brittle and can backfire: while explicit goals can yield large gains, limited visual perception can cause models to misidentify them and, paradoxically, perform worse than with no goal at all; (5) models fail to learn from standard demonstrations under partial observability or unknown dynamics, requiring information-revealing demonstrations that expose hidden states or clarify dynamics to significantly improve supervised finetuning outcomes. Together, these findings establish VisGYM as a unified and extensible framework for diagnosing, understanding, and ultimately improving VLMs in visually interactive decision-making. ## 2. VisGYM **VisGYM** contains 17 visually interactive environments. Each environment exposes initialization parameters that control task configuration and difficulty. We provide a high-level overview of the environments in Tab. 2 and detailed descriptions with visualizations in Sec. B. VisGYM is built on top of the Gymnasium framework (Brockman et al., 2016; Towers et al., 2024), the same library underlying MuJoCo (Todorov et al., 2012) and Atari (Bellemare et al., 2013). Since vision-language agents can interpret images, read instructions, and produce free-form text, we extend Gymnasium with the following enhancements: **Function-Conditioned Action Space.** Instead of the discrete or continuous action vectors used in standard Gymnasium environments, we represent actions as function calls with parameters (*e.g.*, ('swap', (1, 2)), ('rotate', (30.5, 20.4, 15.1))). This abstraction allows models to leverage their function-calling capabilities and compose strategies across domains. **Function Instructions.** Each task defines a set of functions and their parameter spaces. To enable zero-shot rollouts, we provide a natural-language description of these functions and their argument constraints as part of the initial prompt before the model takes its first action. Instructions for each task are shown in Sec. B. **Environment Feedback.** In addition to visual transitions, the environment provides textual feedback describing the effect of each action (*e.g.*, "invalid format," "out of bounds," "executed"). This helps models with weaker visual perception better ground their actions. **Solver.** We implement heuristic multi-step solvers that complete each task using the available actions. The solver supports (1) multiple solving strategies and (2) optional stochasticity, enabling the generation ofFigure 2. Average task success rate for frontier models and our finetuned models. Proprietary models are in **bold** and our finetuned models are *italicized*. Figure 3. Density curve of steps taken for successful trajectories. Colored dashed line marks each model’s mean number of steps. diverse demonstration trajectories for supervised fine-tuning. See Sec. A for the solver design of each task. Together, these design choices yield a highly customizable interface. Each task can define its own action functions, instruction set, and solver, while the unified `step` function handles parsing, validation, execution, and feedback (Algorithm 1 in Sec. D). This modular structure makes it easy to add new tasks, vary action spaces, and generate visual and textual supervision for VLM agents. ### 3. Evaluating Frontier Models with VisGYM In this section, we evaluate vision-language models on VisGYM. We describe our evaluation setup in Sec. 3.1 and present results and observational analysis in Sec. 3.2. #### 3.1. Evaluation Setup We evaluate 12 vision-language models spanning three categories: **proprietary** (Gemini 3 Pro (team, 2025), Gemini 2.5 Pro (DeepMind, 2025), GPT-5 (OpenAI, 2025), Claude Sonnet 4 (Team, 2025), Grok 4 Fast (xAI, 2025), Qwen-VL-Max (Bai et al., 2025)); **open-weight** models (Qwen3-VL-235B-Instruct (Yang et al., 2025), GLM-4.5V (Hong et al., 2025), Llama-4-Maverick (Touvron et al., 2023), Qwen-2.5-VL-72B-Instruct (Bai et al., 2025), Gemma 3-27B-Instruct (Team et al., 2025)); and **specialized** models targeted at GUI/game environments (UI-Tars-1.5-7B (Qin et al., 2025)). We access all proprietary and hosted models through OpenRouter and thus ensure a consistent prompting interface and inference pipeline. We additionally evaluate models that we finetune on solver demonstrations. We provide details of the supervised finetuning setup in Sec. 5.1. All models are evaluated in a multi-turn manner. At each step $t$ , the model receives the full history $$H_t = (I, \{(o_\tau, a_\tau, f_\tau)\}_{\tau < t}), \quad (1)$$ where $I \in \mathbb{R}^{L_I}$ is the task instruction, $o_\tau \in \mathbb{R}^{H \times W \times C}$ the observation, $a_\tau \in \mathbb{R}^{L_a}$ the action, and $f_\tau \in \mathbb{R}^{L_f}$ the environment feedback. The model then outputs an action $a_t$ . If it outputs the stop action, the environment terminates and returns a binary reward indicating task success. In addition, we limit the number of interaction steps to 20 for the easy setting and the tasks of Dot-Pointing and Fetch-Reach, 30 for the hard setting and Fetch-Pick-n-Place task. All tasks are designed to be solvable within these limits, and the environment explicitly provides the number of remaining steps as part of its feedback. We also ensure that the length of interaction history is within models’ context window. We evaluate each model on 70 episodes per task and setting (*i.e.*, easy, hard). #### 3.2. Result and Analysis **Frontier VLMs Fail on VisGYM.** We show the per-task success rate and the average task success rate of the frontier models in Figure 4 and Figure 2, respectively. Even the best-performing frontier model, Gemini-3-Pro, achieves only 46.61% on VisGYM (Easy) and 26.00% on VisGYM (Hard), indicating that VisGYM poses a significant challenge for existing models.

		Task Success Rate per Model (%)
Single-Task Fine-Tuning		90.0	54.3	15.7	55.7	75.7	90.0	88.6	50.0	0.0	88.6	70.0	97.1	77.1	97.1	0.0	92.9	57.1	94.3	84.3	100.0	48.6	2.9	70.0	64.3	28.6	61.4	0.0	15.7	2.9	50.0	54.3
Mixed-Task Fine-Tuning		81.4	54.3	8.6	58.6	52.9	35.7	17.1	44.3	0.0	80.0	57.1	52.9	31.4	90.0	0.0	90.0	35.7	91.4	61.4	98.6	42.9	8.6	67.1	52.9	24.3	38.6	0.0	4.3	0.0	24.3	25.7
Gemini 3 Pro		87.1	54.3	21.4	78.6	51.4	77.1	61.4	91.4	44.3	82.9	75.7	51.4	31.4	37.1	7.1	45.7	24.3	74.3	41.4	7.1	17.1	11.4	1.4	1.4	0.0	31.4	12.9	4.3	1.4	4.3	4.3
GPT-5		67.1	40.0	24.3	50.0	30.0	27.1	17.1	77.1	41.4	58.6	52.9	18.6	4.3	18.6	1.4	11.4	4.3	11.4	4.3	7.1	17.1	0.0	4.3	0.0	0.0	11.4	1.4	5.7	1.4	1.4	1.4
Gemini 2.5 Pro		71.4	51.4	20.0	50.0	40.0	24.3	11.4	48.6	1.4	31.4	22.9	28.6	10.0	27.1	0.0	30.0	14.3	32.9	12.9	10.0	14.3	5.7	2.9	4.3	0.0	1.4	0.0	1.4	1.4	1.4	0.0
Grok 4 Fast		42.9	30.0	10.0	17.1	8.6	18.6	7.1	38.6	17.1	5.7	1.4	10.0	7.1	11.4	1.4	2.9	2.9	0.0	0.0	4.3	4.3	2.9	4.3	7.1	5.7	0.0	0.0	2.9	0.0	1.4	0.0
Qwen3 VL 235B Instruct		90.0	40.0	14.3	27.1	10.0	12.9	8.6	0.0	0.0	0.0	0.0	8.6	2.9	18.6	0.0	11.4	2.9	0.0	0.0	1.4	8.6	0.0	4.3	0.0	1.4	0.0	0.0	1.4	1.4	0.0	0.0
Qwen VL Max		47.1	31.4	8.6	22.9	20.0	22.9	11.4	1.4	0.0	0.0	2.9	8.6	1.4	15.7	0.0	5.7	2.9	0.0	0.0	0.0	4.3	1.4	4.3	0.0	2.9	0.0	0.0	2.9	0.0	1.4	1.4
GLM 4.5V		70.0	20.0	14.3	12.9	4.3	22.9	4.3	7.1	0.0	8.6	2.9	12.9	4.3	8.6	0.0	1.4	2.9	4.3	1.4	1.4	4.3	0.0	4.3	1.4	2.9	0.0	0.0	2.9	0.0	0.0	0.0
Claude Sonnet 4		47.1	14.3	1.4	15.7	5.7	20.0	17.1	1.4	0.0	2.9	0.0	14.3	7.1	22.9	0.0	4.3	2.9	0.0	0.0	7.1	7.1	0.0	4.3	10.0	2.9	0.0	0.0	1.4	1.4	0.0	0.0
Qwen 2.5 VL 72B Instruct		31.4	31.4	14.3	17.1	5.7	17.1	18.6	1.4	0.0	7.1	1.4	4.3	0.0	24.3	0.0	2.9	2.9	0.0	0.0	10.0	1.4	0.0	4.3	1.4	2.9	0.0	0.0	0.0	0.0	0.0	0.0
Llama 4 Maverick		44.3	37.1	10.0	20.0	10.0	12.9	8.6	1.4	0.0	2.9	1.4	0.0	0.0	8.6	0.0	5.7	2.9	0.0	0.0	12.9	5.7	1.4	4.3	0.0	0.0	1.4	0.0	0.0	0.0	0.0	0.0
Gemma 3 27B Instruct		31.4	31.4	14.3	17.1	8.6	10.0	7.1	0.0	0.0	1.4	0.0	4.3	1.4	2.9	0.0	2.9	2.9	0.0	0.0	4.3	7.1	0.0	4.3	17.1	8.6	0.0	0.0	0.0	0.0	0.0	0.0
UI TARS 1.5 7B		10.0	24.3	5.7	8.6	8.6	14.3	5.7	0.0	0.0	1.4	0.0	0.0	0.0	1.4	0.0	4.3	1.4	0.0	0.0	1.4	4.3	0.0	1.4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.4
	Referring Dot-Pointing	Counting (E)	Counting (H)	Mental Rotation 2D (E)	Mental Rotation 2D (H)	Colorization (E)	Colorization (H)	Matchstick Equation (E)	Matchstick Equation (H)	Matchstick Rotation (E)	Matchstick Rotation (H)	Maze 2D (E)	Maze 2D (H)	Jigsaw (E)	Jigsaw (H)	Zoom-In Puzzle (E)	Zoom-In Puzzle (H)	Sliding Block (E)	Sliding Block (H)	Fetch Reach	Video Unshuffle (E)	Video Unshuffle (H)	Fetch Pick-Place	Maze 3D (E)	Maze 3D (H)	Patch Reassembly (E)	Patch Reassembly (H)	Mental Rotation 3D (Objaverse) (E)	Mental Rotation 3D (Objaverse) (H)	Mental Rotation 3D (Cube) (E)	Mental Rotation 3D (Cube) (H)

Figure 4. **Task success rate of frontier and finetuned models.** Proprietary models are shown in **bold**, and our finetuned models in *italics*. (E) and (H) denote easy and hard task settings. Darker cells indicate higher success rates. Models are ordered by average task performance (top = better), and tasks by average model performance, excluding our finetuned ones (right = harder). **Model Specialization.** We compare the 3 strongest models¹: Gemini 2.5 Pro, GPT-5, Qwen3-VL-235B Instruct. GPT-5 shows the best ability to handle long-context visual interactions. This is reflected in its stronger performance on matchstick rotation where the scale is unknown, its higher scores overall on the hard setting (Fig. 2), and its visibly longer tail in the number of steps taken to successfully solve tasks compared to the other models (Fig. 3). Gemini 2.5 Pro is good at low-level visual perception. This is reflected in its strongest performance on Jigsaw, Maze 2D, Zoom-In Puzzle, and Sliding Block, all of which demand tight spatial alignment, accurate correspondence of local patterns, and sensitivity to subtle visual cues. Qwen-3-VL is in particular capable of object localization (*e.g.*, strongest in Referring Dot-Pointing). Examining the step count distribution (smoothed density curve) for successful trajectories across models (Fig. 3), we found that most models (*i.e.*, Gemini 2.5 Pro, Claude Sonnet 4, and Llama-4-Maverick) only peaked around 3-5 steps, followed by a sharp drop in successful trajectories when they spend more steps. This indicates limited capability in effectively handling long-context multi-step visual interactions. **Common Failure Patterns.** We identify recurring failures using automated failure discovery methods (Dunlap et al., 2025; Lisa Dunlap et al., 2025), which employ a VLM annotator (GPT-4.1) to extract negative behaviors from each trajectory and cluster them into categories observed across datasets. This analysis reveals four failure types that appear consistently across multiple tasks (see Sec. F for details): (1) *Restricted action space and action looping*: models often rely on a single repeated operation or fixed-magnitude action, such as continually moving in the same direction in Fetch Pick & Place, using “swap” in Jigsaw instead of “reorder”, or rotating by the same angle in Mental Rotation 3D and Match Rotation rather than converging to an optimal magnitude. (2) *State mismanagement*: models fail to maintain or update internal state across steps. They ignore textual or environmental feedback, revisit previously explored areas, or repeat illegal actions despite prior errors—for ¹Gemini 3 Pro is excluded from this detailed comparison, as it was released after this analysis concluded.example, continuing to move into a wall after being told they have collided, or repeating invalid moves in the Match Equation, Sliding Block, and Toy Maze 2D tasks. (3) *Early termination*: the model terminates before the maximum steps despite not reaching the goal. (4) *Failure to use visual or spatial information*: models ignore the visual information provided, such as the target leaving the frame or the item being successfully aligned (*e.g.*, Mental Rotation). ## 4. Diagnosing Frontier Models with VisGYM In this section, we show the flexibility of VisGYM by presenting controlled diagnoses of how different designs of multi-step interactions can drastically change frontier models’ performance and provide our conjectures on why these designs make a difference. We perform diagnoses with the two best performing proprietary models, *i.e.*, GPT-5 (OpenAI, 2025) and Gemini 2.5 Pro (DeepMind, 2025), and the two best performing open-weight models, *i.e.*, Qwen3-VL-235B Instruct (Yang et al., 2025) and GLM-4.5V (Hong et al., 2025). ### 4.1. Turns to Keep in Conversation History Vision–language models are known to degrade with long visual context (Wang et al., 2025; Wu et al., 2024; Sharma et al., 2024). This creates a dilemma: while long histories provide more information about the environment (*e.g.*, 3D layouts, unknown dynamics), they also introduce redundant observations that may harm performance. We study this trade-off in Maze2D, Sliding Block, MuJoCo Fetch Reach, and Matchstick Rotation, where history provides useful signals such as textual feedback (*e.g.*, invalid actions) or correspondence between action magnitude and perceptual effect, but also introduces stale information. Figure 5. **Effect of truncating conversational context on model performance.** The settings 1, 2, 4, and $\infty$ correspond to retaining only the current turn, the current + previous turn, the current + previous 3 turns, and the full history, respectively. Error bars show the standard error of the mean. As shown in Fig. 5, models benefit from including a limited number of previous turns up to roughly four, following a drop when given the full unbounded history. This indicates that expanding visual context helps multi-step visual decision-making only to a point, after which irrelevant or stale observations become detrimental. We also observe task-specific idiosyncrasies: Gemini 2.5 Pro scales well in Maze2D, GPT-5 scales well on Matchstick Rotation, while Sliding Block exhibits clear *reverse scaling* for Gemini 2.5 Pro. These highlight that the value of interaction history is both task-dependent. ### 4.2. Representing Observation in Text Inspired by prior work examining how different task representations affect agent performance (Hu et al., 2025; Shi et al., 2025; Ruoss et al., 2024), we select four symbolic tasks—Matchstick Equation, Maze 2D, Patch Reassembly, and Sliding Block—and implement alternate versions rendered entirely in ASCII (sample ASCII visualizations are provided in Sec. C). This allows tasks to be solved without any visual encoding module. The results in Fig. 6 show that GPT-5 substantially improves in most tasks, often achieving $3 - 4\times$ higher success rates than in the visual setting, suggesting that its main bottleneck lies in visual grounding rather than long-horizon reasoning. Gemini 2.5 Pro shows mixed behavior: two tasks do not exhibit significant performance change, one task improves, and one task degrades, indicating possible limitations in bothFigure 6. **Effect of visualizing observations with ASCII (text).** “Image” and “Text” denote the observation modalities. Error bars show the standard error of the mean. perception and planning. Open-weight models struggle across all tasks in both modalities, indicating general weaknesses in long-horizon decision-making regardless of representation. Interestingly, Matchstick Equation exhibits a *reverse* trend: all models perform substantially better with the visual representation than with ASCII, likely because the figlet-style ASCII has irregular shapes and spacing that create distorted glyphs which models are known to struggle with (Stojanovski et al., 2025). ### 4.3. Removal of Text-based Feedback Humans can infer action consequences directly from visual changes (Michotte, 1963), but it remains unclear whether VLMs can do the same. To study this, we select four tasks—Maze 3D, Maze 2D, Sliding Block, and Matchstick Equation—in which the environment feedback $f$ (see Eq. (1)) provides not only formatting errors but also constraint violations (e.g., hitting a wall in Maze, sliding a block into an occupied cell). We remove this textual feedback and evaluate model using only visual state transitions; results are shown in Fig. 7. Figure 7. **Effect of removing text-based environment feedback.** “With Feedback” includes environment feedback describing action execution at each turn; “No Feedback” removes this channel. Error bars show the standard error of the mean. All models show consistent drops in average performance. This indicates that models struggle to infer action validity directly from visual transitions. These findings show that current VLMs depend heavily on text-based feedback during visually interactive decision-making and are less sensitive to pure visual feedback. ### 4.4. Providing Final Goal at Beginning Providing the solution image upfront simplifies the tasks to visually aligning current observations with a known target, shifting the difficulty from reasoning to visual perception and tool-calling. We test this on five tasks, Patch Reassembly, Jigsaw, Colorization, Zoom-In Puzzle, and Matchstick Equation, where constructing the goal observation involves significant effort. For these tasks, we augment the instruction with the ground-truth final observation $o_{gt}$ , and show results in Fig. 8. Figure 8. **Effect of providing the final goal observation at the beginning of the episode.** “No Final Obs.” and “With Final Obs.” denote settings without and with access to the goal observation at the start. Error bars show the standard error of the mean. Across tasks, models improve substantially, indicating that a major bottleneck lies in *constructing or imagining the target state*. However, performance remains far from perfect, indicating additional limitations beyondreasoning, such as fine-grained visual perception and action calling. Surprisingly, GPT-5 and Gemini 2.5 Pro *underperform* on the Zoom-In Puzzle and Matchstick Equation when the final goal observation is provided, often terminating early despite visible misalignment. A follow-up test confirms this stems from visual misjudgment due to limited visual perception: we queried Gemini 2.5 Pro on 100 pairs of initial and final-goal observations with the prompt “Do the two images look exactly the same?” and it incorrectly judged images as identical 80% and 57% of the time for these tasks, versus only 18%, 2%, and 0% for Colorization, Jigsaw, and Patch Reassembly. This confirms that perception errors can *invert* the expected benefit of an explicit goal observation. ## 5. Training with VisGYM We describe our supervised fine-tuning experiments with VisGYM, present results, and provide insights on generalization, module specificity, and data curation. ### 5.1. Supervised Fine-Tuning Experiments **Setup.** We generate demonstration trajectories for supervised fine-tuning using the multi-step solver described in Sec. 2. We apply two preprocessing filters: (1) discarding trajectories that fail to complete the task, and (2) removing trajectories with initial states overlapping the test split to prevent data leakage. We evaluate two fine-tuning configurations: *single-task* and *mixed-task*. In the *single-task* setting, we fine-tune a separate model for each task, whereas in the *mixed-task* setting, a single model is trained jointly on all tasks. Notably, demonstrations are sourced exclusively from the easy difficulty level; thus, performance on the hard setting serves as a metric for difficulty generalization. All experiments employ Qwen2.5-VL-7B-Instruct (Bai et al., 2025) with full-parameter fine-tuning, a global batch size of 64, a learning rate of $1 \times 10^{-5}$ , and bfloat16 precision. Models are trained for 1,500 steps in the single-task setting and 5,000 steps in the mixed-task setting. We utilize LlamaFactory (Zheng et al., 2024) for all data preprocessing and training orchestration. **Results.** As shown in Figs. 2 and 4, finetuned models achieve state-of-the-art performance on most tasks, validating both the learnability of our environments and the effectiveness of our multi-step solvers. These gains confirm that current VLMs can substantially benefit from structured, solver-generated demonstrations in visually grounded multi-step settings. ### 5.2. Stronger Base Model Generalizes Better Existing work has discussed the limitations of supervised finetuning (Ross et al., 2011) and found that it exhibits limited generalization to task variants (Caccia et al., 2024; Deng et al., 2023; Jang et al., 2022). This motivates re-examining generalization in the context of modern VLMs, whose capabilities may shift the boundary of what supervised finetuning can or cannot retain. To this end, we select a set of environments where the easy-to-hard difficulty gap introduces substantial state changes (e.g., more views in the Zoom-In Puzzle, more patches in Patch Reassembly, larger maze sizes; details in Sec. E). We finetune Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct on the same mixed-task training data using identical optimization hyperparameters (see Sec. 5.1), and report performance in Fig. 9. As shown, both models achieve comparable perfor- Figure 9. **Generalization to unseen difficulty from mixed-task supervised finetuning.** (Left): average success rate across 7 tasks in easy (seen) and hard (unseen) settings for Qwen2.5-VL and Qwen3-VL. (Right): task-level plot comparing success rates; X-axis = easy, Y-axis = hard.mance on the easy variants they were trained on (e.g., 0.59 vs. 0.64), but the more recent Qwen3-VL generalizes substantially better to the harder variants, nearly doubling the success rate on average relative to Qwen2.5-VL. This trend highlights that newer VLMs provide stronger out-of-distribution generalization in multi-step visual decision-making despite being finetuned on an identical setup. ### 5.3. Vision and LLM Both Matter Classic perception-action theories emphasize that fine-grained visual encoding and temporal integration are jointly necessary for interactive behavior (Gibson, 1979). We examine whether this holds for VLMs by fine-tuning variants that modify either the vision encoder or the LLM backbone to isolate each module’s contribution, where the vision encoder provides fine-grained perceptual features and the LLM performs temporal integration across steps. As shown in Fig. 10, most tasks benefit from finetuning both components, with the LLM contributing the larger performance gain—particularly in tasks with partial observability or unknown environment dynamics. This highlights that temporal reasoning and history integration remain the primary bottlenecks for current VLMs, while strong fine-grained visual encoding is necessary (e.g., Zoom-In Puzzle primarily benefits from vision finetuning) but often not sufficient for multi-step decision-making. Figure 10. **Tasks benefiting from finetuning different modules.** “Vision Gain” and “LLM Gain” denote improvements from jointly finetuning both components, compared to finetuning only the *LLM* or the *vision* part. The dashed line ( $y = x$ ) divides vision-favored (above) and LLM-favored (below) synergy. “Full” and “Partial” denote whether observability and dynamics are fully known. ### 5.4. Importance of Information-Revealing Behaviors for SFT Curation Not all experiences contribute equally to decision-making: trajectories that reveal hidden states or disambiguate perceptual aliasing are often far more valuable (McCallum, 1994; Fujii et al., 1998). We ask whether inducing such information-revealing behaviors during supervised finetuning helps VLMs form more accurate state representations. We evaluate this on two tasks, Matchstick Rotation (unknown dynamics) and Mental Rotation 3D Objaverse (partial observability), with results in Figs. 11 and 12. In Matchstick Rotation, the baseline demonstrations perform three stochastic moves toward the target. In contrast, the information-revealing demonstrations first perform two unit-scale steps to expose the correspondence between action magnitude and perceptual effect before executing the final aligning move. This structured exploration raises success from 32.9% to 70.0%. In Mental Rotation, the baseline trajectories rotate along each principal axis once to reach the goal, while the information-revealing ones deliberately fully rotate along each axis to expose the full 3D geometry before settling on the target orientation. This strategy improves performance in both metrics. To verify that gains are not simply due to longer trajectories, we further continue training on baseline demonstrations Figure 11. **Effect of data curation strategies on task performance when environment dynamics are unknown.** Numbers represent average task success (higher is better). “3 Moves” and “2 Unit Moves + 1 Move” are two curation strategies. Figure 12. **Effect of data curation strategies on task performance when environment is partially observable.** “Solve-Only” and “Rotate-Then-Solve” are two curation strategies, and “Continued Training on Solve-Only” denotes further finetuning on Solve-Only after training on Rotate-Then-Solve. (Left): final angular error on the test set (lower is better). (Right): average task success rate (higher is better).starting from the model already finetuned with information-revealing data. Performance deteriorates in this setting, confirming that the observed improvements stem from the *informative structure* of the demonstrations rather than quantity or length. These results highlight that SFT effectiveness depends on whether demonstrations induce state-disambiguating behaviors, not merely on the number of examples. ## 6. Related Work The development of foundation models (Achiam et al., 2023; Team et al., 2023; Team, 2025; Dubey et al., 2024; Yang et al., 2025), particularly vision-language models (VLMs) (Hong et al., 2025; Bai et al., 2025; Team et al., 2025; Zhu et al., 2025; Team et al., 2025; Liu et al., 2023; Alayrac et al., 2022; Li et al., 2022) and vision-language-action models (VLAs) (Black et al., 2024; Bjorck et al., 2025; Team et al., 2025; Kim et al., 2024; Octo Model Team et al., 2024; Huang et al., 2025; Team et al., 2025; Zhou et al., 2025; Niu et al., 2024; Shi et al., 2025), has reshaped how AI agents perceive, make decisions, and act across physical and simulated environments. To properly assess the capabilities and limitations of the models, a plethora of benchmarks have been developed. Early benchmarks such as Atari, OpenAI Gym, and DeepMind Lab (Mnih et al., 2013; Brockman et al., 2016; Towers et al., 2024; Beattie et al., 2016) were developed to evaluate vision-based control and decision-making in fully observable environments. These platforms laid the groundwork for reinforcement learning but focused primarily on low-level motor control. Subsequent efforts extended these ideas to robotic manipulation and navigation, introducing partially observable, multi-task, and long-horizon settings that better reflect real-world complexity (Yu et al., 2020; Ahmed et al., 2020; Shridhar et al., 2020; Li et al., 2024; Cao et al., 2025; Liu et al., 2023; Khanna et al., 2024; Choi et al., 2024; Yang et al., 2025; Ehsani et al., 2021; Szot et al., 2021; Srivastava et al., 2022; Mees et al., 2022; Mandelkar et al., 2021; James et al., 2020). These modern suites enable training and evaluation of multi-task imitation learning and meta-learning policies across diverse embodiment and task horizons. Concurrently, many VLM benchmarks have been developed to probe models’ cognitive and perceptual limits. Early efforts focused on visual question answering—first as multiple-choice tasks and later as open-ended reasoning (Yue et al., 2024, 2025; Liu et al., 2024; Li et al., 2024; Chen et al., 2024). As visual grounding and reasoning improved, newer benchmarks began representing actions through text instead of fixed, predefined action spaces, enabling studies of the interplay between perception, reasoning, and control (Wang et al., 2025; Jang et al., 2024; Liu et al., 2023; Shi et al., 2025; Stojanovski et al., 2025; Abdulhai et al., 2023; Coelho et al., 2025). For example, G1 (Chen et al., 2025) introduces VLM-Gym, a suite of visual game environments with unified interfaces and adjustable difficulty. Broader evaluation suites such as VisualAgentBench (Liu et al., 2024), EmbodiedBench (Yang et al., 2025), and WebArena (Zhou et al., 2023) aggregate tasks across embodied control, graphical interfaces, and visual reasoning, challenging agents with multi-step planning and tool use. VisGYM unifies reasoning and control under an RL-style “gym” paradigm, combining 17 multimodal tasks spanning visual puzzles, spatial reasoning, manipulation, and grounding. Each environment includes an oracle solution to ensure solvability and allow synthetic trajectory generation for post-training. Moreover, VisGYM introduces controllable difficulty along with targeted diagnostics—such as history utilization, representation variants, feedback specificity, and perception–action causality—allowing researchers to examine not only whether models fail but also what causes failures. We hope these designs enable more systematic analysis of VLMs and VLAs across domains and levels of interactivity (Tab. 1). ## 7. Conclusion We present VisGYM, a unified suite of 17 visually interactive environments that challenge and train vision–language models in multi-step visual decision-making. VisGYM establishes a rigorous playground for building the next generation of multimodal agents, bridging perception and reasoning toward more capable, adaptive visual intelligence.## Acknowledgement We thank Trevor Darrell, Jacob Steinhardt, Yichuan Wang, Ryan Yixiang Wang, Haiwen Feng, Baifeng Shi, Jihan Yang and Ziqiao Ma for their feedback. We also thank OpenRouter for its support in our model evaluation. Authors, as part of their affiliation with UC Berkeley, were supported by gifts from Accenture, AMD, Anyscale, Broadcom, Cisco, Google, IBM, Intel, Intesa Sanpaolo, Lambda, Lightspeed, Mibura, Microsoft, NVIDIA, Qualcomm, Samsung SDS, and SAP. Authors, as part of their affiliation with UC Berkeley, were supported in part by the National Science Foundation, US Department of Defense, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program. This research was also developed with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. W912CG-24-C-0011. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of any sponsor, the Department of Defense, or the U.S. Government. ## References - [1] Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. *arXiv preprint arXiv:2311.18232*, 2023. 10 - [2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. *arXiv preprint arXiv:2303.08774*, 2023. 10 - [3] Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Yoshua Bengio, Bernhard Schölkopf, Manuel Wüthrich, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. *arXiv preprint arXiv:2010.04296*, 2020. 10 - [4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. *Advances in neural information processing systems*, 35:23716–23736, 2022. 10 - [5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. *arXiv preprint arXiv:2502.13923*, 2025. 4, 8, 10 - [6] John C. Baird. *Psychophysical Analysis of Visual Space*. International Series of Monographs in Experimental Psychology. Pergamon Press / Elsevier, 1970. ISBN 978-0-08-013876-3. URL . 3 - [7] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. *arXiv preprint arXiv:1612.03801*, 2016. 10 - [8] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. *Journal of Artificial Intelligence Research*, 47:253–279, jun 2013. 3 - [9] Johan Björck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots. *arXiv preprint arXiv:2503.14734*, 2025. 10 - [10] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. $\pi$ 0: A vision-language-action flow model for general robot control. corr, abs/2410.24164, 2024. doi: 10.48550. *arXiv preprint ARXIV.2410.24164*, 2024. 10 - [11] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. 3, 10 - [12] Massimo Caccia, Megh Thakkar, Léo Boisvert, Thibault Le Sellier de Chezelles, Alexandre Piché, Nicolas Chapados, Alexandre Drouin, Maxime Gasse, and Alexandre Lacoste. Fine-tuning web agents: It works, but it’s trickier than you think. In *NeurIPS 2024 Workshop on Open-World Agents*, 2024. URL . 8 - [13] Guanqun Cao, Ryan McKenna, Erich Graf, and John Oyekan. Learn from the past: Language-conditioned object rearrangement with large language models. *arXiv preprint arXiv:2501.18516*, 2025. 10- [14] Liang Chen, Hongcheng Gao, Tianyu Liu, Zhiqi Huang, Flood Sung, Xinyu Zhou, Yuxin Wu, and Baobao Chang. G1: Bootstrapping perception and reasoning abilities of vision-language model via reinforcement learning. *arXiv preprint arXiv:2505.13426*, 2025. 2, 10 - [15] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? *Advances in Neural Information Processing Systems*, 37:27056–27087, 2024. 10 - [16] Jae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jaehong Kim, and Minsu Jang. Lota-bench: Benchmarking language-oriented task planners for embodied agents. *arXiv preprint arXiv:2402.08178*, 2024. 10 - [17] João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, et al. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research. *arXiv preprint arXiv:2505.19253*, 2025. 10 - [18] Lynn A. Cooper and Roger N. Shepard. Chronometric studies of the rotation of mental images. In WILLIAM G. CHASE, editor, *Visual Information Processing*, pages 75–176. Academic Press, 1973. ISBN 978-0-12-170150-5. doi: . URL . 3 - [19] Google DeepMind. Gemini 2.5 Pro Technical Report. Technical report, Google DeepMind, 2025. URL . Accessed: 2025-11-06. 4, 6 - [20] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 13142–13153, 2023. 3, 23 - [21] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. *Advances in Neural Information Processing Systems*, 36:28091–28114, 2023. 8 - [22] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. *arXiv e-prints*, pages arXiv-2407, 2024. 10 - [23] Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E Gonzalez. Vibecheck: Discover and quantify qualitative differences in large language models. In *International Conference on Learning Representations (ICLR)*, 2025. URL . 5, 24 - [24] Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4497–4506, 2021. 10 - [25] Atsushi Fujii, Kentaro Inui, Takenobu Tokunaga, and Hozumi Tanaka. Selective sampling for example-based word sense disambiguation. *Computational Linguistics*, 24(4):573–597, 1998. URL . 9 - [26] James J. Gibson. *The Ecological Approach to Visual Perception*. Houghton Mifflin, Boston, MA, 1979. ISBN 0-395-27049-9. 2, 9 - [27] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. *arXiv preprint arXiv:1803.07728*, 2018. 3 - [28] Solomon W. Golomb. *Polyominoes: Puzzles, Patterns, Problems, and Packings*. Princeton University Press, Princeton, NJ, 2nd edition, 1994. ISBN 978-0-691-02444-8. 3 - [29] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In *Proceedings of the IEEE international conference on computer vision*, pages 5842–5850, 2017. 3, 23 - [30] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 5356–5364, 2019. 3, 23- [31] John M. Henderson. A sensorimotor account of vision and visual consciousness. *The Behavioral and brain sciences*, 24 5:939–73; discussion 973–1031, 2001. URL . 2 - [32] Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. *arXiv e-prints*, pages arXiv–2507, 2025. 4, 6, 10 - [33] Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Imggame-bench: How good are llms at playing games? *arXiv preprint arXiv:2505.15146*, 2025. 2, 6 - [34] Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, and Pieter Abbeel. Otter: A vision-language-action model with text-aware feature extraction. *arXiv preprint arXiv:2503.03734*, 2025. 10 - [35] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. *IEEE Robotics and Automation Letters*, 5(2):3019–3026, 2020. 10 - [36] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappeler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In *Conference on Robot Learning*, pages 991–1002. PMLR, 2022. 8 - [37] Lawrence Jang, Yinheng Li, Dan Zhao, Charles Ding, Justin Lin, Paul Pu Liang, Rogerio Bonatti, and Kazuhito Koishida. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks. *arXiv preprint arXiv:2410.19100*, 2024. 10 - [38] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? *arXiv preprint arXiv:2310.06770*, 2023. 2 - [39] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 787–798, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1086. URL . 3, 23 - [40] Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi. Goat-bench: A benchmark for multi-modal lifelong navigation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 16373–16383, 2024. 10 - [41] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. *arXiv preprint arXiv:2406.09246*, 2024. 10 - [42] Günther Knoblich, Stellan Ohlsson, Hilde Haider, and Detlef Rhenius. Constraint relaxation and chunk decomposition in insight problem solving. *Journal of Experimental Psychology: Learning Memory and Cognition*, 25(6):1534–1555, November 1999. ISSN 0278-7393. doi: 10.1037/0278-7393.25.6.1534. 3 - [43] Anurag Koul. maze-world: Random maze environments with different size and complexity for reinforcement learning and planning research. , 2024. Accessed: 2025-11-07. 3 - [44] Russell Kruger, Sheelagh Carpendale, Stacey D. Scott, and Anthony Tang. Fluid integration of rotation and translation. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '05*, page 601–610, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1581139985. doi: 10.1145/1054972.1055055. URL . 3 - [45] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13299–13308, 2024. 10 - [46] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, et al. Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. *arXiv preprint arXiv:2403.09227*, 2024. 10- [47] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In *International conference on machine learning*, pages 12888–12900. PMLR, 2022. 10 - [48] Lisa Dunlap, Trevor Darrell, Jacob Steinhardt, and Joseph Gonzalez. Stringsight: Automating analysis of model outputs. , 2025. 5, 24 - [49] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. *Advances in Neural Information Processing Systems*, 36:44776–44791, 2023. 2, 10 - [50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In *NeurIPS*, 2023. 23 - [51] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. *Advances in neural information processing systems*, 36:34892–34916, 2023. 10 - [52] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In *The Twelfth International Conference on Learning Representations*, 2023. 10 - [53] Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. *arXiv preprint arXiv:2408.06327*, 2024. 2, 10 - [54] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In *European conference on computer vision*, pages 216–233. Springer, 2024. 10 - [55] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. *arXiv preprint arXiv:2310.02255*, 2023. 2 - [56] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. *arXiv preprint arXiv:2108.03298*, 2021. 10 - [57] R. Andrew McCallum. Instance-based state identification for reinforcement learning. In G. Tesaro, D. Touretzky, and T. Leen, editors, *Advances in Neural Information Processing Systems*, volume 7. MIT Press, 1994. URL [https://proceedings.neurips.cc/paper\\_files/paper/1994/file/d2ed45a52bc0edfa11c2064e9edee8bf-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/1994/file/d2ed45a52bc0edfa11c2064e9edee8bf-Paper.pdf). 9 - [58] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. *IEEE Robotics and Automation Letters*, 7(3):7327–7334, 2022. 10 - [59] Albert Michotte. *The Perception of Causality*. Routledge, 1 edition, 1963. doi: 10.4324/9781315519050. URL . 7 - [60] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In *European conference on computer vision*, pages 527–544. Springer, 2016. 3 - [61] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. *arXiv preprint arXiv:1312.5602*, 2013. 10 - [62] Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. Llarva: Vision-action instruction tuning enhances robot learning. *arXiv preprint arXiv:2406.11815*, 2024. 10 - [63] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In *Proceedings of Robotics: Science and Systems*, Delft, Netherlands, 2024. 10 - [64] OpenAI. GPT-5 System Card. System card, OpenAI, August 2025. URL . Accessed: 2025-11-06. 4, 6 - [65] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. *arXiv preprint arXiv:2501.12326*, 2025. 4- [66] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? *arXiv preprint arXiv:2410.06468*, 2024. 3 - [67] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In *Proceedings of the fourteenth international conference on artificial intelligence and statistics*, pages 627–635. JMLR Workshop and Conference Proceedings, 2011. 8 - [68] Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. *arXiv preprint arXiv:2412.01441*, 2024. 6 - [69] Aditya Sharma, Michael Saxon, and William Yang Wang. Losing visual needles in image haystacks: Vision language models are easily distracted in short and long contexts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, *Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 5429–5451, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.312. URL . 6 - [70] R. N. Shepard and J. Metzler. Mental rotation of three-dimensional objects. *Science*, 171(3972):701–703, 1971. doi: 10.1126/science.171.3972.701. 3 - [71] Jiajun Shi, Jian Yang, Jiaheng Liu, Xingyuan Bu, Jiangjie Chen, Junting Zhou, Kaijing Ma, Zhoufutu Wen, Bingli Wang, Yancheng He, et al. Korgym: A dynamic game platform for llm reasoning evaluation. *arXiv preprint arXiv:2505.14552*, 2025. 2, 6, 10 - [72] Modi Shi, Li Chen, Jin Chen, Yuxiang Lu, Chiming Liu, Guanghui Ren, Ping Luo, Di Huang, Maoqing Yao, and Hongyang Li. Is diversity all you need for scalable robotic manipulation? *arXiv preprint arXiv:2507.06219*, 2025. 10 - [73] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10740–10749, 2020. 10 - [74] Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. *arXiv preprint arXiv:2501.17399*, 2025. 2 - [75] Ruben Spaans. Solving sliding-block puzzles. Specialisation project report, Norwegian University of Science and Technology (NTNU), 2009. URL . 3 - [76] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In *Conference on robot learning*, pages 477–490. PMLR, 2022. 10 - [77] Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefiyo, Jean Kaddour, and Andreas Kopf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. *arXiv preprint arXiv:2505.24760*, 2025. 7, 10 - [78] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. *Advances in neural information processing systems*, 34:251–266, 2021. 10 - [79] Claude Team. Claude 4 sonnet, 2025. URL . 4, 10 - [80] Gemini team. A new era of intelligence with Gemini 3, November 2025. URL . Accessed: 2026-01-23. 4 - [81] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. *arXiv preprint arXiv:2312.11805*, 2023. 10 - [82] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. *arXiv preprint arXiv:2503.20020*, 2025. 10- [83] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. *arXiv preprint arXiv:2503.19786*, 2025. 4, 10 - [84] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. *arXiv preprint arXiv:2504.07491*, 2025. 10 - [85] TRI LBM Team, Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, Naveen Kuppuswamy, Kuan-Hui Lee, Katherine Liu, Dale McConachie, Ian McMahon, Haruki Nishimura, Calder Phillips-Grafflin, Charles Richter, Paarth Shah, Krishnan Srinivasan, Blake Wulfe, Chen Xu, Mengchao Zhang, Alex Alspach, Maya Angeles, Kushal Arora, Vitor Campagnolo Guizilini, Alejandro Castro, Dian Chen, Ting-Sheng Chu, Sam Creasey, Sean Curtis, Richard Denitto, Emma Dixon, Eric Dusel, Matthew Ferreira, Aimee Goncalves, Grant Gould, Damrong Guoy, Swati Gupta, Xuchen Han, Kyle Hatch, Brendan Hathaway, Allison Henry, Hillel Hochsztein, Phoebe Horgan, Shun Iwase, Donovan Jackson, Siddharth Karamcheti, Sedrick Keh, Joseph Masterjohn, Jean Mercat, Patrick Miller, Paul Mitiguy, Tony Nguyen, Jeremy Nimmer, Yuki Noguchi, Reko Ong, Aykut Onol, Owen Pfannenstiehl, Richard Poyner, Leticia Priebe Mendes Rocha, Gordon Richardson, Christopher Rodriguez, Derick Seale, Michael Sherman, Mariah Smith-Jones, David Tago, Pavel Tokmakov, Matthew Tran, Basile Van Hoorick, Igor Vasiljevic, Sergey Zakharov, Mark Zolotas, Rares Ambrus, Kerri Fetzer-Borelli, Benjamin Burchfiel, Hadas Kress-Gazit, Siyuan Feng, Stacie Ford, and Russ Tedrake. A careful examination of large behavior models for multitask dexterous manipulation. *arXiv preprint arXiv:2507.05331*, 2025. URL . 10 - [86] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In *2012 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pages 5026–5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109. 3 - [87] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*, 2023. 4 - [88] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. *arXiv preprint arXiv:2407.17032*, 2024. 3, 10 - [89] Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 3221–3241, 2025. 6 - [90] Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, et al. Vagen: Reinforcing world model reasoning for multi-turn vlm agents. *arXiv preprint arXiv:2510.16907*, 2025. 2 - [91] Zifu Wang, Junyi Zhu, Bo Tang, Zhiyu Li, Feiyu Xiong, Jiaqian Yu, and Matthew B Blaschko. Jigsaw-r1: A study of rule-based visual reinforcement learning with jigsaw puzzles. *arXiv preprint arXiv:2505.23590*, 2025. 10 - [92] Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. *Advances in Neural Information Processing Systems*, 37:113569–113697, 2024. 2 - [93] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. *arXiv preprint arXiv:2504.12516*, 2025. 2 - [94] Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Visual haystacks: A vision-centric needle-in-a-haystack benchmark. *arXiv preprint arXiv:2407.13766*, 2024. 6 - [95] xAI. Grok 4 Model Card. Model card, xAI, August 2025. URL . Accessed: 2025-11-06. 4- [96] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. *Advances in Neural Information Processing Systems*, 37: 52040–52094, 2024. 2 - [97] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. *arXiv preprint arXiv:2505.09388*, 2025. 4, 6, 10 - [98] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. *arXiv preprint arXiv:2502.09560*, 2025. 10 - [99] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In *Conference on robot learning*, pages 1094–1100. PMLR, 2020. 10 - [100] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9556–9567, 2024. 2, 10 - [101] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 15134–15186, 2025. 10 - [102] Alex L Zhang, Thomas L Griffiths, Karthik R Narasimhan, and Ofir Press. Videogamebench: Can vision-language models complete popular video games? *arXiv preprint arXiv:2505.18134*, 2025. 2 - [103] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In *European conference on computer vision*, pages 649–666. Springer, 2016. 3 - [104] Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoeye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, et al. Vlabench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 11142–11152, 2025. 2 - [105] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, Bangkok, Thailand, 2024. Association for Computational Linguistics. URL . 8 - [106] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. *arXiv preprint arXiv:2307.13854*, 2023. 10 - [107] Zhongyi Zhou, Yichen Zhu, Junjie Wen, Chaomin Shen, and Yi Xu. Vision-language-action model with open-world embodied reasoning from pretrained knowledge. *arXiv preprint arXiv:2505.21906*, 2025. 10 - [108] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. *arXiv preprint arXiv:2504.10479*, 2025. 10## Table of Contents

1. Introduction	2
2. VisGYM	3
3. Evaluating Frontier Models with VisGYM	4
3.1. Evaluation Setup . . . . .	4
3.2. Result and Analysis . . . . .	4
4. Diagnosing Frontier Models with VisGYM	6
4.1. Turns to Keep in Conversation History . . . . .	6
4.2. Representing Observation in Text . . . . .	6
4.3. Removal of Text-based Feedback . . . . .	7
4.4. Providing Final Goal at Beginning . . . . .	7
5. Training with VisGYM	8
5.1. Supervised Fine-Tuning Experiments . . . . .	8
5.2. Stronger Base Model Generalizes Better . . . . .	8
5.3. Vision and LLM Both Matter . . . . .	9
5.4. Importance of Information-Revealing Behaviors for SFT Curation . . . . .	9
6. Related Work	10
7. Conclusion	10
A Solver Design	19
B Environment Episode Progression	21
C ASCII-based Observation Visualization	21
D VisGYM Interface	21
E Configuration of Environments	23
F. Analyzing Model Failures	24
G Additional Performance Analysis	27

## A. Solver Design This section provides detailed descriptions of the multi-step solvers introduced in Sec. 2 and used for supervised finetuning across all environments. **Colorization.** The solver computes how far the current hue and saturation are from the target, breaks those differences into small incremental steps, and outputs a sequence of rotate and saturate actions that move steadily toward the correct color. If a target number of steps is requested or if the color is already close enough, it fills the sequence with reversible rotate/saturate pairs that cancel out and don't change the final state. **Counting.** *mark\_all strategy:* The solver places a dot at the center of each target instance, then submits the correct total count and stops. *guess\_only strategy:* The solver directly submits the correct total count and stops, without placing any dots. **Jigsaw.** *reorder strategy:* The solver computes a single permutation payload that, when applied via the reorder' action, instantly rearranges the current pieces into their correct target positions. *swap strategy:* The solver generates a minimal sequence of swap' actions by repeatedly finding a misplaced piece and swapping it with the piece at its correct target location. If a target number of steps is requested, it pads this sequence with reversible pairs of swaps (e.g., swapping two pieces and then immediately swapping them back) until the desired length is reached. **Matchstick Equation.** *bfs strategy:* The solver finds the shortest possible sequence of move' actions to correct the equation using a Breadth-First Search (BFS) and then stops. *dfs strategy:* The solver finds a solution using a Depth-First Search (DFS), producing a sequence of move' actions and undo' actions that represent its full exploratory and backtracking process before stopping. *sos strategy:* The solver first finds the shortest solution path (via BFS), then pads this path by inserting random, reversible detours. Before an optimal step, it takes one or more random move' actions and immediately undo'es them, returning to the optimal path before proceeding. **Matchstick Rotation.** The solver first performs one or more translation-only move' actions, which are typically unit-length moves in the general direction of the target. It then executes a final move' action that applies the entire required rotation and corrects any remaining translation error, before stopping. **Maze 2D.** The solver uses a graph search algorithm to find the optimal coordinate path from the agent to the target, which is converted into the shortest sequence of move' actions. If a target number of steps is requested, the solver pads this optimal sequence by inserting random, reversible move' pairs (e.g., move up' followed by move down') at valid locations along the path until the desired length is met, before stopping. **Maze 3D.** The solver uses a graph search algorithm to find the optimal coordinate path from the agent's location to the target. It then converts this path into the shortest sequence of turn' (left, right, or around) and move' actions required to follow that path, accounting for the agent's current orientation. If a target number of steps is requested, the solver pads this optimal sequence by inserting random, reversible turn' pairs (e.g., turn left' followed by turn right') at locations along the path until the desired length is met, before stopping. **Mental Rotation 2D.** The solver first calculates the shortest total rotation angle required to align the current image with the target. If the requested number of steps is 1, it outputs a single rotate' action for that total angle. If a larger number of steps is requested, it stochastically divides the total rotation into that many smaller rotate' actions, which are executed sequentially and sum to the correct total angle, before stopping. **Mental Rotation 3D (CUBE).** The solver decomposes the total required rotation into its yaw, pitch, and roll components. It then corrects each component sequentially. Before applying the corrective 'rotate' action for a specific axis (e.g., yaw), it first executes a padding sequence of four 90-degree rotations around that same axis. After this 360-degree padding, it applies the single action to correct the yaw. It repeats this pad-then-correct process for the pitch and roll axes, then stops. **Mental Rotation 3D (OBJAVERSE).** The same as Mental Rotation 3D (CUBE).**MuJoCo Fetch (PICK-AND-PLACE).** The solver is a state-machine-based oracle. It follows a sequence: (0) move the gripper to a safe height above the object, (1) open the gripper, (2) descend to the object, (3) close the gripper to grasp. (4) Once grasped, it moves the object directly toward the 3D goal position using a greedy, per-axis strategy (correcting the axis with the largest error at each step). (7) Finally, it holds the object at the target location and stops. **MuJoCo Fetch (REACH).** The solver is a greedy, per-axis oracle. At each step, it identifies the single axis (x, y, or z) with the largest error between the gripper and the goal. It then outputs a ‘move’ action along that single axis to reduce the error, repeating this process until the goal is reached, at which point it stops. **Patch Reassembly.** The solver uses a backtracking search to find the optimal sequence of ‘place’ actions that perfectly tile the grid. If a target number of steps is requested, it pads this sequence by repeatedly inserting “mistake-and-correct” actions: it finds a correct ‘place’ action in the solution, finds a valid wrong location for that piece, and inserts this “mistake” ‘place’ action immediately before the “correct” ‘place’ action. If no valid mistakes can be found, it falls back to inserting a ‘remove’ and a duplicate ‘place’ action. This repeats until the desired number of ‘place’ actions is met. **Referring Dot-Pointing.** The solver first samples a random pixel from within the target object’s segmentation mask and also calculates the mask’s center of mass. It then generates a sequence of ‘mark’ actions by linearly interpolating from the random starting point to the center of mass over the requested number of steps. The final action in this sequence places a mark at the exact center of mass, which is then followed by a ‘stop’ action. **Sliding Block.** The solver uses a Breadth-First Search (BFS) to find the shortest sequence of ‘move’ actions from the current board state to the target configuration. If a target number of steps is requested, it pads this optimal path by first reconstructing all intermediate board states. At each state, it identifies all valid “back-and-forth” moves (*e.g.*, move block 1 right, then move block 1 left). It then randomly samples from these opportunities and inserts the required number of ‘move’ and ‘reverse-move’ pairs into the solution path until the desired length is met, before stopping. **Video Unshuffle.** *reorder strategy:* The solver computes a single permutation payload that, when applied via the ‘reorder’ action, instantly rearranges the shuffled frames into their correct chronological order, then stops. *swap strategy:* The solver generates a minimal sequence of ‘swap’ actions to sort the frames. It iterates through the positions, and if a frame is in the wrong place, it finds the correct frame and swaps it into its target position, repeating until all frames are sorted, then stops. **Zoom-In Puzzle.** The same as Video Unshuffle.## B. Environment Episode Progression Referenced in Tab. 2 and Sec. 2, this section presents detailed episode progressions for each environment. A summary index with page numbers is provided below:

Colorization .....	55
Counting .....	57
Jigsaw .....	59
Matchstick Equation .....	61
Matchstick Rotation .....	63
Maze 2D .....	64
Maze 3D .....	68
Mental Rotation 2D .....	73
Mental Rotation 3D (CUBE) .....	74
Mental Rotation 3D (OBJAVERSE) .....	75
MuJoCo Fetch PICK-AND-PLACE .....	76
MuJoCo Fetch REACH .....	82
Patch Reassembly .....	85
Referring Dot-Pointing .....	87
Sliding Block .....	88
Video Unshuffle .....	91
Zoom-In Puzzle .....	92

## C. ASCII-based Observation Visualization In this section, we present example episode variants rendered in text, as discussed in Sec. 4.2, for the Sliding Block, Maze 2D, Patch Reassembly, and Matchstick Equation environments in Fig. 13. Note that the instructions are slightly adapted to fit the text-based format (e.g., in the visual version of Patch Reassembly, we describe the *anchor* as “the cell that shows the patch’s ID number,” while in the text version we note that “the anchor cell for each parked patch is marked with a ‘\*’ instead of its ID number”). ## D. VisGYM Interface In this section, we present the pseudocode for the `step` function (Algorithm 1) used in VisGYM (i.e., Sec. 2). The function initializes the reward and both termination flags, then parses the model’s output string into an action name and payload. If parsing fails, it immediately returns an observation with “invalid format” as feedback. If the parsed action name is supported and its payload is valid for the corresponding action space, the function calls `Apply`, which executes the action and returns the environment feedback. Otherwise, it ends early with “invalid action” as feedback. Termination and truncation are determined inside `Apply`. If the action triggers termination (e.g., `stop`), the function computes the final reward based on the environment state. Thus, the returned reward is always zero for non-terminal transitions and the final score upon termination. Finally, the function returns the new observation, reward, termination, and truncation flags, and the feedback describing the action outcome. --- **Algorithm 1** Generic Step Function (Sec. D). Symbols: $\rho$ = reward, $\tau$ = terminated, $v$ = truncated, $\varphi$ = feedback, $\alpha$ = action name, $\pi$ = payload, $\iota$ = info. --- ``` function STEP( $a$ ) $\rho \leftarrow 0$ $(\tau, v) \leftarrow (false, false)$ Parse $a \rightarrow (\alpha, \pi)$ if invalid format then return $(obs(), 0, \tau, v, \iota("invalid format"))$ if $\alpha \in \mathcal{A}$ and $\pi \in \mathcal{A}[\alpha]$ then $(\varphi, \tau, v) \leftarrow \text{Apply}(\alpha, \pi)$ else return $(obs(), 0, \tau, v, \iota("invalid action"))$ if $\tau = true$ then $\rho \leftarrow \text{ComputeReward}()$ return $(obs(), \rho, \tau, v, \iota(\varphi))$ ``` ---Visual Rendering (default)

Target	Current
3114	3114
3114	3114
226.	5226
576.	5906
5890	7.8.

Text Representation (variant) **Sliding Block** Visual Rendering (default) Text Representation (variant) **Maze 2D** Visual Rendering (default) Text Representation (variant) **Matchstick Equation** Visual Rendering (default)

	0	1	2	3	4
0	0	0	.	.	.
1	0	0	0	0	.
2	.	.	.	.	.
3	.	.	.	.	.
4	.	.	.	.	.

--- Parked Patches --- Patch 1: \* 1 1 1 1 Patch 2: \* 22 Patch 3: \*3 3 3 Patch 4: \*4 444 44 Text Representation (variant) **Patch Reassembly** Figure 13. Visual and text representations across four environments## E. Configuration of Environments In this section, we provide the tunable parameters for each task that determine its difficulty. Note that in some environments, higher difficulty primarily increases the complexity of the input observations (e.g., Sec. 5.2), while in others it tightens the reward criteria and requires more precise control. Details are in Tab. 3. Table 3. **Configuration summary for all tasks:** tunable parameters, easy and hard configurations, and source datasets.

Task	Tunable Difficulty Parameters	Easy	Hard	Src. Dataset
Colorization	Accuracy radius $ar$ (precision required for hue and saturation match).	$ar = 11$	$ar = 16$	LLaVA Liu et al. (2023)
Counting	Minimum and maximum count range $c_{min}, c_{max}$ .	$c_{min} = 2,$ $c_{max} = 20$	$c_{min} = 5,$ $c_{max} = 30$	LVIS Gupta et al. (2019)
Jigsaw	number of rows and columns $nr, nc$ .	$nr = 2, nc = 2$	$nr = 3, nc = 3$	LLaVA Liu et al. (2023)
Matchstick Equation	Number of break moves $bm$ (corruptions to fix).	$bm = 1$	$bm = 2$	—
Matchstick Rotation	Hidden scale range $sr$ , position tolerance $pt$ , angular tolerance $at$ .	$pt = 10,$ $at = 15$	$pt = 5,$ $at = 10$	—
Maze 2D	Maze width and height $mw, mh$ .	$mw = 9, mh = 9$	$mw = 11, mh = 11$	—
Maze 3D	Maze width and height $mw, mh$ .	$mw = 7, mh = 7$	$mw = 9, mh = 9$	—
Mental Rotation 2D	Angular tolerance $at$ .	$at = 10.0$	$at = 5.0$	LLaVA Liu et al. (2023)
Mental Rotation 3D (CUBE)	Number of segments $ns$ , length range $lr$ , angular tolerance $at$ .	$ns = 4$	$ns = 6$	—
Mental Rotation 3D (OBJAVERSE)	Angular tolerance $at$ .	$at = 15.0$	$at = 5.0$	Objaverse Deitke et al. (2023)
MuJoCo Fetch PICK-AND-PLACE	No user-tuned difficulty parameters (standardized task).	Standard	Standard	—
MuJoCo Fetch REACH	No user-tuned difficulty parameters (standardized task).	Standard	Standard	—
Patch Reassembly	Grid size $gs$ , number of patches $np$ .	$gs = (6, 6),$ $np = 5$	$gs = (8, 8),$ $np = 6$	—
Referring Dot-POINTING	No user-tuned difficulty parameters (standardized task).	Standard	Standard	RefCOCO Kazemzadeh et al. (2014)
Sliding Block	Number of shuffle moves $sm$ .	$sm = 30$	$sm = 90$	—
Video Unshuffle	Number of frames $nf$ , sampling strategy $ss$ , minimum frame-diff threshold $mfd$ .	$nf = 4$	$nf = 5$	SS2 Goyal et al. (2017)
Zoom-In Puzzle	Zoom gap $zg$ , zoom variability $zs$ , minimum zoom $mz$ , num. of views $zv$ , nested crop $nest$ .	$zv = 4$	$zv = 5$	LLaVA Liu et al. (2023)

Table 4. Example clusters discovered by StringSight.

Cluster Description
MuJoCo Fetch (Pick-and-Place) The model issues repetitive movement commands without adapting based on task progress or environment feedback, resulting in oscillatory behaviors like moving up and down, left and right, or in a single direction with no meaningful progress toward grasping or placing the cube. For example, it may move forward repeatedly even after overshooting the target or oscillate without ever attempting a grasp. Ignores visual feedback from images and does not adjust its actions in response to clear cues about the state or position of the cube, gripper, or target marker. This results in the model issuing irrelevant or counterproductive actions, such as continuing to move when the cube has not been grasped. Issues the “stop” command prematurely before the end-effector is even close to the target marker, ending the task early without justification or clear success criteria.
Mental Rotation 3D (Cube) The model repeatedly issues the same rotation action, often on a single axis, for many steps without adapting based on visual feedback, resulting in rigid loops or unbroken patterns. The model oscillates between a small set of orientations, alternating or repeating similar rotations (e.g., $+45^\circ$ , $-45^\circ$ ), causing the object to cycle endlessly without making progress toward the target alignment.
Zoom-In Puzzle The model finalizes the arrangement immediately without performing any swaps, reordering, or verification, accepting the initial sequence as correct regardless of its accuracy. For example, it issues a “stop” command on the first step even if the order is incorrect.
Matchstick Rotation The model issues fixed or monotonically decreasing movement and rotation magnitudes without attempting to estimate the unknown scale or using exploratory actions to resolve scale ambiguity, proceeding as if the appropriate step size is already known. Action sequences do not adapt based on feedback or observed outcomes; the model follows a predetermined or repetitive strategy without checking if moves are effective or responding to evidence from the environment.
Maze 2D The model repeatedly issues the same invalid movement commands, such as trying to move into walls, even after receiving explicit feedback that these actions are not possible. For example, it continues to try moving left into a wall after being told each time that the move is blocked. Does not build, update, or use any internal map or memory of previous moves or environmental feedback, resulting in repeated visits to the same locations, blocked paths, and inefficient looping navigation.
Maze 3D Movement decisions are based exclusively on immediate sensory feedback, without building or referencing internal memory or a map of previously explored locations, leading to repeated wall collisions, revisiting dead ends, and inefficient navigation. Frequently issues long sequences of consecutive turning actions—such as alternating left and right—without forward movement or meaningful progress toward the goal, resulting in wasted steps and inefficient pathfinding.

## F. Analyzing Model Failures We run StringSight [Dunlap et al. $2025$](#); [Lisa Dunlap et al. $2025$](#), a pipeline for automatically uncovering failure cases and comparing models. It uses a VLM annotator (GPT-4.1) to extract behaviors from each trace (e.g., “uses `move (1, 1)` for all 20 steps”) and clusters these behaviors into higher-level patterns (e.g., “repeats the same action”). Examples of discovered cluster descriptions are shown in Table 4. We then manually examine the top failure cases for each task and identify four common failure modes across all tasks. (1) *Restricted action space and action looping*: models often rely on a single repeated operation or fixed-magnitude action, such as continually moving in the same direction in Fetch Pick & Place, using “swap” in Jigsaw instead of “reorder”, or rotating by the same angle in Mental Rotation 3D and Match Rotation rather than converging to an optimal magnitude. (2) *State mismanagement*: models fail to maintain or update internal state across steps. They ignore textual or### System Prompt: Labeling failure modes in traces You are an expert model behavior analyst. Your task is to meticulously analyze the trace of a large language model to identify whether it contains any of the following behaviors: - ● **Restricted action space and action looping:** The model keeps repeating the same or nearly identical action without making progress. Look for consecutive turns with the same command or sequence of commands, movements by the same amount, or the same tool being used even when it is ineffective. - ● **State mismanagement:** The model forgets or ignores what it already learned in earlier steps. It may revisit old states, contradict past reasoning, or repeat mistakes it was corrected for (e.g. being told it hit a wall and then continuing to move forward in the same direction). Do not include if this is simply action looping, where the model is repeating the same action without making progress; this is specifically when the model is ignoring feedback or not adjusting its behavior based on its previous actions, but is still issuing different commands. - ● **Early termination:** The model stops too early. Early means terminating before the maximum number of steps is reached. - ● **Failure to use visual or spatial information:** The model ignores visible or spatial cues. This applies to traces where either an image or ASCII art is provided, and the model does not react to changes in the scene. For example, if the object leaves the frame, but the model continues to move towards it. Look for actions that contradict what's visually or spatially clear. Do not include if the model is simply action looping, where the model is repeating the same action without making progress; this is specifically when the model is not utilizing the visual information when it is available. If the trace does not provide visual information (either images or ASCII), do not include this label. If the trace contains any of the behaviors, return a list of objects with the following structure. If a trace has more than one behavior, return a list of objects with the structure below for each behavior. If the trace contains none of the behaviors, return an empty list. #### JSON Output Structure ``` [ { "property_description": "which behavior is present in the trace", "reason": "an explanation of the exact behaviors in the trace that fall under the property_description (1-2 sentences)", "evidence": "What exactly in the trace exhibits this property? Include quotes/tool calls/actions when possible." } ] ``` environmental feedback, revisit previously explored areas, or repeat illegal actions despite prior errors—for example, continuing to move into a wall after being told they have collided, or repeating invalid moves in the Match Equation, Sliding Block, and Toy Maze 2D tasks. (3) *Early termination:* the model terminates the episode before the maximum steps, despite not reaching the goal. (4) *Failure to use visual or spatial information:* models ignore the visual information provided, such as the target leaving the frame or the item being successfully aligned (e.g., Mental Rotation). Finally, we quantify the prevalence of each failure mode by having a VLM annotator (GPT-4.1) label each trace for these behaviors (a trace may exhibit multiple behaviors).**Frequency of failures.** Figure 14 shows the proportion of traces that contain each failure. We see that action looping is very common, occurring in more than 60% of traces, followed in frequency by early termination, state mismanagement, and failure to use visual or spatial information. Looking at how the frequency of the behaviors changes compared across tasks, we see in Figure 15 (b) that certain tasks, like Matchstick Equation and Sliding Block, result in a particularly large amount of action repetition and state mismanagement failures, likely due to the difficulty of the task and the frequency of invalid moves. We additionally see that tasks like the Maze task, which provide clear visual signals of task progress, have a very high (up to 70%) rate of ignoring this important visual information and high action repetition. Based on this information, we see that often when a model is uncertain, it defaults to repeating its previous moves, regardless of the visual or language feedback it is given from previous turns. This is further supported in Figure 15 (a), which shows that weaker models like UI TARS 1.5 7B have very high rates of action looping (87%) and state mismanagement (35%). Figure 14. Frequency of failure patterns. (a) Frequency of failure patterns per model. (b) Frequency of failure patterns per task on easy variants. Figure 15. Detailed Analysis of Failure Patterns by Model and Task. We additionally find interesting cases of early termination, such as giving up on the task entirely, where the model says things like “I give up.” and “I’m stopping. This is unsolvable”. These specific instances of giving up happen much more often for hard tasks like Matchstick Equation, indicating that the models’ limited task comprehension leads them to question whether a solution exists in the current instance. We also see this phenomenon occur more often in Gemini and Gemma models, which we suspect is because these models are chattier and more anthropomorphic and thus may express their internal reasoning more often than others. ## F.1. Failure changes per ablation To examine the effects of our ablations on model behavior, we run the failure labeling pipeline above on the ablations described in Section 4 and show the comparison in Figure 16. We find the following: *Different amounts of chat history (Figure 16a):* As more history is given, the model is less likely to repeat immediate actions, but still suffers from state mismanagement. We suspect the decreased action looping occurs because the model has a default action (e.g., moving left), so with no history, it continues to repeat this move. With history, it is less likely to immediately repeat prior actions, but after a certain amount of context, the model struggles to manage earlier state and reverts to its default behavior. This is reflected by action looping decreasing when full history is given, consistent with decreased performance under full history. *Feedback vs. no feedback (Figure 16b):* When no feedback is provided, the model is less likely to terminate. Inspecting these traces shows that this is largely due to a reduction in “giving up,” since the model often gives up when told its moves are invalid. We also observe decreases in action looping and state mismanagement, which is surprising given that overall performance decreases without feedback. This suggests the presence of additional failure modes not captured by our taxonomy, which we leave for future work. *Ground truth state given at the beginning (Figure 16c):* When given the ground truth state at the start of the task, the model is less likely to “guess” or give up early, reflected in lower rates of action looping and early termination.Figure 16. **Failure-pattern frequency under different information settings.** Due to cost, images were only analyzed in the original split, thus the “failure to use visual or spatial information” case is removed in all but the text vs image representation ablation (d). *Image vs. text representation (Figure 16d):* For tasks aside from Matchstick Equation, models process visual information more effectively when it is presented as text rather than an image. The large reduction in action looping suggests that this text-based representation provides clearer guidance for selecting actions. ## F.2. Failure Trajectories Visualization Using StringSight, we visualize the trajectory for each failure type. In each trajectory, we show the prompt, the image, the models’ raw output, and the action parsed from the raw output. We also show the output from StringSight for each trajectory, tagged “Reason” and “Evidence” at the top, where “Reason” stands for StringSight’s reason for classifying this trajectory into a specific failure category, and “Evidence” stands for the evidence in the trajectory that leads to the conclusion. (1) *Restricted action space and action looping:* As in Sec. F.2.1, we show a case of action looping of GPT-5 on the Jigsaw task. The model repeatedly takes the same action “(“swap”, (0, 0), (0, 1))”, resulting in looping behaviors without making any progress. (2) *State mismanagement:* As in Sec. F.2.2, we show a case of Claude Sonnet 4 on Maze 2D. In Observation 7, the model takes action “(“move”, 2)”, which leads to the environment feedback “Cannot move into a wall.” However, at Observation 16, the model is in the exact same state, disregards the previous feedback, and takes the same action “(“move”, 2)” again. (3) *Early termination:* We show in Sec. F.2.3 a case of Gemma 3 27B Instruct on Matchstick Equation, where the model decides to give up and terminate at step 13, while the model is allowed to take 30 steps in total. (4) *Failure to use visual or spatial information:* As in Sec. F.2.4, we show a case of Gemini 2.5 Pro on Mental Rotation 3D (Cube). In the last three steps, after rotating in the wrong direction, the model does not take the visual information into account and continues to rotate in the same direction, which moves the object even farther away from the target position. ## G. Additional Performance Analysis **Difficulty of Each Task.** In Fig. 17, we compute the average accuracy across models for each task and sort tasks from easiest to hardest based on these averages. In general, we found that Referring Dot-Pointing and Counting are the easiest, with models achieving over 20% accuracy on average, whereas Mental Rotation 3D (Cube), Patch Reassembly, and Mental Rotation 3D (Objaverse) are the hardest with an accuracy around 1%. Sliding Block, Maze 3D, Fetch Pick-Place, and Video Unshuffle also pose significant challenges for the models, with less than 5% on average. This suggests that tasks requiring memory and long-horizon planning,Figure 17. **Average success rate across frontier models on each task.** The easiest tasks are Referring Dot-Pointing, and Counting, with over 20% accuracy on average across all models, while the hardest tasks are Mental Rotation 3D (Cube), Patch Reassembly, and Mental Rotation, with the average accuracy less than 2% on average. Figure 18. **The Number of Steps each Model Takes Over all Tasks.** We calculate the number of steps over all trajectories for each model and visualize the correct trajectories (green) and the incorrect trajectories (red). or strong 3D spatial understanding, remain the most difficult for current models. **Number of Steps.** In Fig. 18, we calculate the number of steps taken on all trajectories for each model and calculate the number of correct trajectories (green) and the number of incorrect trajectories (red). There is a clear cutoff on steps 20 (maximum steps allowed for Easy setting) and 30 (maximum steps allowed for Hard setting), indicating that all models tend to reach the maximum number of steps. We also observed a “U-shaped” trend over the steps for all models, where they tend to either terminate early or continue until the final step. **Easy to Hard Performance Drop.** In Fig. 19, we calculate the average accuracy in Easy and Hard, respectively, on all models, and then visualize the performance gap between easy and hard on each task. The biggest Easy to Hard performance drops occur on Counting and Jigsaw. For Counting, accuracy drops sharply as the number of objects increases. For Jigsaw, performance drops to near zero as the puzzle changes from 2x2 to 3x3, suggesting that this task can be further scaled to even more difficult $n \times n$ configurations. For some tasks (*e.g.*, Patch Reassembly, Sliding Block, Video Unshuffle), the absolute gap is smaller, likely because Easy performance is already very low ( $\approx 0$ ). These Figure 19. **Easy → Hard Performance Drop.** For each task, we calculate the average accuracy on Easy and Hard, respectively, over all models, and then visualize the performance drop between Easy and Hard.Figure 20. **Model Rankings Per Task.** We rank all the models on each task and show the ranking in the table. tasks are also naturally scalable in terms of difficulty. For example, increasing the number of patches for Patch Reassembly, the number of blocks for Sliding Block, or the number of frames for Video Unshuffle. As VLMs improve and begin to reliably solve the Easy settings, we expect to see larger easy-to-hard gaps on these tasks, and that our gym can be correspondingly scaled to provide harder task variants. **Model Rankings.** In Fig. 20, we plot each model’s ranking on every task. When multiple models tie on a task, we assign them the worse (higher) numerical rank. For example, if two models are tied for the best score, we label both as rank 2, so that ties do not overstate how clearly a model is separated from others. While these mostly align with the global leaderboard, they also reveal clear task-specific strengths and weaknesses. For example, on specialization, Qwen2.5 VL 72B Instruct performs well on Counting (rank 2), Gemma 3 27B Instruct performs well on Maze 3D (rank 3), and Qwen3 VL 235B Instruct performs well on Referring Dot-Pointing. On weakness, despite strong average performance, GPT-5 performs poorly on Video Unshuffle and Maze 3D, while Gemini 2.5 Pro performs poorly on Jigsaw (Hard) and Patch Reassembly.## F.2.1 Sample trajectory for “Action Looping” (GPT-5) [$back to Sec. F.2$](#) ### Reason The model repeatedly issues the same swap command between the same two pieces without making any progress towards solving the puzzle, as evidenced by identical actions over multiple consecutive steps. ### Evidence "('swap', ((0, 0), (0, 1)))" is issued in steps 1, 2, 3, 4, 5, and 6, with no change in strategy or progress. ### Instruction I You are solving a 2x2 jigsaw puzzle. The puzzle pieces are currently scrambled. Your goal is to rearrange the pieces to recover the image. Available actions: 1. 1. 'swap': Swap two pieces by specifying their coordinates. Format: '('swap', ((row1, col1), (row2, col2)))' where coordinates start from (0,0) at the top-left corner. 2. 2. 'reorder': Reorder all pieces at once. Format: '('reorder', [0, 1, 2, ..., 3])' where the list represents the desired order of pieces from top-left to bottom-right. 3. 3. 'stop': End the puzzle solving session. Format: '('stop', 'stop')' Please respond with exactly one action and its arguments in the specified format. For example: - - To swap two pieces: '('swap', ((0, 0), (1, 1)))' - - To reorder all pieces: '('reorder', [0, 1, 2, ..., 3])' - - To stop: '('stop', 'stop')' Index-to-cell mapping (0-based rows/cols): - - Index = row \* 2 + col. - - Row 0 covers indices 0..1, row 1 covers 2..3, etc. Example for 2x2: (0,0)->0, (0,1)->1, (1,0)->2, (1,1)->3. This is step 1. You are allowed to take 19 more steps. ### Observation $o_0$ ### Action $a_1$ : ('swap', ((0, 0), (0, 1))) ### VLM Raw Output: ('swap', ((0, 0), (0, 1)))