Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

Analysis

Why This Paper Matters

Spatial intelligence is a critical frontier for AI agents moving beyond static semantic understanding to physical-world interaction. Existing spatial reasoning benchmarks rely on coordinates, options, or text, creating an answer-interface mismatch for image-generation models that naturally express spatial judgments in pixel space. This paper addresses a fundamental gap: how to evaluate image-generation models under the same task semantics as text-output VLMs. By proposing ProVisE and SpatialGen-Bench, the authors provide a framework that bridges this divide, enabling fair comparison and revealing complementary strengths. This matters because it opens the door for image-generation models to be assessed on spatial cognition tasks they are inherently suited for, potentially accelerating progress in embodied AI and robotics.

Technical Contributions

• ProVisE Framework: A benchmark-agnostic system that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics. This allows reuse of existing benchmarks without redesign.
• Agentic Builder: An automated component that constructs and validates task-specific protocols for new benchmarks, reducing manual effort and ensuring consistency.
• SpatialGen-Bench: A curated diagnostic benchmark of 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms, designed to probe spatial cognition comprehensively.
• Unified Evaluation: The framework enables direct comparison between text-output VLMs and image-generation models on the same spatial tasks, revealing that image-generation models excel when answers can be expressed in pixel space, while VLMs dominate compositional reasoning.

Results

The paper evaluates representative text-output VLMs and image-generation models in a unified setting. Key findings: image-generation models are competitive when spatial answers can be externalized directly in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. The Agentic protocol construction is validated on six external spatial benchmarks, demonstrating generalizability. Concrete metrics are not provided in the abstract, but the results establish complementary strengths of pixel-space expression and text-based reasoning.

Significance

This work has broad impact on AI evaluation methodology, particularly for vision-language and embodied AI. By providing a metric-compatible testbed for studying spatial cognition in image-generation models, it enables fair comparison and highlights the unique value of pixel-space reasoning. This could influence future benchmark design, model development, and deployment in applications like robotics, autonomous navigation, and interactive AI systems where spatial understanding is paramount.