Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation

Analysis

Why This Paper Matters

Artist-grounded text-to-image generation is a challenging task that goes beyond simply appending an artist's name to a prompt. Current image models often respond to artist names through canonical shortcuts—recurring motifs, generic palettes, or overrepresented period signatures—rather than preserving the user's intended scene. This paper identifies this shortcut problem as a core bottleneck and introduces Atelier, a framework that explicitly models artistic intent as a control state. This is significant because it reframes the problem: instead of relying on the model's implicit understanding of an artist, Atelier makes the artistic controls explicit and evidence-grounded, leading to more faithful and controllable generation.

The introduction of ArtIntentBench is also a major contribution. The benchmark covers two artists (Van Gogh and Qi Baishi) across multiple tasks, including artwork re-rendering, period/style-controlled generation, historically unseen subjects, shortcut auditing, and human preference evaluation. This provides a standardized way to evaluate artist-grounded generation, which was previously lacking. The benchmark's focus on shortcut auditing is particularly novel, as it directly measures the tendency of models to fall back on canonical shortcuts.

Technical Contributions

• Control-State Planning: Atelier translates underspecified artistic intent into an explicit control state that separates scene anchors, preserve/transform decisions, style-regime hypotheses, role-bound artist evidence, and shortcut-avoidance constraints. This structured representation allows for more precise control over generation.
• Grounding with Artist-Level Knowledge: The framework grounds the control state using artist-level knowledge and local patch references, ensuring that the generated image is not only stylistically accurate but also structurally faithful to the source.
• Backend-Aware Generation Plans: Atelier compiles backend-aware generation plans, meaning it can adapt its approach based on the specific text-to-image generator being used (open-weight or closed-source).
• Iterative Refinement: The framework iteratively refines candidates through global and local authenticity feedback, allowing for continuous improvement of the generated image.
• ArtIntentBench: A comprehensive benchmark that includes multiple tasks and human preference evaluation, providing a robust testbed for artist-grounded generation.

Results

The paper reports that Atelier improves artist-level style fidelity, preserves source structure more faithfully, and substantially reduces shortcut substitution compared with prompt-engineered, retrieval-augmented, and general-purpose agent baselines. These improvements are observed across both open-weight and closed-source generators, indicating the framework's versatility. While specific numerical metrics are not provided in the abstract, the qualitative claims are strong and suggest that Atelier outperforms existing methods in key areas.

Significance

The broader impact of this work lies in its reframing of artist-grounded generation as an inference problem rather than a synthesis problem. By emphasizing the importance of explicit, evidence-grounded artistic controls, Atelier paves the way for more interpretable and controllable generative models. This could influence future research in text-to-image generation, particularly in areas where style and content need to be carefully balanced. The introduction of ArtIntentBench also provides a valuable resource for the community, enabling more rigorous evaluation and comparison of future methods. Overall, this paper highlights the need for planning and reasoning in generative AI, moving beyond simple prompt engineering.