Preprint
Large Language Models

Gaze Target Estimation Anywhere with Concepts

Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg
August 11, 2026
arXivPDF

0

Citations

0

Influential Citations

—

Venue

2026

Year

Abstract

Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.

Analysis

Why This Paper Matters

Gaze target estimation is critical for understanding human attention and intention, with applications in human-robot interaction, autonomous driving, and clinical assessment. Traditional methods rely on multi-stage pipelines that first detect the subject (e.g., head bounding boxes, pose) and then estimate gaze, which are brittle and error-prone. This paper introduces a paradigm shift by making gaze estimation promptable, allowing users to specify the subject via natural language or visual points, similar to recent advances in segmentation and detection. This flexibility not only simplifies the task but also improves scalability and user experience.

The introduction of the PGE task and the Gaze-Co dataset addresses a significant gap: the lack of prompt-annotated gaze data. By providing a large-scale dataset and a strong baseline model, the paper enables future research in this new direction. The open-source release further accelerates adoption and reproducibility.

Technical Contributions

  • New Task Formulation: PGE integrates subject localization and gaze estimation into a single end-to-end framework, eliminating the need for explicit intermediate inputs like head boxes or pose.
  • Promptable Conditioning: The model accepts flexible text prompts (e.g., "the boy in the red shirt") or visual point prompts (e.g., coordinates), making gaze analysis more intuitive and adaptable.
  • Gaze-Co Dataset: A scalable data engine generates 120K high-quality, prompt-annotated image pairs, providing a benchmark for training and evaluation.
  • GazeAnywhere Architecture: A transformer-based detector fuses features from frozen vision and language encoders, jointly predicting subject localization, in/out-of-frame presence, and gaze heatmaps. This design leverages pre-trained models while remaining efficient.

Results

The paper reports state-of-the-art performance on multiple PGE benchmarks. Notably, GazeAnywhere sets a strong baseline for the new task, even on a difficult out-of-domain, real-world clinical dataset, demonstrating its generalization capability. While specific numeric metrics are not detailed in the abstract, the consistent SOTA results across benchmarks indicate the effectiveness of the approach.

Significance

This work aligns with the broader trend of making vision tasks more interactive and language-driven, as seen in models like CLIP and SAM. By enabling promptable gaze estimation, it opens new possibilities for human-AI collaboration, where users can query gaze behavior in natural language. The dataset and model provide a foundation for future research in attention modeling, social robotics, and assistive technologies. The open-source release ensures that the community can build upon this work, potentially leading to more robust and flexible gaze analysis systems.

gaze-estimationpromptingvision-languagetransformerdatasetend-to-endhuman-ai-interaction
TransformersPromptingBenchmarks

Related Papers