prompt
FreeVision-language system prompt for desktop GUI automation
FreeFree tier
About prompt
This is a system prompt for the UI-TARS Desktop Agent Operator, an open-source multimodal GUI agent built by ByteDance. The prompt instructs a vision-language model to control a computer desktop by analyzing screenshots and emitting structured actions. It emphasizes a 'look before acting' principle, one action per reasoning cycle, and precise coordinate usage. The prompt includes a defined action vocabulary (click, type, scroll, etc.) and operating principles for reliable GUI automation. It is found in the ai-boost/awesome-prompts repository.
Key Features
Vision-language driven: operates by analyzing screenshots
Structured action vocabulary including click, type, scroll, and drag
Look-before-acting principle to identify UI elements before interaction
One action per reasoning cycle for reliability
Supports GUI, browser-DOM, and hybrid modes
Plan-then-execute workflow for multi-step tasks
Uses OCR text, cursor position, and action history as input
Coordinates can be bounding-box or normalized coordinates
Pros & Cons
Pros
- Open-source and free to use
- Multimodal (visual + text) understanding
- Precise coordinate-based actions reduce errors
- Clear operating principles for consistent behavior
- Supports multiple interaction modes (GUI, DOM, hybrid)
- Includes fallback strategies for ambiguous screens
Cons
- Requires a vision-language model with high accuracy
- Cannot access DOM in pure GUI mode (no direct element references)
- Relies on screenshot quality and latency for responsiveness
- Not suitable for tasks requiring real-time or low-latency interaction
Best For
Desktop GUI automation and task executionBrowser automation via visual observationControlling applications without API accessAutomated software testing across platformsAccessibility assistance through visual understanding
FAQ
What is the UI-TARS Desktop Agent Operator?
It is an open-source multimodal GUI agent stack by ByteDance, designed to control a computer by analyzing screenshots and emitting structured actions, as described in this system prompt.
What input does this prompt expect?
The prompt expects screenshots of the desktop, browser, or application windows, plus optional OCR text, cursor position, and previous action history.
What actions can the agent perform?
The agent can perform click, double_click, right_click, type text, press keys, scroll, move cursor, drag, request a screenshot, wait, and report task completion or failure.
Is this prompt free to use?
Yes, it is open-source under the Apache-2.0 license and available for free in the ai-boost/awesome-prompts GitHub repository.
What is the source of this prompt?
The prompt is derived from ByteDance's UI-TARS-desktop project (github.com/bytedance/UI-TARS-desktop).