Can AI Learn Complex Skills Just by Watching?

Imagine teaching a child to tie shoelaces not by hand-holding through steps, but by showing a video of an expert doing it repeatedly. What if AI models could learn similarly? This is the core idea behind a groundbreaking technique called Model See, Model Do (MSMD), which enables vision-language models (VLMs) to imitate expert behaviors captured in screen recordings. Unlike traditional methods requiring extensive paired data, MSMD leverages short video clips of specialist AI agents performing tasks, allowing generalist models to acquire new skills efficiently.

In this exploration, we'll dive into the challenges of imitation learning, unpack how MSMD works step-by-step, review its impressive results on real-world benchmarks, and discuss its potential to transform AI agent development. By the end, you'll see why this approach is a game-changer for building versatile AI systems.

What Challenges Does Traditional Imitation Learning Face?

Imitation learning has long promised to bridge the gap between human demonstrations and AI proficiency. In classical setups, like behavioral cloning, a model learns a policy directly from state-action pairs collected from an expert. However, scaling this to complex, long-horizon tasks—such as navigating websites or controlling mobile apps—runs into several hurdles:

• Data Scarcity: Collecting high-quality, expert-level trajectories for diverse environments is labor-intensive and expensive.
• Partial Observability: Real-world interfaces (e.g., screens) provide rich visual information that tabular state representations miss.
• Long Sequences: Tasks spanning dozens of steps amplify compounding errors in autoregressive imitation.
• Specialization Gap: Generalist models struggle to match niche experts without fine-tuning on massive datasets.

Behavioral cloning on text or images often fails here, as it ignores temporal dynamics. Video-based imitation, inspired by how humans learn from YouTube tutorials, offers a natural solution—but processing videos naively is computationally prohibitive.

Enter MSMD, which distills video demonstrations into compact textual plans, making imitation feasible for off-the-shelf VLMs.

How Does Model See, Model Do Work?

MSMD operates in a streamlined pipeline that turns screen videos of expert agents into imitable instructions for VLMs. Here's the breakdown:

Step 1: Record Expert Demonstrations

Specialist models—pre-trained agents excelling in domains like web browsing or Android control—perform tasks on screen. For instance:

• A web agent surfs OSWorld environments.
• An Android agent interacts via AndroidWorld.

These sessions are captured as videos, typically 1-3 minutes long, at 2 FPS for efficiency.

Step 2: Generate Structured Plans with a VLM

A powerful VLM like GPT-4V processes the full video and outputs a structured plan in JSON format. This plan includes:

• High-level subgoals: E.g., "Navigate to the shopping cart."
• Action descriptions: Natural language summaries of each step.
• Visual grounding: References to specific screen elements.

Example output for a web navigation task:

{
  "subgoals": [
    "Locate the search bar",
    "Enter query 'laptop'",
    "Click first result"
  ],
  "actions": [
    "Move cursor to top-center input field",
    "Type 'laptop' and press Enter",
    "Hover and click on product link"
  ]
}

This condensation reduces a 100+ frame video to a handful of actionable insights, preserving intent while discarding noise.

Step 3: Imitate via Textual Rollouts

During inference, the imitating VLM (e.g., GPT-4o) receives the current screen state plus the structured plan. It generates actions autoregressively:

1. Predict the next high-level action based on progress.
2. Describe low-level mouse/keyboard commands.
3. Execute via an action tokenizer (e.g., converting "click button" to coordinates).

Crucially, no video processing occurs at test time—only text and screenshots—enabling real-time performance.

Key Innovations

• Plan Distillation: Videos → JSON plans act as a lightweight "expert memory."
• Hierarchical Reasoning: Subgoals guide long-term planning, mitigating error accumulation.
• Zero-Shot Transfer: Works across domains without retraining the VLM.

The researchers released their code and interactive demos, letting you experiment with web and Android tasks yourself.

What Do the Experiments Reveal?

MSMD was rigorously tested on established benchmarks emphasizing visual interfaces and extended interactions:

OSWorld (Web Navigation)

• Setup: 12 everyday tasks (e.g., "Book a flight") in browser environments.
• Baselines: SeeAct (state-of-the-art VLM agent), smaller VLMs.

MSMD nearly doubles SeeAct's performance, succeeding where others fail on multi-step reasoning.

AndroidWorld (Mobile Control)

• Setup: 10 mobile tasks (e.g., "Schedule a meeting") using realistic Android screenshots.
• Results: MSMD achieves 28.2% success vs. 14.7% for direct VLM prompting—doubling efficacy.

Ablations: Why Structured Plans Matter

• Without subgoals: Drops 15-20%.
• Text-only (no video distilation): Halves gains.
• Shorter videos: Still effective, showing efficiency.

These gains stem from MSMD's ability to encode expert foresight into reusable plans, transferable even to unseen tasks.

Real-World Applications and Examples

Example 1: E-Commerce Automation

Prompt GPT-4o with an MSMD plan from an expert shopping agent:

• Screen: Amazon homepage.
• Plan: ["Search 'wireless headphones'", "Filter by price < $50", "Add to cart"].

The model outputs precise actions: "Click search icon at (x:100, y:50), type query, hit enter." This scales to custom e-commerce bots without per-task training.

Example 2: Mobile App Testing

For AndroidWorld, an MSMD-trained VLM automates QA:

• Task: "Set alarm for 7 AM."
• Actions: Tap clock app → Select time → Confirm.

Success rate jumps from random poking (5%) to reliable execution (30+%).

Broader Impacts

• Agent Tooling: Integrate MSMD into frameworks like LangChain for hybrid specialist-generalist teams.
• Data Efficiency: Democratizes expertise—record once, imitate forever.
• Multimodal Expansion: Extend to robotics by swapping screens for camera feeds.

Limitations? VLMs occasionally hallucinate actions, and plans assume expert perfection. Future work could incorporate self-correction loops.

Why Should Developers Care? Actionable Next Steps

1. Try the Demos: Visit modelsee-modeldo.github.io to see MSMD in action.
2. Fork the Repo: Clone https://github.com/Khaled-2023/MSMD and adapt for your domain.
3. Build Your Own: Record your specialist agent's sessions, distill with GPT-4V, and bootstrap generalists.
4. Benchmark Locally: Use OSWorld setups to measure gains on custom tasks.

Code snippet to get started:

import openai

def distill_plan(video_path):
    response = openai.ChatCompletion.create(
        model="gpt-4-vision-preview",
        messages=[{"role": "user", "content": [
            {"type": "text", "text": "Extract structured plan from this expert demo."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{video_base64}"}}
        ]}]
    )
    return response.choices[0].message.content  # JSON plan

# Imitate
 def imitate_step(screen_img, plan_json):
     # Prompt VLM with screen + plan
     pass

The Future of Imitative AI

MSMD exemplifies how VLMs can evolve from passive observers to active imitators, blurring lines between narrow experts and general intelligence. As video data proliferates, expect this paradigm to underpin next-gen agents in software testing, UI automation, and beyond. Researchers from Stanford, UC Berkeley, and Mila pioneered this, proving that sometimes, seeing is indeed doing—for machines too.

(Word count: 1,248)