Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

Analysis

Why This Paper Matters

Computer-use agents today often re-derive routines that users have already performed, wasting inference tokens and time. This paper addresses a critical gap: agent memory typically records what users say, not what they do. By compiling passively captured screen activity into structured, deterministic frames, the authors offer a memory system that is both efficient and auditable, directly tackling the cost and reliability challenges in agent deployment.

The significance extends beyond memory. The paper introduces a novel use of the compiler as a demand-side cost instrument, measuring parameters that agent cost models have long assumed but never empirically grounded. This could reshape how we estimate and optimize token budgets for fleets of agents, making large-scale deployment more predictable and affordable.

Technical Contributions

• Deterministic, zero-model compilation: The pipeline segments raw screen capture into typed activity frames without any model in the loop, ensuring byte-identical, cacheable, and mechanically auditable outputs.
• Typed activity frames: Each frame is a bounded episode with application, site, timing, input volume, and evidence pointers to raw rows, enabling precise replay and analysis.
• Cost parameter measurement: The compiler reads passive human activity to derive Routine Overhead Ratio (R) and routine recurrence (h), providing first empirical values for these cost-model inputs.
• Zero-token replay: Compiled routines can be replayed deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit.
• Open-source artifacts: Schema, compiler, and evaluation harness are released, facilitating reproducibility and further research.

Results

The compiler processes a day of raw capture into a context block 86x smaller in just 68 ms, a dramatic efficiency gain. In question-answering tasks, an agent reading the compiled block achieves 98.4% accuracy (Wilson 95% CI 91.7-99.7%), outperforming LLM summaries of the same capture (66-80%). Notably, a mid-tier model reading the block matches a frontier model, suggesting that the structured representation reduces the need for powerful reasoning. The measured Routine Overhead Ratio R ranges from 60x to 343x, and routine recurrence h is 9.0% in-sample and 7.7% out-of-sample, leading to an all-fleet token ceiling near 8%—a realistic bound for delegable routines.

Significance

This work introduces a practical, deterministic memory layer for computer-use agents, potentially reducing inference costs and improving reliability. By providing empirical cost parameters, it enables more accurate budgeting for agent fleets, which is crucial for commercial deployment. The open-source release encourages adoption and further innovation, possibly leading to standardized activity frame formats. The approach also raises interesting questions about privacy and the trade-offs between passive capture and active logging, which future work must address. Overall, this paper is a step toward more efficient, auditable, and cost-effective autonomous agents.