Sample-Efficient Learning from Agent Experience

Analysis

Why This Paper Matters

Real-world agent learning is often bottlenecked by the high cost of environment interactions—whether running lengthy experiments or obtaining human feedback. This paper tackles a critical gap: while in-context learning (ICL) is highly sample-efficient, its benefits vanish once the experience is removed from the context. The authors propose Experience Distillation, a method that internalizes ICL gains into model weights without requiring additional environment samples. This is significant because it bridges the gap between the sample efficiency of ICL and the permanence of weight-based learning, offering a practical solution for deploying agents in cost-sensitive domains.

The paper's focus on software-engineering tasks and text-adventure games demonstrates broad applicability. By showing that Experience Distillation retains over 64% of ICL gains while supervised fine-tuning recovers only 3.8%, the authors highlight the inadequacy of naive fine-tuning and the promise of their approach. Furthermore, matching classical RL baselines with at least 9.6× fewer samples underscores the potential for substantial cost savings in real-world applications.

Technical Contributions

• Experience Distillation: A novel application of context distillation to agent interaction histories, converting ICL signals into model weights without further environment interaction.
• Empirical validation: Extensive experiments on 749 software-engineering tasks and six text-adventure games, providing robust evidence of effectiveness.
• Comparison with baselines: Demonstrates that direct supervised fine-tuning on collected experience is highly ineffective (3.8% recovery), while Experience Distillation retains 64.8% of ICL gains.
• Sample efficiency: Shows that ICL + Experience Distillation matches classical RL baselines with at least 9.6× fewer environment samples, highlighting practical efficiency.

Results

• Software-engineering tasks: Experience Distillation retains 64.8% of ICL gains; supervised fine-tuning recovers only 3.8%.
• Text-adventure games: Similar pattern, with Experience Distillation retaining at least 64.8% of ICL gains.
• Comparison with RL: ICL + Experience Distillation matches classical RL baselines with at least 9.6× fewer environment samples.
• Sample efficiency: The method requires no additional environment interaction beyond the collected experience, making it highly practical.

Significance

This work has broad implications for AI practitioners deploying agents in cost-sensitive environments. By enabling agents to internalize ICL gains without extra environment interactions, Experience Distillation reduces the barrier to effective learning from limited experience. It also opens avenues for combining the flexibility of ICL with the permanence of weight updates, potentially leading to more robust and sample-efficient learning systems. The demonstrated success on both software-engineering and text-adventure tasks suggests applicability across diverse domains, from automated coding assistants to interactive game agents.