Preprint
Reinforcement Learning

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

Yihao Chen, Shi Chang, Feng Lin, Khaled Chawa, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
July 29, 2026
arXivPDF

0

Citations

0

Influential Citations

—

Venue

2026

Year

Abstract

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

Analysis

Why This Paper Matters

This paper addresses a critical blind spot in LLM-based program synthesis: the inability to construct programs from scratch. While agents excel at editing existing codebases, they fail when given only documentation and a binary oracle, with frontier models solving fewer than 1% of instances in the ProgramBench benchmark. The authors identify that current frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, leading to insufficient probing, loss of behavioral intent, and propagation of early misinterpretations.

By drawing inspiration from classical requirements engineering, SpecFirst introduces a principled decomposition that forces specification elicitation before any code is written. This is a significant departure from end-to-end approaches and offers a structured way to resolve ambiguities early. The paper's empirical evaluation across four models and 200 instances provides strong evidence that this separation of concerns is beneficial.

Technical Contributions

  • Two-stage framework: SpecFirst separates behavioral specification elicitation (spec agent) from code synthesis (code agent), preventing context drift and ensuring a stable behavioral reference.
  • Spec agent design: The spec agent probes the binary oracle, combines observations with documentation, and produces a structured specification that resolves ambiguities before coding begins.
  • Code synthesis agent: Uses the specification as a stable guide, enabling earlier and more sustained code construction compared to single-loop baselines.
  • Comprehensive evaluation: Tested on all 200 ProgramBench instances across four models (two families, order-of-magnitude capability range), with statistically significant improvements in test pass rates (6.9%-21.3%) and binary exploration coverage (9.4%-18.5%).

Results

SpecFirst consistently outperforms the single-loop baseline across all models and instances. Key metrics: test pass rate improvements of 6.9%-21.3% and binary exploration coverage improvements of 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction, indicating that the decomposition reduces context drift and improves focus.

Significance

This work has broad implications for AI-assisted software engineering. By demonstrating that an explicit requirements-engineering phase is effective for from-scratch program construction, it opens a new paradigm for building reliable autonomous coding agents. The framework could be extended to other domains where behavioral oracles exist, such as API development or hardware-software co-design. It also highlights the importance of structured specification in reducing hallucination and improving task decomposition in LLM agents.

llm-agentsprogram-synthesisrequirements-engineeringbehavioral-specificationreinforcement-learningsoftware-engineeringcode-generation
Benchmarks

Related Papers