MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

Analysis

Why This Paper Matters

This paper addresses a critical bottleneck in AI-assisted software engineering: the inability of current coding agents to construct complete programs from scratch. While existing systems excel at modifying existing codebases (e.g., bug fixing), they struggle with greenfield development—frontier models resolve fewer than 1% of tasks on ProgramBench. The core issue is the lack of scalable training environments that cover the entire software engineering life cycle, from design to deployment. MindForge tackles this by automating the creation of such environments from open-source command-line programs, providing a practical path to generate high-quality training data without requiring access to proprietary source code.

The significance lies in its potential to level the playing field. By enabling small models (e.g., 27B parameters) to achieve performance comparable to much larger frontier models, MindForge reduces the computational and data barriers that have favored large-scale proprietary systems. This could accelerate the adoption of AI-assisted programming in resource-constrained settings and foster more open research in code generation.

Technical Contributions

• Source-Free Environment Construction: MindForge converts open-source command-line programs into environments that expose only a compiled executable and its documentation, eliminating the need for source code access. This is achieved through an automated pipeline that compiles the program and strips source artifacts.
• Teacher-Agent Trajectory Generation: Using GLM-5.2 as a teacher, the system generates program synthesis trajectories—sequences of actions and observations—within these environments. The trajectories are curated for quality, ensuring they cover the full software engineering life cycle.
• Fine-Tuning Recipe: The trajectories are used to fine-tune Qwen3.6-27B, a relatively small model, demonstrating that targeted training on whole-life-cycle tasks can yield substantial gains without scaling model size.
• Comprehensive Evaluation: The fine-tuned model is tested on seven diverse benchmarks (e.g., RepoZero-C2Rust, DeepSWE, SWE-bench Verified), showing consistent improvements across repository generation, translation, bug fixing, feature implementation, and cross-language issue resolution.

Results

The fine-tuned Qwen3.6-27B achieves a 49.51% test pass rate on ProgramBench, a 11.53 percentage point improvement over the base model (37.98%). This performance is comparable to frontier models despite being significantly smaller. On other benchmarks, absolute gains include 31.00 points on RepoZero-C2Rust, 14.16 on DeepSWE, 10.70/4.56 on NL2Repo-Bench (with/without tests), 5.04 on SWE-bench Verified, 5.93 on SWE-bench Pro, 5.22 on SWE-bench Multilingual, and 4.94 on FeatBench. These results underscore the effectiveness of the training environment and data recipe.

Significance

MindForge introduces a scalable, automated approach to generating training environments for whole-life-cycle software engineering, addressing a key limitation in current AI coding agents. By enabling small models to match frontier performance, it democratizes advanced code generation capabilities and reduces reliance on massive proprietary models. The method's generalizability across diverse benchmarks suggests it could become a standard tool for training and evaluating code generation systems, potentially accelerating progress in automated software development and lowering barriers to entry for researchers and practitioners.