OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Analysis

Why This Paper Matters

Long-context language models have become essential for tasks like in-context learning, self-improvement, and long-horizon agentic workflows. However, existing long-context corpora are dominated by books, academic articles, and code repositories, which are finite and often lack the long-distance dependencies that are crucial for complex reasoning. OctoLong tackles this bottleneck by introducing a context engineering pipeline that generates dependency-rich code contexts, offering a scalable alternative to traditional data sources.

The paper's significance lies in its demonstration that a relatively small proportion of such engineered data (12%) can substantially boost long-context performance across multiple benchmarks. This suggests that data quality and structure matter more than sheer volume, and that targeted context engineering can unlock capabilities in models without requiring massive increases in training data. This is particularly relevant for code understanding and agentic tasks, where dependencies span across files and functions.

Technical Contributions

• OctoLong Pipeline: The core innovation is a pipeline that combines an AST parser, a language server backend, and a package manager to recursively retrieve code references. This enables the construction of contexts that capture cross-repository dependencies, reaching millions of tokens in length.
• Mid-Training Strategy: The authors train OctoLong-Instruct models by mid-training on a ~50B-token mixture that includes ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. This two-stage approach extends context length while preserving instruction-following abilities.
• Comprehensive Evaluation: The models are evaluated against 18 state-of-the-art open-weight long-context LMs, covering tasks like long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks. This broad evaluation provides strong evidence of the method's effectiveness.
• Transfer to Short-Context: An interesting finding is that OctoLong data also improves API usage in short-context coding scenarios, indicating that the learned representations generalize beyond long-context settings.

Results

The paper reports substantial gains from adding OctoLong data. Specifically, supplanting just 12% of traditional context-extension corpora with OctoLong data yields significant improvements in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks. The models range from 600M to 14B parameters, showing scalability across sizes. The evaluation against 18 state-of-the-art models highlights the competitive edge of OctoLong-Instruct, though exact metrics are not provided in the abstract.

Significance

OctoLong addresses a critical gap in long-context modeling by providing a method to generate dependency-rich data that is not limited by finite resources like books or academic papers. This could accelerate progress in code intelligence, enabling models to handle entire repositories and complex agentic workflows. The finding that a small proportion of engineered data yields large gains suggests that future work may focus on data quality and structure rather than just scale. This work also opens avenues for applying similar context engineering to other structured domains, such as scientific literature or legal documents, where dependencies are also crucial.