Can LLMs Do Retrieval and Reasoning in 1 Million Context Window?

Analysis

Why This Paper Matters

Long-context processing is a frontier challenge for large language models, with applications ranging from document analysis to code understanding. Existing benchmarks often conflate retrieval with reasoning or suffer from knowledge contamination by using real-world texts. NeedleBench addresses this gap by offering a fully synthetic, controllable evaluation framework that isolates retrieval and reasoning demands. This is particularly timely as models like GPT-4 and Claude now support million-token contexts, yet systematic stress tests remain scarce.

The paper's identification of 'under-thinking' in advanced reasoning models is a striking finding. It suggests that even state-of-the-art models may not fully utilize available context, instead defaulting to shallow reasoning patterns. This has direct implications for deploying LLMs in high-stakes domains where thorough evidence integration is critical.

Technical Contributions

• Synthetic data generation: NeedleBench creates bilingual (Chinese/English) contexts with key data points ('needles') placed at controlled depths and densities, eliminating reliance on real-world texts.
• Two distinct scenarios:
  • Information-sparse: Single needle in vast irrelevant text, testing basic retrieval.
  • Information-dense (Ancestral Trace Challenge): Multiple related needles distributed continuously, requiring multi-step reasoning.
• Adaptive context lengths: Supports lengths from 4K to 1M tokens, enabling systematic scaling analysis.
• Open-source implementation: Integrated into OpenCompass, allowing easy reproduction and extension.

Results

• Deepseek-R1 and OpenAI o3 achieve near-perfect scores on mathematical reasoning benchmarks but drop significantly on NeedleBench's information-dense tasks, even at 4K context.
• Under-thinking is quantified: models answer correctly only 60-70% of the time when relevant information is present, compared to 90%+ for simpler retrieval tasks.
• Performance degrades more sharply with context length for reasoning tasks than for retrieval, indicating a fundamental limitation in multi-step reasoning over long contexts.

Significance

NeedleBench exposes a critical blind spot in current LLMs: the inability to perform sustained reasoning across dense, long contexts. This challenges the assumption that scaling context length alone suffices for complex tasks. The framework provides a rigorous tool for future model development, encouraging architectures that better integrate retrieval and reasoning. For practitioners, it underscores the need to validate models on realistic long-context scenarios beyond standard benchmarks.