Procedural Knowledge in Pretraining Drives Reasoning in LLMs

Analysis

Why This Paper Matters

This paper addresses a fundamental challenge in evaluating LLMs: the impossibility of train-test set separation due to the massive scale of pretraining data. Traditional generalization metrics are confounded by potential data leakage, making it unclear whether LLMs truly generalize or simply retrieve memorized answers. By directly analyzing which pretraining documents influence model outputs, the authors provide a novel window into the mechanisms underlying reasoning.

The findings are significant because they challenge the notion that LLMs rely on retrieval for reasoning. Instead, they show that reasoning tasks are driven by procedural knowledge—documents that demonstrate how to solve problems—rather than answer-containing documents. This suggests that LLMs can synthesize generalizable strategies from examples, which is a more sophisticated capability than simple memorization.

Technical Contributions

• Influence Analysis on Pretraining Data: The authors apply influence analysis to attribute model outputs to specific pretraining documents, scaling to 2.5B tokens and models up to 35B parameters.
• Contrastive Analysis: They systematically compare reasoning tasks (three mathematical reasoning tasks) with factual questions to highlight differences in data reliance.
• Quantitative Metrics: They measure the overlap of influential documents across questions within the same task, showing that reasoning tasks share influential documents while factual questions do not.
• Qualitative Characterization: They manually inspect top-ranked documents for reasoning questions, confirming the presence of procedural knowledge like formulae and code.
• Negative Results: They show that answers to reasoning questions and intermediate steps are not among the most influential documents, ruling out retrieval-based explanations.

Results

The paper reports that for factual questions, the correct answer often appears in the most influential documents. In contrast, for reasoning questions, neither the final answer nor intermediate reasoning steps appear as highly influential. Instead, the influential documents often contain procedural knowledge, such as step-by-step solutions or code snippets. The overlap of influential documents across reasoning questions within the same task is high, indicating shared procedural knowledge. This pattern holds across both model sizes (7B and 35B), suggesting robustness.

Significance

This work provides a new methodology for studying generalization in LLMs without relying on train-test separation, which is a major step forward. It also offers evidence that LLMs can learn generalizable reasoning strategies from procedural examples, which has implications for how we train and evaluate models. Understanding that reasoning is driven by procedural knowledge could lead to more targeted data curation and training objectives. Furthermore, the contrast between factual retrieval and procedural synthesis clarifies the nature of LLM capabilities, potentially guiding future research on reasoning robustness and out-of-distribution generalization.