Abstract Reasoning Composition

Analysis

Why This Paper Matters

This paper addresses a critical inefficiency in current LLM reasoning: the insights and patterns discovered during long reasoning traces are discarded after each query. By introducing concept-level memory, the authors propose a way to persist and reuse these discoveries across queries, enabling test-time continual learning without modifying model weights. This is particularly relevant as inference-time scaling becomes more prevalent, as it allows the benefits of extended reasoning to accumulate over time.

The focus on abstract, modular concepts rather than instance-specific memories is a key differentiator. This aligns with the goal of compositional generalization, where solving new problems requires recombining previously learned patterns. The evaluation on ARC-AGI, a benchmark designed to stress abstract reasoning, makes the approach particularly compelling, as it directly targets the kind of generalization that static models struggle with.

Technical Contributions

• Concept-level memory: Instead of storing exact query/response pairs or summaries, the method distills reusable, modular abstractions from solution traces. These are stored in natural language, making them interpretable and flexible.
• Abstraction strategy: The paper introduces a method to extract generalizable takeaways from rollouts, focusing on patterns that can apply to future queries.
• Retrieval mechanism: For each new query, relevant concepts are selectively retrieved and integrated into the prompt, ensuring that the memory is used efficiently.
• Dynamic memory updating: The memory is updated during test-time as new solutions are generated, allowing the system to improve its performance over time without retraining.
• Scaling with inference compute: The method is designed to benefit from increased inference compute, with performance improving as more reasoning is allowed.

Results

The method achieves a 7.5% relative improvement over a strong no-memory baseline on ARC-AGI. This gain is consistent across all tested inference compute scales, indicating that the concept memory provides a robust advantage. Furthermore, dynamically updating the memory during test-time outperforms fixed memory settings, supporting the hypothesis that accumulating and abstracting patterns enables further solutions in a form of self-improvement. The results suggest that the approach not only improves accuracy but also scales effectively with additional compute, making it a promising direction for future LLM reasoning systems.

Significance

This work has significant implications for the field of AI, particularly in the areas of continual learning and reasoning. By enabling LLMs to build and reuse a memory of abstract concepts, it moves toward more adaptive and self-improving systems that can learn from experience without weight updates. This could lead to more efficient and capable models for complex reasoning tasks, reducing the need for retraining and allowing for personalized or domain-specific knowledge accumulation. The approach also opens up new research directions in memory-augmented reasoning, such as exploring different abstraction formats, retrieval strategies, and integration methods. Overall, this paper provides a strong foundation for future work on test-time continual learning and compositional generalization in LLMs.