Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems (April 2026)
FreeMemory mechanism that retrieves compressed reasoning "thoughts" rather than raw context — enables more efficient and reasoning-aware memory for long-horizon agents
About Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems (April 2026)
Thought-Retriever is a model-agnostic algorithm designed to enhance large language models (LLMs) by enabling them to retrieve compressed reasoning 'thoughts' from a self-evolving long-term memory, rather than raw data chunks. This approach overcomes the context length limitation of traditional retrieval-augmented generation (RAG) by leveraging an LLM's own intermediate responses from past queries. The system filters meaningless thoughts, organizes them in a thought memory, and retrieves relevant ones for new queries. Extensive experiments on the new AcademicEval benchmark and two public datasets show that Thought-Retriever outperforms state-of-the-art baselines with at least 7.6% improvement in F1 score and 16% in win rate. It also demonstrates self-evolution over time and the ability to use deeper thoughts for more abstract queries.
Key Features
Pros & Cons
- Outperforms state-of-the-art retrieval methods by at least 7.6% in F1 score and 16% in win rate across various tasks
- Enables LLMs to self-evolve and improve after solving more user queries
- Can incorporate arbitrarily long external data without being limited by context window
- Learns to leverage deeper, more abstract thoughts for complex queries
- Requires generating and storing intermediate LLM responses, which may increase computational overhead
- Performance depends on the quality of the LLM's intermediate reasoning steps
- Currently validated only on academic benchmarks; real-world deployment may face additional challenges