RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation

Analysis

Why This Paper Matters

Retrieval-Augmented Generation (RAG) has become a cornerstone for integrating external knowledge into large language models, but its practical deployment is hampered by high latency due to long sequence generation and retrieval dependencies. This paper identifies that while retrieval introduces performance challenges, it also offers unique optimization opportunities through its retrieval pattern and streaming search behavior. By addressing these, RAGCache provides a systematic solution to reduce end-to-end latency, which is crucial for real-time applications like chatbots and question answering.

The significance is underscored by the growing adoption of RAG in industry. As models scale and knowledge bases expand, the latency bottleneck becomes more pronounced. RAGCache's approach of caching intermediate states and overlapping retrieval with generation is a novel contribution that could influence future RAG serving architectures.

Technical Contributions

• Knowledge Tree Caching: RAGCache organizes retrieved knowledge into a tree structure, caching intermediate states (e.g., KV caches) across GPU and host memory. This reduces redundant computation during generation, as common prefixes in retrieved documents are reused.
• Dynamic Speculative Pipelining: This technique exploits the streaming nature of retrieval to overlap retrieval with LLM generation. By speculatively starting generation on partial retrieval results, it hides retrieval latency.
• Implementation on vLLM and Faiss: The system is built on popular open-source tools, ensuring practical applicability and ease of adoption.
• Memory Hierarchy Optimization: By placing caches appropriately across GPU and host memory, RAGCache balances speed and capacity, addressing memory constraints.

Results

The evaluation on both open-source and production datasets shows that RAGCache reduces time to first token (TTFT) by up to 4× and improves throughput by up to 2.1× compared to a baseline of vLLM integrated with Faiss. These gains are substantial, indicating that the caching and pipelining strategies effectively mitigate the latency overhead of retrieval.

Significance

RAGCache addresses a critical bottleneck in RAG serving, making it more viable for latency-sensitive applications. Its techniques could be extended to other retrieval-augmented systems and inspire further research in caching and pipelining for LLM inference. The work also highlights the importance of co-designing retrieval and generation for end-to-end performance, a direction likely to gain traction as RAG becomes more prevalent.