Loading...
Loading...
Loading...
- Combines custom data with LLMs
# RAG (Retrieval Augmented Generation) Implementation - Technical Outline ## RAG Fundamentals ### Core Concept - Combines custom data with LLMs - Addresses limitation of LLMs trained on public data vs private data needs - Three main steps: indexing, retrieval, generation ### Context Window Evolution - Historical: 4-8K tokens (dozen pages) - Current: up to 1M tokens (thousands of pages) - Enables feeding large masses of private data ## Basic RAG Pipeline ### Indexing - Document loading and splitting - Token counting for embedding model limits (512-8000+ tokens) - Embedding generation using models like OpenAI embeddings - Vector representation (fixed-length, e.g., 1536 dimensions) - Storage in vector stores (e.g., Chroma) ### Retrieval - Semantic similarity search in high-dimensional embedding space - K-nearest neighbor (KNN) search - Cosine similarity computation - K parameter determines number of retrieved documents ### Generation - Prompt templates with context and question placeholders - Document stuffing into LLM context window - Chain composition using LangChain Expression Language (LCEL) - Parsing output to structured responses ## Query Translation Techniques ### Multi-Query Approach - Question reframing from different perspectives - Addresses embedding alignment issues in high-dimensional space - Generates multiple sub-questions from single input - Retrieval on each sub-question with unique union of results ### RAG Fusion - Similar to multi-query with reciprocal rank fusion (RRF) - Consolidates multiple retrieval lists into single ranked output - Aggregates documents across independent retrievals ### Query Decomposition - Breaks complex questions into sub-problems - Sequential solving with interleaved retrieval - Chain-of-thought reasoning with retrieval augmentation - Builds solutions using prior question-answer pairs ### Step-Back Prompting - Generates more abstract questions from specific inputs - Few-shot prompting for step-back question generation - Independent retrieval on both original and step-back questions - Combines conceptual and specific knowledge ### HyDE (Hypothetical Document Embeddings) - Maps questions into document space - Generates hypothetical documents using LLM world knowledge - Embeds hypothetical documents for retrieval - Returns actual documents based on hypothetical document similarity ## Routing and Query Construction ### Logical Routing - Uses LLM reasoning with structured outputs - Function calling with Pydantic data models - Routes to different data sources (vector stores, SQL, graph DBs) - Structured object outputs for deterministic routing decisions ### Semantic Routing - Embeds questions and available prompts - Computes similarity between question and prompt embeddings - Selects most similar prompt based on semantic matching ### Query Construction for Vector Stores - Natural language to metadata filters conversion - Function calling with schema binding - Structured queries for range filtering (dates, counts, lengths) - Semantic search combined with metadata filtering ## Advanced Indexing Strategies ### Multi-Representation Indexing - Decouples raw documents from retrieval units - LLM-generated summaries optimized for retrieval - Summary embedding for search, full document for generation - Separate vector store and document store architecture - Document ID referencing between stores ### RAPTOR (Hierarchical Indexing) - Recursive clustering and summarization of documents - Tree-like abstraction hierarchy - Leaf nodes: raw documents/chunks - Internal nodes: cluster summaries - Supports both detailed and high-level questions - Recursive embedding, clustering, and summarization process ### ColBERT Indexing - Token-level embeddings instead of document-level - Per-token vector representations with positional weighting - MaxSim operation: maximum similarity per question token - Final score: sum of maximum similarities across all question tokens - Higher granularity than traditional dense embeddings ## Active RAG and Flow Engineering ### Corrective RAG (CRAG) - Document relevance grading post-retrieval - Web search fallback for irrelevant documents - Knowledge refinement stage - Conditional routing based on document quality ### Self-RAG - Multi-stage grading: relevance, hallucinations, answer quality - Iterative improvement with feedback loops - Question rewriting for failed retrievals - Self-reflective generation process ### LangGraph Implementation - State machine architecture for complex RAG flows - Node-based processing with state modifications - Conditional edges for decision-making - Flow engineering vs traditional chaining - Graph state as shared dictionary across nodes ## Adaptive RAG Systems ### Query Analysis Integration - Combined routing and query modification - Tool use for structured decision making - Multiple data source routing (vector, web, LLM fallback) ### Multi-Stage Grading - Document relevance assessment - Hallucination detection - Answer usefulness evaluation - Automatic retry mechanisms ### Cohere Command-R Integration - Tool use and query writing capabilities - 35B parameter model optimized for RAG - Fast inference for real-time grading - Open weights for local deployment ## Long Context vs RAG Considerations ### Multi-Needle Analysis - Testing retrieval of multiple facts from long contexts - Performance degradation with increased fact count - Reasoning complexity impact on retrieval accuracy - Recency bias in long context models ### Context Stuffing Limitations - No retrieval guarantees in long context models - Higher latency and token costs - Security and authentication challenges - Audit trail difficulties ### Document-Centric RAG Evolution - Full document retrieval vs chunk-based approaches - Long context embedding models (32K+ tokens) - Reduced parameter sensitivity (chunk size, k values) - Multi-representation indexing for document-level operations ## Technical Implementation Details ### Vector Stores and Embeddings - Chroma DB for local vector storage - OpenAI embeddings (1536 dimensions) - Cohere embeddings integration - Sparse vs dense vector representations ### LangChain Integration - LCEL for chain composition - Retriever abstractions - Prompt templates and parsing - Tool binding and structured outputs ### Evaluation and Observability - LangSmith for trace analysis - Multi-stage pipeline debugging - Performance monitoring across graph nodes - Grading chain evaluation and validation
This roadmap outlines planned enhancements to transform cheap-RAG from a functional document retrieval system into a production-ready, state-of-the-art RAG framework. Priorities are based on impact vs. effort analysis and alignment with mainstream RAG best practices.
See `specs/Semblance-MVP-Plan-v2.md` for full technical specification.
All notable changes to AvocadoDB will be documented in this file.
**Goal:** Stand up Toasty as a reliable service wired to BLT/GitHub events; deliver safe, useful summaries early.
